People generally agree investing in usability for Websites and Web applications is a “good idea” when it comes to retaining visitors and users—or in my specialty, survey respondents. The challenge is assigning a value to that investment because product managers, graphic designers, user interface specialists, technical support managers, trainers, and executives will often have wildly differing opinions.

It was W-2 filing time, and while I value my accountant for anything convoluted or when I need advice, I tend to take care of routine reports spewed out by QuickBooks. At the employee information deadline a month before, I’d saved a bunch of Adobe PDF documents and instructions from QuickBooks, but ignored them until now when the reporting deadline was approaching. Little did I know I was about to become an impromptu lab rat.

As an alternative to hardcopies, my accountant had given me a link to http://FileTaxes.com which was a mere $4 per W-2 to submit electronically. So I went to the site and zipped through a straightforward company account setup. Then I brought up my W-2 in an Acrobat PDF I’d saved the prior month, and copied and pasted information into their Web form—an exercise made easier by their form layout being almost identical to my PDF’s. No fuss, no muss. If I had several employees’ information to submit, I expect the import process from QuickBooks would also have been fairly smooth.

Excellent usability is like excellent service: you don’t notice it because what you want is simply there when you need it, unobtrusive and adaptive to different styles (within limits).

Alternatively, think of rental cars—or a new DVR or DVD player. With some, every time you reach out for a control, it’s near where you expected, and any new functions quickly make sense. With others, you wonder whether aliens were involved in the design, hope you don’t hit something awkward in your 60mph grope for the wiper controls, never tap many of the features you purchased for, and resort to calling tech support.

At the checkout screen, just before officially filing, I was reading the last of the instructions QuickBooks had generated to make sure I hadn’t missed anything. An odd paragraph starting with “Caution” sounded as if the Social Security Administration offered its own e-filing Website. While paying $4 to finish on the current site was a shrug, I’d also only invested a few minutes in the current form, and since I was just getting started with e-filing it seemed worthwhile to scope out the source.

So I left my FileTaxes.com page where it was, opened a new browser window, and brought up http://socialsecurity.gov/employer/.

On the Social Security site, I started with “First Time Filers.” I discovered not only was I going straight to the source instead of running my data through a third party, it turned out it would be free instead of pocket change. Small kudos for my few minutes’ investigation!

Next, in an unusually compliant moment, I followed their suggestion to start with the “Learn How” link, and then looked at the tutorial for “Business Services Online Registration.” The apprehension begins. In fact, the reptilian portion of my brain had an excellent impulse—which I unfortunately suppressed—on seeing that tutorial: Close the browser, grab my credit card, and shell out $4 as fast as the first site would take it. At the time I didn’t even notice that the registration tutorial was 64 pages long, just that it was far too much compared to the click-type-click-don’t-read-anything-all-done experience of the first site (here’s the tutorial if you must see).

It’s a usability red flag if you’re directing users to detailed tutorials in advance of basic tasks like registration. They should be able to dive in, and either complete with no assistance or make it through with non-disruptive pop-up style help while in the application. If you suspect your usability has significant snags, look at the amount you’re spending on documentation and technical support, and consider whether any of that budget can be better leveraged at the user interface—i.e. draining the swamp instead of fighting the alligators.

But, persistent me, I didn’t leave the site. Instead I merely fled the tutorial and proceeded through the registration, hoping there was nothing important in the documentation because I wasn’t going back there unless I got stuck.

Within the registration, the first oddity was that even though this was SocialSecurity.gov/employer and “Business Services” I had to register as an individual based on my social security number, home address, and other information. Apart from the conditioning we’ve all been given over the past decade not to base accounts on our SSN (yes, it felt awkward even to do so with the Social Security Administration) I kept wondering whether I should be entirely consistent with personal data or put the contact information I wanted for what was really a business account.

Any time someone is slowing down to debate whether to answer, or what’s the right response, you’ve got a usability snag. Depending on how engaged they are in your site or available competitors are, it’s a place they may exit.

The account setup also wouldn’t let me pick a user name, instead assigning an arbitrary set of letters. While it did let me pick a password, that value will expire in 90 days, at which point I believe I may get to answer all 5—yes, five—of my re-set questions to pick a new soon-to-expire value.

Beware of burdensome security policies like this design—the net result is little Post-it notes with passwords on them, which is far less secure than simply letting users pick a non-expiring username/password combination they can remember.

While the first site was a page or two to register (it went by so fast I barely noticed), the second was many screens with optional services, some of which were baffling and included alarming wait times, but I left it on the defaults and ploughed through—I just wanted to submit my W-2 after all! I’d even started the process from a link that said “Electronically File Your W-2s.” At this point, my reptilian brain’s impulse had been more than validated, and while I was past the registration stage, the difference in the username/passwords alone might be enough to get me to the paid site next year.

But I just had to see it all, so onward to the W-2. Like the first site, the basic form layout was very similar to my PDF. There was only one little issue. In the first site, the SSN and dollar fields were a single blank so I just picked up the whole text string in the PDF and pasted it in. In the Social Security site, they’d “helped” by parsing the fields into multiple pieces: three blanks for the SSN ### - ## - ####, and dollars/cents, for all the currency fields. This meant either triple or double copy/pasting to get the text snippets, or resorting to typing bits—either way it not only increased effort, it also increased the odds of making an error.

Often good intentions have unplanned results. The designer may never use copy/paste, so they may think the design that enforces the character counts in the parsed fields is a slick feature rather than a limitation. This was irritating with the one W-2 I had or a handful, but in any kind of bulk hopefully one would be doing an import instead. Unfortunately, I’ve also seen users take amazingly inefficient paths because once they figured out a way that worked, they stopped looking for new routes.

Once I had the W-2 complete, I had a false start to actually get it officially submitted vs. having it hang out in draft limbo on their server. It turned out the button I needed was “Go to W-3,” which happens to be the official summary report. Once I made it there, I got to the final submission, and saved off a copy of the report for my files.

This snag was another common pitfall: jargon rather than plain language. True, the buttons did have explanations next to them, and if I’d bothered to read all the instructions before clicking on a likely contender I could have avoided my false start. Also true, I had no need to read fine print or pick “likely contenders” on the first site when I was ready to finish and file.

Thus ends my impromptu Web form usability saga:

• One site was quick, polished, intuitive, but $4/form to use
• One site was bloated with text and pages, cryptic, and expensively documented in video/PDF/text, but $0 at low volumes

And next time I’ll sprint to the one that takes my money and doesn’t make my head hurt.

Now if I’m willing to switch sites for better usability when I’m actively pulling out a card to shell out money—no matter how small the amount—imagine the impact usability has on visitors who are passively browsing a site or doing you a favor by completing a survey?

So for your visitors and users, even if $4/form isn’t the right number, what is? Perhaps usability might be worth a bit more of an investment than you thought. And on that note, an excellent book is Steve Krug’s “Don’t Make Me Think.”

Correlation coefficient

Even before running the correlation, always chart the results. Your brain is the best analysis software made, and it will spot many oddities in graphical form which can be masked when the data are reduced to a single value. See also What's your executive dashboard hiding? and the Wikipedia article which shows varied scatterplots with the same coefficients.

A “perfect” correlation is 1 or -1, with 0 being no statistical relationship. On occasion, you’ll get nice strong correlations like these two:

A client whose survey came up with a weak correlation asked if I had a rule of thumb for reporting results—which is a question nearly guaranteed to produce a “It’s more than just the number” spiel from me. Whether your correlation generates a 0.96 or 0.56, it’s always worthwhile to review the reliability of your inputs before making major moves on a statistical output. And remember, your perspective on those inputs is often quite different at the end of a project than at the beginning.

Cause and effect

With any multivariate analysis, the goal is to explain the greatest amount of cause and effect with the fewest number of variables—also known as parsimony, close cousin to Occam’s Razor. Occasionally this leads to oversimplification, but in the real world of humans and complex interactions, a more likely problem is overlooking or working around a hole in the interactions:

• Researchers often have blind spots about what’s truly causing an effect, and the entire study may be conducted without that key question
• Sometimes a factor is too sensitive to ask of respondents
• The true cause may be something you can ask of respondents, but not available for practical purposes such as media buying

In any of those cases, what you may end up with in your analysis, regardless of the strength of your correlation value, is a calculation along this indirect lower route:

This isn’t wrong so much as precarious—especially if you’re dealing with an indirect chain a couple Effects long or where the Faux Cause is fed by several root Causes. If you’re clear on what you’re correlating, and temper your business decisions with the lower reliability, you’re fine. But if you’re on the lower path and think you’re correlating against a root Cause, that’s where you can get into trouble. In that case, not only will you be counting on those results more than they may deliver, as long as you believe you know the Cause, you won’t be looking for more puzzle pieces and explanations.

Sample quality

As always, the more representative your sample is and the more replies you have, the more reliable your statistics are. See Real world sampling for a crib sheet.

How important is it really?

Statistics are one of those “Just because we can, doesn’t mean we should” areas. Numbers have a facade of precision and reliability that make them attractive, and the endless variants—especially when you start digging at this level—can give new meaning to Analysis Paralysis, keeping your team from focusing on what’s really important to your business.

Before you worry too much about a correlation coefficient that comes up at a moderate 0.75 instead of a strong 0.90, step back and look at how much the application of that particular Cause/Effect relationship is going to impact your operations. Is it a bit of understanding or is it a key driver of a multi-million dollar product launch? The statistic is always a tool—your business need is always what drives the decision. Once you get the context back, you can decide whether the 0.75 is simply something to watch in future research while you move on to more important factors, or is critical enough to clarify now with an additional survey.

Alternate analysis

If on the scatterplot you think you see a pattern (versus a uniform blob), but the correlation coefficient isn’t coming up with much, here are two alternate statistics.

• Chi Square
  Straightforward to calculate, and in consolidating the segments you may bring a picture of the relationships into focus at the statistical reliability you're hoping for. See Understanding Chi Square.
• Cluster analysis
  Not straightforward to calculate, but a powerful tool if you are a statistician or have one at hand. If this is of interest, I’d start with the Wikipedia article Cluster analysis (in marketing) and then move on to your favorite cluster-supporting statistics software (SPSS, SAS, etc.) rather than attempting to read the assorted formula-laden pages Google will bring up.

Reporting results

As I told my client with the 0.56 correlation, I think the best research is more about relationships and drivers than tallies. If you have an intriguing but weak relationship, mentioning it as something to be explored and/or watched in the future is not only valid, it adds depth.

A few suggestions:

• If the weak/dubious results would distract from your more substantive findings or potentially cause confusion, I’d leave them out of the main report. You could still publish them as “further reading” for those who are curious.
• Remember that some of the report readers won’t have been in the meetings where you hashed out the dubious reliability of certain charts or explained about all the stats (or their bodies were present but attached to a Blackberry, or it’s been 9 months). Be sure you have captions or commentary explaining why certain figures/stats are less reliable than the majority of the results. In many cases, that forgetful reader will be yourself, and your overworked gray matter will be thankful for the extra notes!
• Make it as obvious as you can to someone skimming the document what’s in this “curiosity” category, such as with sidebars, boxed out figures, or a separate section at the end of the report.

• You were right! All is well in the universe, the sun continues to shine.
  They like your feature best, frequent buyers have higher satisfaction levels, and last year’s hybrid matrix re-org was the best thing since sliced bread.

• Crickets. Nothing seems to be happening.
  Everything’s just, well, Good, and you can’t find much in the way of differences between customer segments or before and after the re-org. Huh.

We all love to be right, but the important thing is to remember that a survey is all about getting information. Because if we’re not open to getting accurate information—whatever it may be—we’re just putting blinders on and setting ourselves and others up to stumble around the business landscape.

This is setting aside the minority who make ethical choices to slant "research" through loaded questionnaires or outright falsification of data. Instead it's a common sense reminder to everyone who's trying to juggle their organization's performance metrics, group goals, survey data, personal theories, and any other elements, of what's important to the organization at the end of the day. Also, remember that survey data is always only part of your decision process, integrated with secondary research and the knowledge of key stakeholders—never taken by itself as Sole Truth, no matter how good your sample and methodology.

Happy Surveying!

For non-cash rewards, your first thoughts may be of items related to your survey or industry. The challenge there is coming up with an item they haven't already bought for themselves. Instead, try to think outside the box—like the best gifts, it may be something the respondent wouldn't think to buy themselves:

• Is anything hot and in scarce supply, such as the iPhone or Wii when first released? Even if the recipient doesn't want it themselves, odds are they have a friend or family member who does.

• What about luxury indulgences? (My personal favorite.)

• How about something experiential? Depending on the population, it could be anything from a dinner cruise to the Zero G.

• While no incentive will appeal to everyone, try to avoid anything which is potentially distasteful (such as the steak a month club to a vegetarian).

• Can you pair the reward with a cash equivalent or give them a choice of prizes?

Business populations are also better motivated through emotion than logic. If you're running a conference, you may be inclined to give away a free pass for next year. That's a great carrot if attendees are self employed or personally paying for their continuing education. However, if most of the attendees are sent by their employer, they'll be less motivated by your $1,500 pass than a $200 goodie. Likewise, offering a copy of the survey results is only a significant incentive if the respondent's company wouldn't be willing to buy a copy.

Divvying up your budget

Since the point of incentives is to increase your response rate, it's worth a moment to consider whether some of the funds may be more effective elsewhere. This could take the form of improving the respondent experience by getting help tuning the survey or using better technology. It could also be used for follow-up, such as hiring a temp to call non-respondents.

When you have the total you plan to spend on respondents, look at what you could spend individually, on a few items, or on one prize—a chance at a big item can be more motivating than a modest guaranteed amount (there's a reason we have the lottery).

Remember that what matters is the respondent's perception of the reward. I saw one project where they planned to offer $100 per person, which is almost an insult when asking a CEO to complete a 3 stage 60+ minute survey. However, that $100 could become a nice sign of appreciation when sent to the charity of the CEO's choice, or a tempting carrot if the budget for all 1,000 respondents was pooled into a few prizes.

Managing individual compensation

If you are providing individual payments, there are a few logistics:

• Qualify respondents at the beginning of the survey, exiting them to a "sorry" page if they aren't part of your target population.

• Make sure the survey cuts off at your budgeted number of completions, and check with tech support about exactly how the limit works. Some hosting services which cap your data file at a basic/gold/platinum subscription level will continue to allow submissions beyond that number. This can be great when you want to ransom additional data, but a serious problem when everyone is expecting a check.

• To prevent duplicates, you'll have to either issue passwords or carefully clean the data.

• If you don't want to deal with printing checks, an Amazon.com gift certificate is almost as flexible as cash.

• If you're surveying a general population rather than customers or employees, a panel may make sense.

Despite being in a quantitative industry, surveyors rarely conduct tests to measure what happens when we rearrange questions, add a neutral point in a scale, or make other adjustments. If you're ready to absorb some more advanced issues, complete with footnotes, this is a great book to pick up. Note that the 1996 copyright is simply a reprint of the 1981 text.

Amazon

When you want to broaden your perspective, this will help you understand how customer and employee satisfaction mixes with and reinforces other marketing efforts. While the author focuses on services, it's useful in any industry—we're all competing on intangibles these days.

Amazon

Recommend for what? I'm aware of the school's solid reputation in engineering and veterinary medicine, but have no notion where their other programs rank.

Recommend for whom? I have to think of an individual as to whether the programs, lifestyle, location and tuition (in-state resident vs. full) would be a fit.

In your own surveys, do you have questions people would answer with "It depends..."?

There are a few common ways they'll slip in:

• Trying to shorten a survey, replacing a set of targeted questions with a general one
• A quest for The Metric that will encapsulate the respondent's likely action or overall impression
• Relying on context—even in a survey about a specific product, "Would you recommend us?" will have some respondents thinking of your company (diverse) instead of that product line

Sometimes, all it takes is a little qualifier:

Or even making the question more generic:

The more generic approach helps respondents whose "It depends" is based on inconsistent experiences.