• I'm collecting examples for my library of screenshots
• I'm interested in the experience of answering surveys
• I'm opinionated and enjoy giving people my views
• I even quite like getting rewarded for my time
• Sometimes I'm compiling statistics about how many invitations I get, and how many I manage to respond to.

But today, even I began to crack. I needed a break from work and thought I'd tackle a few of the survey invitations lurking in my email in-box. Two of them were from a panel that I'm a member of; one was from a research institute. I'm going to start by ranting trying to talk calmly about the panel experience first, and then bring in the research one.

Why do you repeatedly ask me the same questions?

Here's what I'd like to say to the owner of the panel I'm a member of.

I'm supposed to be a *member* of your panel. Membership implies some sort of belonging, doesn't it? Shouldn't there be some sort of continuity in that experience?

But no. Every single survey that I get from you starts by asking me a slew of the same questions. I'm simply getting tired of telling you, again and again:

• my age
• where I live
• my marital status
• my employment status
• my household income
• that I'm a business owner
• the number of employees in my business
• that I'm the main decision-maker for my business...
• and on and on and on.

Why can't you just remember? Please?

Why don't you let me answer, occasionally?

One of the surveys today started with this message (I've lightly anonymized it):

"Thank you for agreeing to take part in this evaluation of the XXXX website. Your feedback will help us improve it".

As it happens, at that point I hadn't agreed to anything - I'd just clicked on a survey invitation that made no mention of what type of survey I'd be getting, or for whom. But we'll ignore that problem and focus on the cheery idea that 'my feedback will help to improve a website'. Great! I'm an opinionated person and I enjoy giving advice about how to improve websites. Actually, I enjoy that so much that it's what I do for a living, what I do for my academic activities, and what I do for relaxation. My lovely, long-suffering husband might complain that I don't do much else. I was feeling genuinely rather happy that I'd get a chance do to it as part of this survey.

But no. Screened out again.

As it happens, this particular panel has screened me out of every single survey invitation I've received this year. I'm at the point where I'm going to screen them out.

So here's another thing I'd like to say to them:

If you don't want my responses, why keep asking me? If you do want my responses, why not humor me by letting me respond, sometimes?

Why do you pound me into cheating?

Greg Peterson posted on The Survey Geek's blog about a respondent panel at the CASRO Online Research Conference in March 2012. His post is called The end of "don't ask don't tell" in online survey research. That's because some opinion-formers in the market research industry consider that online panels have been suspect for ages, full of 'professional respondents' who are interested only in the money.

The panel was formed of not just any respondents, but actual 'professional respondents', the dedicated people who sign up for multiple panels because they do enough survey-answering to gather some actual financial reward. (After two or three years of trying to amass a reward from the panel I'm a member of, I have yet to gain enough points to be able to redeem them).

And indeed Greg Patterson's post starts with some stories that should make us all wince, such as respondents cheerfully admitting to taking the same survey more than once.

Despite that, he devotes a big chunk of his post to pointing out that the respondents do in fact prefer to answer honestly. For example: "These folks take their survey taking responsibilities seriously. They did surveys at least in part because they really like giving their opinions".

I've observed my own survey behavior includes both approaches:

• Honesty, trying to give my truthful opinion, mostly for the intrinsic reward of feeling good about it.
• Speeding, trying to answer everything as quickly as possible without thinking about it, just to get to the end.
  

Let's be straight here: "speeding" is also known as "cheating".

And here's a thing: speeding isn't as much fun as you might think. It's even more boring than honesty. At least with honesty, you occasionally notice that they've varied the question slightly (although many surveys are so mindlessly repetitive, that doesn't happen anything like as often as I'd wish).

Speeding on the highway often only shaves a rather small amount of time, especially over shorter, more interesting journeys. Speeding in surveys is the same: it doesn't save you much time if the survey is shortish, and it's actually more stressful than relaxing into honesty. It's only really worthwhile if the survey is especially long and remarkably boring.

What I've noticed in my own survey behavior is that I always start honest, and then sometimes resort to speeding when a survey has pounded me into that bad behavior by offering me a series of questions so boring, irrelevant, or repetitive that I've given up trying to make sense of them.

That's exactly what happened to me as the luckless respondent to the survey from the research institute. I was happy at the start because it was delivered as one page - a very, very long one, but at least I knew it wasn't going to screen me out. Result!

Then it got straight into real questions. No series of deeply boring demographic questions to fight through at the start. Result!

I attacked it positively, but was ground down by a hideously complicated set of instructions and then a series of rating tasks that I found incomprehensible. I tried, but it gradually dawned on me that I'd given up and was speeding. It wasn't a conscious decision to cheat - it was a behavior I'd been pounded into.

Three reasons why our response may not be what we want it to be

As survey designers, I think we need to look first at what we're asking respondents to do before we jump to condemn them as cynical 'professional respondents'. Are *we* guilty of any of these bad behaviors?

• Asking the same things again and again, particularly from panellists who are supposed to be 'members' with a continuing experience?
• Failing to let our respondents actually respond?
• Pounding them into bad behavior?

• purchase the recording from User Interface Engineering
• view the slides on SlideShare (or at the end of this post)
  

Resources that I mention in the seminar

• Post with my survey process: How to do a survey in six steps
• Slides that describe the process: Surveys in practice and theory

Books that I mentioned:

• Dillman, Smyth and Christian (2008) "Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method", my Survey book of the Month, January 2011
• Krosnick, J. A. and S. Presser (2009). "Question and Questionnaire Design", chapter in "Handbook of Survey Research" (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier. The chapter is available online: Question and Questionnaire Design (.pdf)
  
• Reiss, E. (2000) "Practical Information Architecture: A Hands-On Approach to Structuring Successful Web Sites". Amazon.com claims that it's out of print now, but it seems to be easily available second-hand for the cost of the postage. Obviously the examples are out of date, but if you can get beyond that then there is plenty in it that is still relevant today. Eric often talks about the concept of 'shared reference', for example in his talk "Content Strategists" (SlideShare)

Other resources:

• Stanford guidelines on web credibility
• Tips and tools for plain langauge, from Plainlanguage.gov
• My slides on how to do buttons and labels on forms
• Perceptual Edge (Stephen Few) has a graph design IQ test
• Case study by Cole Nussbaumer of how to improve an Excel graph so that it communicates a clear message: Storytelling with data
  
• Stan Sthanunathan, VP of Marketing Strategy and Insights at Coca-Cola, talks to Research Magazine on 'Why quality doesn't matter'
• Learning the languages of other disciplines to make our reports have more impact: "Who are you talking to?"
• My article on UXmatters: "How to get yourself started in statistics"
• Emily Geisen and I teach a workshop that explains how to run a usability test of questionnaire. The slides are here: "Introduction to usability testing for survey research"
  

Slides for "Designing Effective Surveys" virtual seminar

• Improving questions (step 1)
• The five further steps that make the difference between a set of questions and a survey
• Some key ideas from the survey methodologists.
  
• A final challenge: do you think those ideas are practical for you?

Surveys in practice and theory

One for fun - or to provoke some thought

First, the snags - and my take on them

1. It's probably got the wrong title.
   
2. It doesn't work properly on the grayscale Kindle, because of the importance of the color plates.

There are also some complaints that it's too long and too wordy. I agree with the two snags, but definitely not that it's too long: it's only 245 pages of text, plus notes, and I would have loved it to be three times as long. Maybe it is a bit too wordy, but it rattled along and I found the stories engaging.

Learning about languages and colors

A better title for the book might be "How languages deal with color", the main topic.

Deutscher begins with Gladstone's monumental treatise on Homer, published in 1858. The section of Gladstone's work that interests Deutscher is the one that points out that Homer rarely used color in his descriptions, and the few examples of a color description can be quite odd. For example: 'wine-dark sea'. Why describe the sea as the same color as wine? Isn't the sea blue, whereas dark wine would be dark red?

From this 19th century starting point, Deutscher traces theories about language and color through to the current century. Along the way, we learn about the Sapir-Whorf hypothesis and why it is now discredited. Don't know what the Sapir-Whorf hypothesis is? Neither did I: read the book and find out!

Along the way, he throws in a couple of chapters on the different ways that languages deal with gender and on egocentric compared to geographic systems of co-ordinates (e.g. is something to your left, or to the north of you?).

Here's a section of the discussion of gender:

The original sense of 'gender' had nothing to do with sex: it meant 'type, 'kind', 'race' - in fact, 'gender' has exactly the same origian as the words 'genus' and 'genre'. ... [Linguists] nowadays use it for any division of nouns according to some essential properties. These essential properties may be based on sex, but they do not have to be. ... The African language Supyire from Mali has five genders: humans, big things, small things, collectives, and liquids. Bantu langauges such as Swahili have up to ten genders and the Australian language Ngan'gityemerri is said to have fifteen different genders, which include, among others, masculine human, feminine human, canines, non-canine animals, vegetables, drinks, and two different genders for spears (depending on size and material). 

All of this had me thinking about how we classify, and why, and how our cultural context influences that classification. In UX, we don't worry all that much about spears, but don't we just love the subtleties of whether we describe something as UX, UI, HCI, interaction design, or whatever?

Terms for colors in languages follow a predictable pattern

Let's return to color, which is the major focus of the book. It turns out that there is a sequence of development of terms for colors in languages:

1. All languages have some terms for colors.
   
2. If a language only has two terms for colors then they will be black (or dark) and white (or light).
3. If a language has three terms for colors, they will describe black, white and red.
4. If a language has four tems for colors, they will describe black, white and red and either green or yellow.
5. If a language has a five terms for colors, they will describe black, white, red, green and yellow.
6. Languages only have a term for blue if they have at least 6 terms for colors.
7. After that, it gets complicated.

If you find that fascinating, then this might be the book for you. If you already knew it, then maybe give the book a miss.

An informal style of writing that I found appealing

Here's one of Deutscher's points about the color sequence:

There is undoubtedly something biologically special about our relation to red: like other Old World monkeys, humans seem to be designed to get excited about it. I once saw a sign in a zoo that warned people dressed in red not to venture too close to the cage of a gorilla. And experiments with humans have shown that exposure to red induces physiological effects such as increasing the electrical resistance of the skin, which is a measure of emotional arousal.

I love the throw-away reference to Old World monkeys, and this passage is typical of the way he brings in personal anecdotes, and lightly-done appeals to the research. He offers notes in the back of the book if you want to follow up, but this is definitely not the type of book that makes you negotiate long reference lists or wrestle with p-values.

I started reading it because I rebelled from my usual pile of statistics books, and the way it's written appealed to me so much that I found myself lost in it like a novel, a real page-turner.

It's a survey book because it helps to think about how other people think

Why do I think of this as a survey book? I have to admit it's a bit of a stretch. You won't find anything in here about asking questions, constructing a sample, or dealing with the analysis of a data set.

But it did make me think about how other people think, and why my assumptions about those thought patterns might be wrong. That's a crucial skill for anyone who is doing survey research.

For example, Deutscher points out:

Try to translate a dishwasher operating manual into the language of a tribe from the Papuan highlands, and you will get stuck fairly quickly, since there are no words for forks, or plates, or glasses, or buttons, or soap, or rinsing programmes, or flashing fault indicators. But it's not the deep nature of the language that prevents the Papuans from understanding such concepts; it's simply a fact that they are not acquainted with the relevant cultural artefacts. Given enough time, you can perfectly well explain all these things to them in their mother tongue.

Let's put that into the context of something rather more familiar - for example, the idea of a web site being easy to use. In our culture, by which I mean the one that I hope I share with you as a user experience specialist, we have all sort of different concepts that we bring to that discussion: navigation, interaction, findability, typography, branding, and so on.

Now let's think of someone who doesn't share our UX culture; not someone from the Papuan highlands, but maybe my friend Paul the builder. He works outdoors all day, and uses his computer possibly twice a week to check the TV schedules and follow the news of his favorite sports teams. He'd be entirely capable of grasping the difference between web site navigation and web site branding, but right now if you said 'navigation' to him it means something to do with sailing, and 'branding' is about burning a mark onto a cow.

What happens if a survey pops up and asks him to rate the 'navigation' of that TV site? Cultural confusion, that's what. And definitely not good, useful data.

Recommended for relaxation or to get you thinking about cultural assumptions

Mostly, I picked this book because I found it relaxing and engaging to read. But if it also helps us all to review our assumptions about how other people think, then it definitely counts as a survey book.

1. Questions: Start with the questions but don't stop there
2. Goals: Work out the goals for your survey
3. Users: Talk to your users about the topics in your survey
4. Build: Create and test your questionnaire
5. Deploy: Send it out and watch the responses as they come back
6. Analyze: Investigate the data and create your report
   

1. Questions: Start with the questions but don't stop there

2. Goals: Work out the goals for your survey

3. Users: Talk to your users about the topics in your survey

4. Build: Create and test your questionnaire

• construct your survey invitation and reminders and
  
• think about whether you need a pre-notice.

5. Deploy: Send it out and watch the responses as they come back

6. Analyze: Investigate the data and create your report

Summary: the six steps

1. Questions: Start with the questions but don't stop there
2. Goals: Work out the goals for your survey
3. Users: Talk to your users about the topics in your survey
4. Build: Create and test your questionnaire
5. Deploy: Send it out and watch the responses as they come back
6. Analyze: Investigate the data and create your report

One to borrow rather than buy?

The order of the questions is important

1. Do you think the United States whould let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it? ("Communist" item)
2. Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news at they see it? ("American" item)

"[Context  effects] can be very large [and] are difficult to predict".

Open questions elicit a wider range of answers, but are not as open as they seem

"Inadvertent phrasing of the open question itself can constrain responses in unintended ways ... we can see no way to discover subtle constraints of this kind except by including systematic open-closed comparisons when an investigator begins development of a new question on values and problems".

• Explore your questions in interviews with users,
  
• Test the questions,
• Check the results and make changes,
  
• Test again, and repeat until it all settles down.

"since our results fail to provide strong support for the superiority of open questions, the implication may seem to be that after sufficient pilot work an investigator can rely exclusively on closed items. But we think that total elimination of open questions from survey research would be a serious mistake.... Open "why" questions can also be especially valuable as follow-ups to important closed questions, providing insight into why people answer the way they do... They are needed as well where rapidly shifting events can affect answers, or indeed over time to avoid missing new emergent categories. And of course in some situations the set of meaningful alternatives is too large or complex to present to respondents".

If you offer the option of 'don't know', some people will take it

• "In general, do you think the courts in this area deal too harshly or not harshly enough with criminals?"
• "In general, do you think the courts in this area deal too harshly or not harshly enough with criminals, or don't you have enough information about the courts to say?"

1. Have thought about it and have yet to make up my mind
2. Haven't thought about it
3. Have an opinion but don't want to reveal it to you
4. Sort of remember having an opinion but can't be bothered to recall it.

Sometimes "no opinion" is a valid opinion... and sometimes it isn't

"Respondents make an educated (though often wrong) guess as to what the obscure acts represent, then answer reasonably in their own terms about the constructed object".

"a substantial minority of the public - in the neighborhood of 30% - will provide an opinion on a proposed law that they know nothing about *if* the question is asked without an explicit 'don't know' option".

Balance your questions for more accuracy

The tendency to agree ("acquiescence bias") is not as strong as sometimes claimed

1. It is hardly fair to bring children into the world, the way things look for the future.
2. Children born today have a wonderful future to look forward to.

"Which in your opinion is more to blame for crime and lawlessness in this county: individuals or social conditions?"

Do your respondents care as much as you do, or much more?

"What if the minority prefers its alternative much more passionately than the majority prefers a contrary alternative?"

• Intensity is the subjective strength of the attitude
• Centrality is subjective importance to the respondent
• Committed action is doing something about it e.g writing to your senator.

"people find it easier to say that they feel extremely strong about an issue than that they would regard it as one of the most important issues on which to decide their vote"

Attitudes can be crystallised or wobbly

"Forbid" is not the same as "not allow"

A: "Do you think the United States should forbid public speeches against democracy?"
B: "Do you think the United states should allow public speeches against democracy?" 

And finally: Context effects are a serious hazard

"General summary type questions are especially susceptible to context effects and should probably be avoided if the needed information can be built up from more specific questions"

Key points to take away

1. Start with open questions and test a lot
2. If you want to collect informed opinion, offer a 'don't know' option
3. When collecting attitudes towards statements, try to use balanced questions
4. Ask for strength of opinion as well as direction of opinion

----------------------------------------------------------------------

Transcript of "Yes Prime Minister" where Sir Humphrey demonstrates acquiescence bias.

Sir Humphrey: "You know what happens: nice young lady comes up to you. Obviously you want to create a good impression, you don't want to look a fool, do you? So she starts asking you some questions: Mr. Woolley, are you worried about the number of young people without jobs?"
Bernard Woolley: "Yes"
Sir Humphrey: "Are you worried about the rise in crime among teenagers?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think there is a lack of discipline in our Comprehensive schools?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think young people welcome some authority and leadership in their lives?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think they respond to a challenge?"
Bernard Woolley: "Yes"
Sir Humphrey: "Would you be in favour of reintroducing National Service?"
Bernard Woolley: "Oh...well, I suppose I might be."
Sir Humphrey: "Yes or no?"
Bernard Woolley: "Yes"
Sir Humphrey: "Of course you would, Bernard. After all you told you can't say no to that. So they don't mention the first five questions and they publish the last one."
Bernard Woolley: "Is that really what they do?"
Sir Humphrey: "Well, not the reputable ones no, but there aren't many of those. So alternatively the young lady can get the opposite result."
Bernard Woolley: "How?"
Sir Humphrey: "Mr. Woolley, are you worried about the danger of war?"
Bernard Woolley: "Yes"
Sir Humphrey: "Are you worried about the growth of armaments?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think there is a danger in giving young people guns and teaching them how to kill?"
Bernard Woolley: "Yes"
Sir Humphrey: "Do you think it is wrong to force people to take up arms against their will?"
Bernard Woolley: "Yes"
Sir Humphrey: "Would you oppose the reintroduction of National Service?"
Bernard Woolley: "Yes"
Sir Humphrey: "There you are, you see Bernard. The perfect balanced sample."

• My visits to survey methodology conferences
• Survey methodologists run pilot tests
• Survey methodologists do cognitive interviewing
• Introduction to usability testing at SAPOR
  

• The European Survey Research Association conference, where I co-presented with Richard Coombe of Kantar Operations on Using EEG in a usability test of a survey
• The 5th Internet Survey Methodology Workshop, which I attended at Statistics Netherlands in the Hague. We were given a charming and inspirational book about Dutch statistics that I chose as my August Book of the Month

Survey methodologists run pilot tests

• starting with any prenotice, then
  
• the invitation (and any reminders), 
• collecting the responses,
  
• dealing with any issues that arise during data collection,
• cleaning the data,
  
• analysing it, and
  
• producing whatever report or other output you want (I discussed reporting when talking about my August Book of the Month)

Survey methodologists do cognitive interviews

Survey methodologists are becoming enthusiastic usability testers

• Coverage: although most people have Internet access, there are still important populations who do not, or who prefer not to access surveys online.
• Sampling frames: there are comprehensive directories of street addresses, and of landline telephone numbers, but no easy way of getting hold of a statistically robust sample of email addresses.
  
• Mode effects: a human interviewer asking a question is a different mode to a computer asking the same question. People behave differently when interacting with another person compared to interacting with a computer, and their answers differ as well. 
  

Introduction to usability testing at SAPOR

Your survey is judged by how you report

• made sure that you had clear objectives
• thoroughly understood the target audience you want to reach and what they want to tell you
• produced a clear, concise set of questions
• built a questionnaire that looks appealing and works like a dream
• tested the heck out of everything
• got those questionnaires to the right users and had a wonderful response
• worked out what your collected data is telling you

• get the results to the people who need to act on them.

Great reports have great information design

• a headline that summarises the topic
• a graph
• three or four crisply written paragraphs of explanatory text
• a single illustration, photograph, or key quote from the text

Lessons for reporting

• How can I sum up this result in a four- or five-word headline?
• Can I show this data in a simple graph?
• Can I cut the explanatory text to a maximum of four paragraphs?
• What illustration, photograph, or key quote will grab my readers' attention here?

• An instructor asks a class to respond to a feedback survey about the quality of the teaching: person-to-person, word of mouth
• A university sends a letter to a sample of individuals to ask them to respond to an academic survey that will be sent separately: organization-to-person, paper letter
• A national statistics organization runs a series of TV advertisments to alert the general public to a census: organization-to-person, broadcast. The UK 2011 Census campaign starred a purple bus with the slogan "Help tomorrow take shape".
  

The Dillman recipe for a prenotice letter that increases response rates

• Use the headed notepaper of a respected institution e.g. a university
• Write to a named person, or failing that a specific address
• Appeal for help with a specific project
  
• Describe what will happen in general terms
• Say that a small token of appreciation will be offered (but not exactly what it is)
• Say how much effort will be required of the respondent
• Close with the real signature of a respected, named individual
• Provide the full contact details of the named individual
• Send the prenotice letter about a week before the invitation.

• Offer the opportunity to receive the survey in alternative formats, such as large print for a paper survey.
  

A prenotice letter that didn't work

What about prenotices for Internet surveys?

How to make an in-person prenotice more effective

• Finish a little early, so that attendees have some time to spare (reduces effort)
• Explain why you, personally, value the feedback (creates a reward - be nice to the speaker)
• Make sure that the questionnaire can be handed in anonymously, and carefully avoid reading the responses in front of other respondents (increases trust).

• Get someone other than the speaker to come along and advertise the survey.
• Give out a handout with key details of the survey: the link, or the name and email address of the person who will send the link.
• Consider making the handout something that's an incentive in its own right: maybe a pen. Or adapt an idea from Dillman and give people an envelope printed with the details and with a dollar bill inside. I've even wondered if it might work to adapt an idea from the US Census and hand out mugs.
  

Share your experiences

References

The untrustworthy invitation

• It's not addressed to me. It's addressed to #fullname#.
• The incentive seems rather high for a survey. 25 euros (approx US $36) guaranteed? For 10 minutes?
• And a bit further down, not shown in this screenshot, there was an address of a US corporation but the contact email address was in the Netherlands.
  

Response relies on trust, perceived effort and perceived reward

A reminder that looks like a reproach

Recently, we sent you an invitation to complete a Guest Satisfaction Survey concerning your stay with us at XXX YYYY, where you checked out on August 3, 2011.

We noticed that you did not have time to complete the survey. We are concerned that you may not have responded because we have somehow failed to live up to your expectations. At XXXX, we are committed to providing a superior guest experience to every customer.

Please take a few minutes to tell us how well we met your expectations. To complete the survey, please click on the web address below. If that does not work, please copy and paste the entire web address into the address field of your browser.
http://survey.xxxxxx
Thank you again for choosing XXXX. I look forward to hearing about your stay with us.

Sincerely,
Dave ZZZZ
Manager,
XXXXX Hotels
------------ Please do not 'Reply' directly to this invitation.

Technical assistance: Should you have any problems accessing or completing this survey, please e-mail our survey vendor at hhctechsupport.xxxx.com

To unsubscribe: We rely on feedback from guests to ensure that your hotel stay meets and exceeds your expectations. If, however, you prefer not to receive a survey invitation in the future, you may unsubscribe by clicking this link: http://survey.xxx

For all other requests, please visit http://www.xxxx

At XXXX, we are committed to providing a superior guest experience to every customer.

I look forward to hearing about your stay with us.

Please do not 'Reply' directly to this invitation.

A survey invitation has three purposes

1. Offer a preceived reward.
   
2. Explain the amount of effort.
3. Inspire trust.
   

Offer a perceived reward

Explain the amount of effort

1. The possible benefit from saying how long a survey is, either in minutes or as a number of questions
2. Widespread cynicism about the value of these promises.

Features that inspire trust

Design your site so it looks professional (or is appropriate for your purpose).

Tips for better invitations and reminders

• Look first at trust. Have you made it clear who you are? Is the person likely to know you, and if not have you made it easy for them to find out that you're trustworthy? Does your invitation look as carefully considered and designed as the rest of your brand?
  
• Then check perceived reward. Why would this person want to be nice to your organization in the context of this request?
• Then check perceived effort. What will the respondents get in return? Is this all about you, or is there something in it for them?
• And finally: Test! Test! Test! Make sure your testing includes the invitation and any reminders. Don't let silly mistakes like sending your invitation to #fullname# ruin your survey.
  

Usability testing of individual questions

Using EEG to get insight into overall UX in a survey

ESRA presentation: using EEG

"Respondents can complete the questionnaire when it suits them"

"Assumes respondents have answers available in an organized fashion".

1. Take each person's response in turn.
2. Go through each one, highlighting substantive statements - the statements that make a key point, that really say something. In written form this will be ture of most of the material, so your job is to separate them out. However, when you are working on the transcript of a recorded interview it is more difficult because speech has a lot of 'redundancy' - repetitions, irrelevant material, digressions of one kind or another.
3. As you mark out the substantive statements, you will find categories forming in your mind. Note this process but don't do anything about it at this stage.
4. Take a break, reflecting on what you have just done. This is to give your impressions a chance to settle, without becoming fixed.

Aimed at academic researchers

Does not include anything about the web

Plenty in it for the UX practitioner