5 Simple But Powerful Airtable Features

So I wrote/recorded a whole IG series about my love for Airtable and I decided to put it all together and post it as one big post to make it easier to find and watch.

So Exactly What Is Airtable?

 

View this post on Instagram

A post shared by Karyn (@hey.karyn)

 

1. Column Data Types

View this post on Instagram

A post shared by Karyn (@hey.karyn)

2. Multiple Select ❤

View this post on Instagram

A post shared by Karyn (@hey.karyn)

3. Forms!

View this post on Instagram

A post shared by Karyn (@hey.karyn)

4. Checkboxes ✔

View this post on Instagram

A post shared by Karyn (@hey.karyn)

5. Grouping & Filtering

View this post on Instagram

A post shared by Karyn (@hey.karyn)

 

 Let me know if this was helpful and if you have any Airtable questions!! Thanks!

Tips for Selecting Customers to Interview

 

You Want Diversity of Perspectives

Whenever possible, you want people with different perspectives, experiences and goals. This gives you the widest range of stories for others to connect with if you are using your research for writing copy and for insights and feedback you get to hear from people who are using your product/service (or not using) in different ways for different reasons. You aren’t just hearing from the loudest fans in the front row who love everything. So whenever possible it is important to look for a diverse group to interview.

1. What does that look like?

It looks like interviewing several of your biggest clients/users who are your ideal clients you wish you could clone. These are the people who you believe are in the ideal situation to hire your business and get the most value out of it. AND maybe they are just a pleasure to work with and have good manners that you expect from a client/customer.

It then looks like talking to people who maybe left a program early, or went quiet. It also can look like people who were super interested in your business (perhaps they open every email and have purchased smaller offers in the past) but never actually hire you. Or maybe they used to be engaged but suddenly are not. Finding out what WASN’T working for these people is extremely helpful.

2. Other People to Look For

 

People at different levels of involvement in your groups. Introverts/Lurkers have opinions.

People who you know have different life experiences/perspectives. Example: If your target customers/members are moms with kids. Talking to moms with different family structures/dynamics can help identify truly common threads, help identify areas where you could provide more support, and if you are using these for testimonials, provide diverse examples from different types of situations to appeal to more new customers.

Do not be afraid to talk to people who said no.

People who have experienced different parts of your services.

3. Make a Ranked List

 

Start with your ideal customers (who have actually paid your business money) at the top of the list and then start adding in the other types of customers. The more interviews you do the more diversity you can achieve in the data you gather.

4. PLEASE NOTE: Research Fatigue

Pay attention to whether or not you are asking the same group of people for feedback in a relatively short time frame, or asking them the same questions. It’s one thing to follow up on a questionnaire or survey with a more in-depth interview (GOOD). It’s another to send them the same questions over and over again for no reason. Point = Keep track of who you are talking to. (This is where awesome database tools like Airtable.com come in handy.)

 

 

Why Jobs To Be Done?

 Have you ever heard of the phrase, please describe your ideal client? Did you panic?

Did you freeze or did you say something like, well, she’s 35 and she drives a minivan and she really likes red and she shops at Barnes & Noble or something like that. Which is great, but none of those things have anything to do with why she was your ideal client. Right? 

I want to talk today very briefly about the concept of Jobs To Be Done.

 

There are a lot of great resources out there, but I just wanted to give a short and simple description of how it works for people who weren’t really ready to go down the rabbit hole yet.

 Why should you care about Jobs To Be Done?

 Basically it can help you determine why people are hiring your service or not. And it’s just a unique way to go about getting to know your customers and to understand how you could best serve them.

 I’m going to give you a brief overview of the concept and then I’m going to share an example to just help explain it a little better. And then I’m going to share resources because I have a whole lot of links. All of which are free for you to read if you really do want to go down the rabbit hole with me because it’s great, you should come.

Customer Situations and Progress

The first mistake is that we try to use the descriptions of people as a reason for why they buy something. So for example, just because I drive a Honda Odyssey (sliding doors, woohoo) and I also just bought a whole bunch of balloons at party city last week, doesn’t mean that I bought those balloons because I have a minivan. This is the whole concept of correlation does not equal causation. To pull out my nerd glossary and drop a little jargon on you.

So I bought the balloons because my son had a birthday last week and we like to decorate the night before so he can wake up to a happy surprise on his birthday. So I actually bought the balloons for the look on his face on his birthday. Right? That’s really why I bought them. That has nothing to do with the fact that I drive a minivan.

Clayton Christensen, who is one of the pioneers of Jobs To Be Done said this, “What they really need to hone in on is the progress that the customer is trying to make in a given circumstance. What the customer hopes to accomplish. This is what we’ve come to call the Job To Be Done.” So the key here is the job is talking about what someone is really hoping to accomplish, what progress they’re trying to make in the situation.

And one aspect, I don’t want to get too far into it, but jobs themselves don’t really change over time. We just come up with better solutions for them. For an example, if a job is to get me from point a to point B with as little effort as possible, past solutions include horses, rowboats cars, trains, taxis, Uber, driverless cars.

An Example

So what I love about Jobs To Be Done is that it allows you to approach getting to know and understand your customers by taking into account their motivation, the context of the situation and their emotions. And yes, this does involve talking to them. So you approach it by looking at what situation and the context they were in when they bought or hired your service and also when they fired it. So here I’m just going to use myself and my business as an example. Like what’s an example of a Job To Be Done? Right. All right. So what I do is I help small businesses by interviewing people’s clients and extracting information from those interviews for different purposes.

People come to me and for lots of reasons like to get testimonials to get an external perspective on the value of their business from somewhere other than their own brain. And how do I know this is because they talked to me when we get on the phone and I take notes and I pay attention to why they’ve said they needed to hire me or possibly might need to hire me. And so these are both the beginning of possible job descriptions that my clients need help with. So we’re going to look at the desire for testimonials.

What Caused Them To Act

First, what situation are they in that caused them to act? They’ve been busy serving clients and they haven’t had time. Right. But what happened to make them notice? They didn’t just out of the blue, all of a sudden it became really important for no reason.

So what were the events that could have triggered them noticing that they needed testimonials? Well, they might have been doing a website update and they needed to update their website and they’re like, oh my testimonials are like two years old. I need new testimonials. Or they might have an event or a speaking gig coming up and they know that a lot of people are going to be looking at their website and they just want fresh content and fresh testimonials. Or maybe they don’t have any on there at all. And they might be launching a product or service and they might need, they might be wanting to get testimonials from the people that are about to serve or they might want new testimonials for copy for selling launch. Another reason was they might be pivoting and they might realize that all of their testimonials are talking about an old service or an old product and they don’t have anything talking about what they’ve been doing for the past year.

Right? So those are events that can trigger them noticing that they don’t have testimonials. They’ve got this new deadline that made them aware. Why is knowing the triggering event useful? Because it tells me when people might be looking for a service like mine, which can inform my marketing and help me understand what frame of mind my potential clients might be in when they’re looking for help.

So if we’re going to try to define their Job To Be Done for this situation, one way you could say it would be that my customers want to show their potential customers they can be trusted so those potential customers will feel comfortable hiring them.

Jobs Stay the Same

So a couple other things about “Jobs”.

They don’t necessarily depend on the solution. The job stays pretty constant over time. Like I mentioned with the point A to point B example earlier and they don’t really include activities. It is more about the progress they made or want to make versus the way you make the progress. And you can’t just have a job be describing what you do in your business. It’s about them not about you. And your business might do more than one job. I know mine does, so that’s totally common.

Jobs Are A Starting Point

All right, so now what? We’ve got this figured out. What do I do with this information? You can do a ton. This is really the starting point that you can build off of and learn from and one way of thinking about it is that the job becomes the higher level mission. How can I be in service of this job for my clients?

How can I have alignment in my business so that it is all happening. The marketing, the actual product delivery, the follow-up, the whole thing. How is it in service of doing this job and letting them know that that’s what I’m trying to help them do. So you have to examine your business through the lens of this job.

Another tool is called a job story, which I’m going to share a great article about by Alan Clement over at Intercom. And you can read all about it. I can’t go into it here because this video would never end. But I wanted to point out that you’ll notice the only descriptor I’ve used to define my clients is that they’re small business owners. My clients have all different kinds of businesses, from agencies to accountants and all over the place. So it’s not about the type of business for me and none of that matters because that’s not a defining characteristic and it doesn’t have anything to do with whether or not they would need my service.

Next JTBD Video

So I just wanted to do a quick overview there of the basic concept. There is much more to learn. For example, one of the things I’m excited to talk about in another video – I love the way that Jobs To Be Done allows you to examine who your true competition is. To me when I learned that it kind of just, it was big, it was a totally different way of thinking about your competition. So, thank you for watching. If you have any questions, you can reach me at heykaryn.com I’d be happy to answer any questions. Seriously I could geek out on this all day. If you found this content valuable, please like and share and I’m going to have posts or links to recommended reading and articles mentioned and other ones. Thanks and have a great day!

Want to Read more about Jobs To Be Done?

Read These:

Using Jobs To Be Done For Copywriting

New Book: Intercom on Jobs To Be Done

Customer Interviews: Voice of the Customer and Jobs-to-Be-Done

Know Your Customers “Jobs to be Done” (HBR – linked above)

Designing Features Using Job Stories (linked above) 

Happy Reading!

What Is a Leading Question?

It’s a question with an opinion.

We break down leading questions in this week’s rant. 

• Why they are bad for learning what people really think
• How they cause suspicion about your motives and erode trust
• The key questions to help you identify bad leading questions
• We walk through how to fix a real life leading question
• Examples from: Client intake questionnaires, corporate employee surveys, community surveys and more…

This is a long one folks. (Just treat it like a podcast and you’ll be fine. 😉)

via GIPHY

Full research nerd level this week.

But if you are serious about writing surveys or interviewing customers, community members or doing any research with people, you need to know a leading question when you see one – and know how to avoid writing them. If you want to get better at spotting them in the wild, this is for you too. You’ll be amazed at how many leading questions are out there.

The Transcript

Karyn:                   Today’s video is not going to be short. I’m just putting it out there. If you were expecting      something in the 10 minute           range,  that’s not going to be this one because we have big nerdy subjects to talk about today. We’re going to talk about leading questions and just to give you a quick description. A leading question is a question with an opinion. It has a position. Leading questions are not inherently evil, but they can be very problematic. They show up everywhere. In surveys and interviews and regular conversation, in political polls. I’m here with Maggie, of course, my fellow nerd buddy. Why should our viewers or listeners care about leading questions?

Maggie:               What a good question. And like Karyn said, this is not going to be a short video because there are lots of reasons to care about leading questions. Number one, because they give you bad data. Leading questions bias responses and biased responses aren’t usable and they’re not useful. So you’ve just asked a question and because it’s crap, you have to throw away all the responses. So that’s a bummer. Number two, you lose trust when people feel like something is being prescribed to them either in the way that the question is asked or in what the potential responses are.

Maggie:               For those of us who are maybe a little bit conspiracy minded, it tips people off a little bit to think like why are they asking this? What is the purpose behind this? Will this be used in some kind of nefarious way that I don’t consent to? And third, you should care because you get asked a leading questions all the time. And Karyn just shared some great examples of the kinds of places you might be asked leading questions like, political polls, poorly written surveys, twitter polls, the Internet in general, and to be able to spot these in the wild.

Karyn:                   So that is why you should care. Because if you’re doing this and you think you’re getting unbiased data, you’re not. So you’re wasting your time. And it makes you look bad to everyone you’re trying to survey and we want to help you be able to recognize them because they can be sneaky. Not all leading questions are blatant. Some of them are sneaky, they fit in emotional words. We’ll get into the nitty gritty of that, but we want to help you recognize them, so that when you’re writing your own you can give yourself a checklist and say, am I leading everyone to a certain answer and I don’t mean to do that. And when you’re seeing other ones in the wild and you recognize them.

Karyn:                   Who are we and where can you find us? I am Karyn Kelbaugh and you can find me at heykaryn.com. I help small businesses with customer research and I help nonprofits with evaluation, which is another fancy way of doing customer research.

Maggie:               I’m Maggie Hodge Kwan and like Karyn, I work with nonprofits and philanthropic foundations around their evaluation efforts and other projects as needed and you can find me at www.creativeclarity.ca.

Karyn:                   So let’s get into the fun stuff. Leading questions, which again, they are not inherently evil, but like if you’re designing a conversation. Let’s think of a few examples where they’re fine or they’re innocent, like you’re having an interview with somebody and say you’re a podcast interviewer and you’re interviewing someone and they have a really exciting story to tell. And so you say, what was the most exciting thing about that? And you’re thinking, well, I’m assuming that it was inherently exciting and I am leading you to tell me something exciting. So it’s more of a guiding question that’s helping the interview along that you’ve already agreed to do. Right? So it’s less about, I want your unbiased opinion of what happened. We’re telling a story together and I’m helping guide you along in that story. That’s one example where it’s fine, it’s harmless. You’re not necessarily doing damage by guiding the questions.

Maggie:               Yeah, yeah. You might be in the checkout line at the grocery store and the cashier might say, “Are you enjoying this beautiful weather?” when the sun is shining after days of rain. It’s not a harmful question.

Karyn:                   So it’s not like every time we should say “Hey! That was a leading question. You should ask me how do I feel about the weather today?” You can’t. You can’t call them all out. They’re not all bad because that cashier is likely not trying to collect unbiased data for a report. They’re just making conversation. So there are plenty of situations in our daily lives where we use leading questions in an innocent way and not in a nefarious and sneaky way. Now granted you can use them with your kids. Like “Are you sure…” I’m trying to think of a good one.

Karyn:                   Something about brushing teeth and eating vegetables. I’m sure there’s a million of them that I use probably every day. Are you sure that you want to say that? You said you brushed your teeth? And they look at you and say “I’m gonna do it again just to be extra sure I got them all.” So they can be totally innocent so we don’t want to just bad mouth anytime a leading question gets used.

Examples of Leading Questions

Karyn:                   But let’s go through some examples of leading questions that might not be the best use of a question.

Maggie:               And we have some fantastic examples that we’ve put together from our own experience and from all the corners of the Internet. Karyn, do you have a favorite you want to share?

Karyn:                   We were doing a little googling around and seeing what we could find to find some really good examples. And there was this one. I kind of want to save it. Let’s start with the, “do you believe in yourself” one first. We tried to get some examples from different contexts, like from like a business to business relationships or corporate and employee relationships. Political polls. I don’t know if we actually pulled one from a political poll, but we have a good political one and then just some other ones from different contexts that we wanted to pull together.

Karyn:                   So one that we got, and this was a real one pulled from the audience that people had seen on a coaching intake form or something like that. And it was “Do you believe in yourself enough to invest in your business and success?”

Karyn:                   I have feelings about that.

Maggie:               Lots of feelings.

Karyn:                   This one was kind of shows up as a double (barreled questions) too, doesn’t it? Because they’re saying, do you believe in yourself? And, are you willing to invest in your business? It’s a solid guilt trip.

Maggie:               Yes. Right up front. It’s telling you, you joined this coaching program and if you don’t see the results you want or if I as a coach don’t deliver, I have an automatic out. I’m going to say that it’s because you didn’t believe in yourself enough.

Karyn:                   My first gut reaction to that question is I do believe in myself enough to invest in my business but not with you because you’re being manipulative, right? That’s a solid example of okay, I have more questions for you because this feels slimy.

Karyn:                   Another example that we got from a corporate environment where there’s an employee and it’s a corporate survey trying to gauge how the employee feels about his or her contribution to the company. And the questions was “I know how my everyday actions contribute to the company’s overall goals.” And you’re thinking that doesn’t feel too bad? Right? I either know, or I don’t. But who’s going to answer this honestly? This is your boss reading it. Who’s going to put “I have no idea how my actions tie in to the company’s overall goals. Why am I here?” There’s a lot of subtext in this question. “I don’t know my purpose in this company. Why should you keep me?” Oh, we should keep the people who know why they’re here. It’s got a lot of subtext.

Maggie:               The whole time you’re crossing your fingers hoping that it’s anonymous and there’s no IP address connection or anything like that.

Karyn:                   Is this linked back to my desk? And that goes into the whole thing called the Hawthorne Effect, which is that people act differently and respond differently to questions when they know they’re being watched. You know, your boss who’s in charge of your employment is reading this survey and so you’re going to have a natural tendency to people please and to do whatever. Go along with it. You put “yes, I absolutely know how my everyday actions tie into my company’s goal. That’s a ridiculous question. Why would you even ask that?” There’s that people pleasing element that’s already there and have the questions that are pushing them in a certain direction and then they get data. And even if everyone answer that question honestly, this is the part that kills me. Even if everyone in that company answered that question honestly, that data is still not helpful because they don’t know that. I mean, there’s always going to be a few people who put C C C right?

Karyn:                   But there’s no way for the people administering the survey to really know that that’s the truth, because there’s that effect of people pleasing and they just want to look good for their boss.

Karyn:                   Do you want to read another example?

Maggie:               Yes. I have some examples up in front of me. One that I think is great for its simplicity is a survey question that says “You don’t smoke, do you?”

Karyn:                   You don’t do that do you? There’s no judgement in that questions at all! Not any bit of judgment. OMG, so judgy.

Maggie:               When it would be so easy to redesign a question to find out if people smoke or not by just asking them if they smoke. Yes, no, sometimes, blah blah, blah.

Karyn:                   Have you smoked in the past 30 days? Yes, No. Ok. Thanks. Super easy!

Maggie:               Without judgment.

Karyn:                   And this is on a real survey?

Maggie:               Yes And of course when you say “you don’t smoke, do you”, you’re judging people, and shaming people from the get go. And if for some horrible reason you’re the survey giver and you’re there. Hello Hawthorne Effect. Like Karyn mentioned, of course you’re going to be like, nope, no, I don’t smoke. “You don’t drink, do you?” Nope.

Karyn:                   It’s like when there’s that bias and judgment and then bullying in the question, it’s clear what answer they want you to give. Then this might be a leading question. What else do you got?

Karyn:                   I have another good one. Again, this one you should be able to spot from miles away. It’s a question that says, “I assume you would agree that teachers do a heroic job for our children.”

Karyn:                   And the answers would be something along the lines of strongly agree, agree, also agree, completely agree. Those are hard because you read that and you’re like, of course I agree. It’s almost like they get into this ridiculous redundant self pleasing question kind of thing where, well yeah teachers are heroic, but why are you even asking that question?

Maggie:               Yes. And that’s exactly one of those times where this red flag raises. Okay, so who is asking this question for what purpose? What are they doing with the data? And don’t get me wrong because I truly do believe that teachers are heroic and do a wonderful job of teaching children. Even I who loved teachers would look at that and be thinking like, is this from a lobbyist, a union that wants to say that 100 percent of Americans agree that teachers are the nation’s true heroes. Like how is this going to be used? And why are they asking it that way?

Karyn:                   Yeah. Like when, when there’s inflammatory language in there, one of the things that we were talking about earlier was how sometimes really people aren’t actually out to collect the data. You will get a survey or a phone call that’s literally just trying to sell you something and they’re covering up their sales call or their pitch by pretending to ask you questions about the topic so that they can tell you about their product. So there’s that. And then another one was an example of political parties or groups or community groups trying to fire up their base of people by sending out questions that use inflammatory language that they know they’re all going to agree with. And then they’ve got data to feed back to them and it’s just, they’re using it as a hype machine almost. And with all that language in there.

Maggie:               And do we have a good example of one of those questions?

Karyn:                   I have to read this in my best southern lawyer voice. I’m sorry. I love lawyers. I love people. I live in Texas, but I just pictured this in a courtroom setting with somebody from a movie. Please forgive me. See if you can spot all the inflammatory language. There’s emotion, there’s extra adverbs and adjectives that don’t need to be in there. Okay. “What should be done about murderous terrorists who threaten the freedom of good citizens and the safety of our children?”

Karyn:                   Yes, every single adjective builds and builds.

Karyn:                   And you’re just like, whoa. That was loaded and you know, it’s just, it’s just a good example of they’re leading you to a certain answer. Does this questions even need to be asked? Of course we don’t like terrorists. They cause terror, okay. We don’t have to do this thing where we build in and have this question that’s just leading people in a big direction. So those are some big outrageous examples so that we could kind of show you the different gamut of those.

How to Spot Leading Questions

 

Karyn:                   But the next thing that we wanted to talk to you about was how to identify them. So we’ve just given you some good examples of outrageous questions. But they aren’t all outrageous. We came up with a quick two question filter or a checklist. Maggie, would you like to talk about that?

Maggie:               Yes, I would. So super easy. A couple things to keep in mind when you’re looking. So if you read a question and you feel kind of unsure about it or maybe something to ask as you read any questioning and get more familiar with this.

Maggie:               First thing to think: Is this question seeding a particular response? So with the one that Karyn just mentioned about terrorism, the question is clearly built to get you to say yes, terrorists threaten the safety of our nation and our citizens and our children. Like she said, does that question need to be asked? When I think probably we’re all in agreement that terrorism is not good. And like Karyn said, probably that was designed to elicit a strong response to feedback to the people who feel that way. So is it seeding a particular response? That’s the first question.

Maggie:               And the second: Is the purpose of this question to gather unbiased data? So, by saying, “I assume you agree that teachers are heroic” or something along those lines, we know just by looking at that question statement that it is not designed to collect unbiased data. A question that was designed to collect unbiased data, would lose the word heroic, would ask for your perspective or feelings, would give lots of options or would leave an open ended question where you could share your own feelings and perspectives about the role of teachers.

Karyn:                   I was thinking the other day about another way of approaching whether or not something is a leading question. Especially if you’re writing your own and most of us, I hope are not going to have these dramatic goals of really wanting to get this one response out of people. Because we want to actually learn from our clients and our community members.

Karyn:                   Would the person responding, feel comfortable giving any answer to this? If they gave an answer that they knew wasn’t maybe a mainstream response. Would they feel comfortable based on your question giving any answer that they felt compelled to give without having to get up the courage to say, well actually no. Do they have to give qualifiers before they answer it? But in general, do they feel comfortable giving any answer to that question? To me also look at it from their perspective on the answerer side and not just the question asker side.

Maggie:               So if we think about that heroic teachers example, if you were given a statement that says, “I assume you would agree that teachers do a heroic job for our children” and your response options were to agree or disagree, you’re kind of bullied into agreeing no matter how you feel versus, you know, if you’re asked a non leading open question about teachers where you can say, “The public school teachers I had growing up were wonderful influences on me. I appreciated their energy and enthusiasm. I have some concerns about my children’s teachers because blah, blah, blah, blah blah. But overall I’m, I’m grateful for them and thankful that my kids are happy at school” or something. You’ve got the full nuance of a person’s own experience rather than a person sort of having reservations but being goaded into saying I agree because that’s the road they’ve been led down.

Karyn:                   Because they’ll look bad if they don’t. Especially if any of these are in a public setting where you have to answer in front of other people. Social pressure is real. I clearly have to give this answer because I don’t want the backlash of not. And even though I have all these other things I would like to say about the topic, there’s no spot for that.

How to Fix a Leading Question: Example Walk-through

Karyn:                   So how do we improve? We’re going to go ahead and give you another example that is a real example from real surveys that currently exist in the world. And that’s all you need to know. So this one would be an example of a community survey for youth about alcohol and drug use. So these are really common. They are all over the country, communities all over the country use these types of surveys to try and see how young people feel about alcohol use and all that kind of stuff.

Karyn:                   So one example of a question that is absolutely a leading question. “How wrong do you think it is for people your age to drink alcohol? The response choices for this question, are “not wrong at all”, “a little bit wrong”, the neutral option, “wrong” and “very wrong” See if you can spot the bias in that question and those responses. Do you see anything wrong with that?

Maggie:               I can’t even believe there’s a neutral option. I thought you were going to say medium wrong.

Karyn:                   Maybe there’s just like wrong, very wrong, super wrong.

Maggie:               Completely wrong.

Karyn:                   An undecided. I’m sure it’s undecided, the good old standby of undecided.

Maggie:               So from the hopper, it has a judgment, right? How wrong do you think this is? Well, I guess I know it’s some kind of wrong.

Karyn:                   And then you’re the kid and this question is being asked of teenagers. And so, I feel for the people who wrote the question. These are people who are trying to honestly get young adults and teenagers to not drink alcohol. Like that is their goal and their mission. So to them it feels completely wrong to ask a question that might give the perception at all that they’re okay with alcohol consumption. I understand where they’re coming from when they wrote this. However, it is a totally biased question because it is putting judgment on the kid immediately.

Karyn:                   Even if they have a slightly different opinion, the furthest away is saying “not wrong at all”. That’s the furthest they go away from wrongness.

Karyn:                   So now we’ve got a nice solid example of a leading question. How do we fix it? How would we go about improving it? I was just going to go through some of our ways of looking at it. And then we’ll go ahead and apply those to the question and we did this ahead of time because it took some thinking.

Maggie:               Well before you even do that, Karyn, can I say I appreciate you talking about the intent of the people who wrote this survey. Again, I just wanted to point out, it goes back to what you were saying. Sometimes it really is egregious. Sometimes the person writes a leading question with the intent to get a politicized or a response that leans heavily in one direction or the other.

Maggie:               In this case, it’s a leading question without that intent, but with some blinders on around potential responses and actual youth perspectives. So not villainizing this question or the people who wrote it, but just noting as you are working through your own surveys, be aware of blind spots in your own thinking.

Karyn:                   And it goes back to one of our little tests. Would a teenager feel comfortable, I mean they’re teenagers so they do what they want. But for the ones that might care what you think, are they going to think, well everyone else is going to put this. They have that extra social pressure and not that anyone is going to know what their response is, but they might think their teachers are going to read it. So that all plays a part, anonymous or not. Would they feel comfortable giving whatever response based on the question alone.

Karyn:                   So how to improve these. That is an excellent point about the purpose. Most leading questions I feel happen because we’re trying to get a certain answer and we just asked the question that way and we don’t think about it. We don’t stop to think about it , we’re not all malicious people.

Karyn:                   How to improve these. Identify the words or the phrases that are biased. So using adverbs, adjectives, one sided adjectives, like only using wrong and words that are like wrong and wrongish and wrongness and heroic, super, all those extra adjectives. Emotional language, words that imply judgment, and ensuring that the response options are also not leading. So you could have a totally fine question and then might actually have leading responses so they can go together and work together or they can work against each other and mess up your question. So just going through our question and right off the bat, WRONG. I would circle the word wrong because that implies a judgment and that’s our first problem with that question.

Karyn:                   Identify the purpose of the question. This is the point of all of these videos that we’re doing.

Maggie:               I’m fully with you on the purpose of the question. And I think that people go so sideways in survey writing here that they lose the thread on the purpose or they try and dress it up or make it academic. But what are you actually asking people? What information are you trying to get out of them? And then how can you ask that in a way that makes sense, doesn’t lead, isn’t loaded and still provides you useful information? And yeah, this is a rant for a whole other video. But if you can’t justify why you’re asking a question or what you’re planning to do with the responses, please don’t ask it.

Karyn:                   I have a budgeting wizard friend that talks about everything you spend money on needs to earn a spot on the budget. Every question needs to earn a spot on the survey or in your group of questions, whether it’s a survey or an interview or whatever it is. It needs to have a purpose and you need to be able to say what you are going to do with that information. So yes, we could soapbox this all day, but that happen in another video. So “it’s nice to know” is not a reason to ask a questions because that just means that no one’s ever going to do anything with it and now you’ve wasted everyone’s time. So knowing the purpose of the question can help you. We had to drill down the purpose of this “how wrong” question. Now I don’t know what the original intent was, but based on my experience with this question, the main purpose is to know what young people think about alcohol use by people their age.

Karyn:                   So why don’t we ask that question? I have my handy dandy notes here because we wrote them down. So we thought, okay, what if we rewrote it so that’s the question we asked. What do you think about alcohol use by people your age? It feels like it could go either way. I don’t feel immediately pushed. And it’s hard to eliminate every ounce of bias, but you can make it as neutral as possible.

Maggie:               Yes. You are leaving room for someone to say, I think it’s perfectly okay, or for someone to say, oh my goodness, that’s awful. Or really anywhere, any of the gradients in between.

Karyn:                   This next one was tricky because you could leave it open ended, but some of these surveys go out to thousands and thousands of kids. That’s a lot of data. So if we were to replace the response options, that all said some version of wrong, how would we do that? And so another way to help improve your question is removing and replacing the biased words and phrases with neutral language to create more balance. So we were able to totally eliminate the “wrong” word from the question, which was awesome. We just took it away. We didn’t have to replace it with anything other than just “think”. But if you’re giving a range of options, you have to put something there. And so it’s about creating balance and so it’s not just wrong and not wrong. What are words that are less judgy than “wrong”? And there could be a lot of different ways to do this, but the ones that we came up with were “harmful” being on the one end and for the other end we thought it can’t just be safe.

Karyn:                   What’s an actual antonym or a balance word for harmful? And the one we came up was “beneficial”. I don’t know how many times I’ve seen a study talking about the benefits of alcohol. There’s data out there (good or not). For the record, I’m not arguing about the benefits of alcohol at all. I’m just saying this is a way to approach it. There’s stuff out there saying that it’s beneficial and the opposite of harmful would be beneficial. Now I know the folks doing the survey are probably not going to put beneficial on the survey for kids because then the parents would get mad. But anyhow, we were trying to fix the question.

Maggie:               When we talked through it, we were looking at what response the survey writers were looking for and what kinds of information could they harness from these potential responses. And so, it’s pretty obvious that if a large contingent of youth say harmful, you know where they’re sitting. But what about kids who might say beneficial and who are thinking, “I’m super stressed from trying to keep up a high GPA and get into a good school and whatever. And so I sneak one of my dad’s beers and drink it to fall asleep at night. And to me it’s beneficial because I’m anxious and it helps me be less anxious and it helps me sleep.” And again, we’re not endorsing this as a public health practice. But we’re just saying that if you had a high number of kids or an interesting number of respondents choose that, that would be a spot to do some more digging and to figure out, what are the attitudes and behaviors and perspectives there? Because that would inform the work of this entire coalition that wants to see teens and young adults drinking less or not at all.

Karyn:                   Right. And so it goes back to the purpose of the question. What are they trying to learn? Are they trying to learn the actual beliefs? Or are they trying to learn the forced beliefs within this certain window of allowed beliefs? Well, you’re allowed to think these things. If you’re out here we don’t want to worry about you. You must respond these ways. Versus, okay, we’ve got a nice solid group of kids who said harmful, but we’ve got these other kids, say 25 percent, who were on the zero to beneficial. Let’s find out WHY they think it’s beneficial. Let’s dig, let’s do more research.

Karyn:                   So it all comes down to are you just trying to get data to fill in a report and meet your grant requirements or are you actually trying to learn? Don’t get me started.

Maggie:               We can do a whole other video about that!

Karyn:                   Are you just trying to meet your quota or do you actually want to know what people think about it so that you can learn and help them and learn from them.

Karyn:                   I could come up with a million scenarios we can dig into that, but we are just trying to talk about the question.

Maggie:               I know I’m going down a rabbit hole of. Okay. Why else would you say it’s beneficial? Does it have to do with parental attitudes or coping mechanisms?

Karyn:                   But again, it goes to the integrity of the question and the integrity of the question asker. Think about it from the kids’ perspective. Okay, you’re asking me to answer this? Yeah, yeah. I’ve seen this survey a million times. And then they see one that literally gives them real choices. Now, granted there would still be that suspicion of my teacher’s going to look at this. So there’s still always that Hawthorne effect. It’s just there when they know they’re being watched. But if you gave people the leeway to answer how they really felt, you would learn so much more. I could go on for days.

Karyn:                   Moving on to other ways to think about improving the question. Consider respondent comfort. So this the one I talked about already. Would the respondent feel comfortable providing any answer. I can answer what I actually think and I don’t feel like I’m going to be penalized for that.

Karyn:                   And also consider using open ended questions. So we wouldn’t have to put beneficial or harmful if it was open ended, but there would be a lot more data to look through. But that’s why you hire people.

Karyn:                   Those were our main things, like if you see a question and you think, okay, this is probably a leading question. How do I fix it? Find the bias, find the judgment words, do your best to eliminate or neutralize them and/or create more balance in the question. If there has to be a range, do you like this or do you not like this? Actually we looked at poll questions and I was pleasantly surprised that they were a lot less inflammatory. I thought they were going to be more inflammatory. I could tell for a lot of them that the polling people did their work, the reporters….ehhh We’ll save that for another topic.

Maggie:               They must have needed a hook. We could do a whole other video on presenting data with integrity. That’s going to be a lot of videos.

Karyn:                   That’s a lot of soap boxes. Oh my gosh. Okay. So wrapping this up, I feel like we’ve covered it pretty well. Is there anything that you think we’re missing or anything we need to go over again and again and again and again?

Maggie:               I think provided some great obvious examples, where average non survey expert would read those and think, oh my goodness, this is strong phrasing. And we’ve provided some ideas, some questions to use as a framework to figure out if you’re being asked a leading question. We have provided some ideas on how to de-lead your questions to how to make them not leading questions. I think we’ve left people in good hands.

Karyn:                   Excellent. So yeah, the benefits of all this are that if you are trying to get unbiased data and you can keep your questions as neutral as possible, then your data will be useful. You’ll be able to feel confident that these people gave you real answers and not just tried to make you happy. Which is good because it’s not wasting your time or theirs and then you’ve got data that you can actually use and for whatever purposes, and it’s quality data that is actually giving you real information. So that was our fun rant on leading questions.

Wrap Up

Maggie:               Thank you for sticking with us.

Karyn:                   And I think it was less than an hour! We were fully prepared. We were like, okay, this is going to go forever. Oh my gosh. I was fully ready to just ditch it and go off on like 10 tangents.

Maggie:               Me too but I just mentally bookmarked them for future conversations.

Karyn:                   If you have questions about our questions, feel free to comment anywhere you see this video. We will do our best to answer and help you out. And if you found this content valuable, please like and share it with anyone you think needs to see it. You can send it to somebody who really needs to see it who is on a question writing team or something. You’re like, “hey, they talked about your questions in this video, maybe you should watch this.” You can totally do that and just subtly leave it open on somebody’s computer.

Maggie:               And if you are an educated person, you will agree that this needs to be shared. (Maniacal laughter)

Karyn:                   Are you leading them? Shame on you! You’re going to make them disagree just out of spite. Anyways, thank you so much for watching. If you would like to keep up with the rest of our video series, please go ahead and subscribe to this channel and we will see you in the next video. Thank you.

 

Customer Interviews Are Full of Gold

This post is all about the big picture, the value that comes from having conversations with your customers. It’s about being aware of and extracting the value out of your follow up conversations and interviews with your customers.

Why should you care?

This whole post is about explaining why you should care. About what value you can get out of your customer conversations. But the shortest version I can manage of why you should care is that there is gold in the customer interviews. There is messaging gold, there is social proof gold, feedback gold, insight gold, and relationship gold. And yes, it takes effort but the payoff is so worth it on many levels for your business.

If you aren’t engaging in these conversations with your customers and getting feedback and really checking in with them and you aren’t recording that information, you are leaving gold on the table.

What Do I mean by Customer Interviews and Conversations?

I will be going into more detail on all of these areas in other videos. There’s a whole lot more to talk about. I wanted to touch on the main points of each quickly here. I have a graphic to show you what I’m talking about so that you can visualize what I’m talking about. But before we get into that, I wanted to define what I mean by interview or customer conversation.

So what I mean here is any conversation or interview that you have with a client or someone else has with your client on behalf of your business, where the purpose is to check in with them and get feedback or hear their thoughts on their experience with your product or service in the context of their story and lives.

How is your service or product fitting into their lives? What was that like for them? That is what I’m talking about when I talk about a customer conversation or a customer interview in this context. And it doesn’t have to be just at the end, but that is an entirely separate video.😉

The Spiral of Value

Here is the framework I wanted to share. I’m calling it the Spiral of Value. [Click to Enlarge]

 

I wanted a way to visually represent what I’m talking about when you look at all of the value that you can get from one customer conversation. This doesn’t only apply if you have 50 of them. You can get all of this information and value out of one customer interview or a customer conversation. Imagine what more you can get as you talk to more and more customers. As you can see we’re talking about messaging, social proof, feedback, about insights and I would argue most importantly we are talking about relationships.

Messaging: Make Writing Content Easier

Messaging gold. Now when I say gold, I hope you know what I mean. But just to be clear, I mean raw data, raw original content that you can use for your business that comes from the mouths of your customers. Their actual voice, not your imagined customer voice. You get to capture your client’s actual words. You don’t have to guess what they would say. You don’t have to imagine what they would say. You get to hear what they actually say about your business, about your business value. You don’t have to assume or make anything up and you can use it for all of your copy needs.

They’re going to talk about benefits, anxieties, and features. They’re going to talk about the situation they were in and why they hired you, and how it’s been going. They’re going to talk about results that then you get to talk about.

They give you in the context of this conversation, all of the words for the copy that you need to write. That is a huge pot of gold right there, which allows you to then speak to your ideal clients with the words of others like them. And your copywriters will totally thank you if you come to them and say, “So I have all of this customer data and interviews and transcripts.” And they’re like, “What? I don’t have to do any of that? And you already have their words!” So your copywriters will thank you. That’s a quick touch on what value you get out of the conversation for messaging.

Social Proof: Get Better Testimonials

The next step is social proof gold. This is quotes. This is testimonials. This is case studies. These are the things where it says, “I hired so and so and I was able to achieve this because of our work together.” Boom. So you get quotes, testimonials and it’s all there if you can just pay attention and then all you have to do is say, “So can I use this quote?” And they say “Yes!” And you’re like, “Okay!”

The most important thing that social proof allows you to do is to let your clients do the work of explaining your value to others. They’re the ones saying you’re awesome. They’re the ones saying the results that they got and then linking it back to you. You don’t have to do that. Which I know a lot of us can feel uncomfortable and we feel like we’re bragging. We should be able to brag, but we can totally let our customers do the bragging for us. So you get the quotes, you get all the words that you can use to let your clients explain your value.

Feedback: Do Business Better

Next up is feedback gold. And I know people hear feedback and go, ugh. This isn’t like a corporate performance review. This is totally different. So it’s not as scary as you think.

When you are talking to them you can get information that will allow you to make process improvements, communication improvements, and allow you to live out your goal in your business of constantly striving to make your business better at serving your clients. A lot of the times you’re going to get things that are little. It’s not always going to be this epic, monumental shift. But you’re going to get things that will allow you to make those small adjustments that you need as you go along to just have a better business.

Insights: Know Your Customers

Insight gold. This is valuable for both new business owners who are trying to validate an idea and established businesses. It allows you to listen for opportunities to serve your current clients in new ways. So when you are talking about, well, I’ve had this one program for so long but I want to pivot or I want to make a new program but I don’t know what they need. This is where you can learn that and then create things based on what they’ve said. It allows you to listen for opportunities to create what they already need instead of guessing.

Relationships: Build the Trust

Next up is my favorite, relationship gold. This is just straight up connection and trust building. Know, like, and trust? This is part of where the trust building happens because you care enough to ask them what they think and then do something with that.

Think of how it makes you feel when people follow up with you and reach out and say, “Did you enjoy that?” or “Thank you very much.” All of those things that we really appreciate in our daily lives. This is a touch point. This is you saying, “I care about you as a person. I care about you as a client and I want to make sure that your experience is as good as it can be.”

You give them the opportunity to process their experience and reflect on the progress they’ve made. I can’t tell you how many times I’ve been in an interview where that’s the first time they’ve really said out loud their results or described the successes they’ve had. It matters.

You’re not treating them like just another sale. You are building the relationship. You’re not saying “Thanks for the money. I’m never going to talk to you again.” You’re building the relationship which leads to better word of mouth and referrals and a solid network of people who want you to succeed.

More to Come

So that’s a quick overview of the five areas that I outlined on the diagram that are valuable from a customer interview perspective. I will be diving deeper into each of these on future videos. If this sounds like something you might need help with, message me and I would love to answer any questions you have. And if you think this is something other people would find valuable, please like and share.
Thank you!

Summary

We are talking about using plain language in your questions and cutting out the jargon.

Why does that matter? This should be obvious, but it is not. It matters because if people can’t understand what you’re asking they are going to leave, or worst GUESS. And having a bunch of data that is comprised of guesses is not helpful to you at all.

Jargon is a gatekeeper, and not the good kind.

Video Transcript

Introductions

Karyn:                   My name is Karyn Kelbaugh and I help small businesses with customer research. I also do evaluation for nonprofits, which is a big word for research. I will de-Jargon myself and just say – evaluating is measuring how well they are doing what they are trying to do. So there we go and you can find me at heykaryn.com

Maggie:                Perfect. And My name is Maggie Hodge Kwan. Like Karyn, I do evaluation or in other words, research for nonprofit and philanthropic foundations and you can find more about me and my work at creativeclarity.ca.

We ALL Use Jargon

Karyn:                   Awesome. Yeah, we. We’re like the worst offenders, researchers, in general about using jargon, so we’re preaching to ourselves at this point. This covers everybody in all sorts of fields but man, researchers just, we’re like, “I know all the words! I’m going to use, all the words! Why wouldn’t I use all the words?”

Maggie:               And with good intent, is that we can also be a little bit pedantic and

Karyn:                   What does pedantic mean? Can you tell us what pedantic means? We’re gonna have to define everything. 😂

Maggie:               We survey people and researchers also want to be really careful and act in integrity in how we present things and sometimes that causes us to assume that our knowledge is everyone else’s knowledge.

Karyn:                   So the main problem here is that when you’re writing a question, you know the things, you know the words, you know the best word in your field for this topic or this problem or symptom. You’ve been in the field for awhile and every single type of job or type of field has its own language. It has its own words that mean different things like there’s the Internet people and the business internet people who are using all their acronyms and things and there’s engineers, oh my goodness, and they have all their acronyms.

Karyn:                   Everyone who has expertise in an area or is used to the common language of that area, you get used to it. Then when you’re trying to ask a question of someone who may or may not be knowledgeable about those things, we have to stop and realize that we need to use the most plain language possible when we’re asking a question so that the most number of people can understand it. I was trying to think of a good example. I mean there’s so many.

Maggie:               There are tons, I work with philanthropic foundations and even that probably requires a plain language explanation, but basically, groups of people governed by law and overseen by a board, that fund raise and then manage giving those funds. So giving money away to support local charities or nonprofits that work in any number of different sectors. That’s so jargony still, but I’m thinking about the particular language they have that I have seen in questionnaires to potential donors where they ask things about endowment funds and spend down funds and named funds and all these things that 99 point nine, nine, nine, nine, nine percent of the population would have no idea what that is, and don’t really need to know to be honest. But wouldn’t be able to respond to that kind of question with confidence because who knows what those answers are.

Using Jargon is a Form of Bias

Karyn:                   Right? And another way to look at it is that if you use…it’s impossible to remove everything that might have some jargony or more complicated words out of it. But if you are using jargon and complicated language in your question, that is a form of bias and what we are trying to do is help you understand how to make bias free questions. And the way that that has bias is that it is eliminating everyone who doesn’t know what you’re talking about from answering the question. So you’re only going to get answers that are real from a smaller portion who already knows what you’re talking about. So if you’re trying to reach people who are new to you, are new to the thing that you do, they’re going to be like, I don’t know what you’re talking about. Skip, I’m out of here and you’re not going to get answers from them.

Like Explaining Gravity to a 7 Year Old

Karyn:                   So you have to think about who the potential responders are, who is your target community or target group that you are trying to get answers from? And think about what their knowledge level probably is on the subject. And say, okay, what if they don’t know anything about what I’m talking about? And Maggie had a good comment earlier. When your kids ask you complicated questions like, “Mom, can you tell me how gravity works again?” And you’re like, okay, and you literally have to go, okay, a seven year old is asking me to explain gravity. I have to use a seven year old vocabulary to explain gravity. And you kind of have to approach it that way. What is the simplest, most direct way I can ask this question and still get the answer that I am looking for. So it’s pretty much, it’s that simple.

Maggie:               Yeah. So if you had written survey questions and you wanted to screen for jargon, there are a couple of ways you could do it. One, you can come back to it with fresh eyes and look to see if there’s anything that would need to be defined. So for example, if you were an epidemiologist writing a survey that was meant for the general public to gauge perceptions around public health, you wouldn’t want to be using words like epidemiology because people are like, what the heck is that? And you know, you wouldn’t want to be using really scientific language that would require people to say “What is that and how do I respond to this?”

Karyn:                   The other one is slang. It’s like we think jargon in terms of just fancy words in the field, but it’s also just slang that gets used in that field that only means something to the people in that field.

Maggie:               Capacity building. 😂

Karyn:                   Things that mean one thing in one field can mean something completely different somewhere else. So then it can get really confusing if they’re thinking, well in the general public this word means this, but over here you’ve used it in a clever way so we have no idea what you’re talking about. And, even if it’s just a matter of showing those questions to other people that you trust but do not get what you do, those are perfect people to show it to. “Do you know what I’m asking?”

Maggie:               Yeah. Are there any words that you would need to Google for the meaning so that you could complete this survey? Because if so we should know about that and we should get rid of them.

Karyn:                   Exactly. Yeah. If your people have to go, okay, so what do you mean by that? Then it’s too complicated. Well that that’s it for today. So please take a look at your questions. Give them to your family members, see what they think, and if you found this content valuable, please like and share. Thank you very much. And if you want to keep up with our video series on questions, please subscribe to this channel.  

Summary

Why should you care? Anytime you want to gather data from your clients, community members, heck your kids… Question design matters. It helps you get the kind of answers that you can use, vs data that isn’t really helpful, or is biased, or incomplete, or even WAY too much to handle. Pick the right tool for the job.

Video Transcript

Why Should You Care About Open-Ended vs Multiple-Choice Questions?

Maggie: So excited to be with you today talking about an extremely exciting subject in survey design. Open-ended versus multiple choice questions. Why should our viewers care?

Karyn: You should care because you want to pick the right tool for the job. Anytime you want to gather data from your clients or community members or your kids, the way you design the question matters and it helps you get the kind of answers that you can use versus data or answers that aren’t really that helpful or are biased or incomplete or too much. If you’ve got 150 people that you need to get data from, having a bunch of open ended questions is going to be, oh great, this is what I’m going to spend the rest of my life doing is reading these answers. So it’s all about designing the question for the kind of data that you were trying to get. 

Introductions

Maggie: Fantastic. So before we jump into that, who are you? Where can we find you? What’s your purpose in life?

Karyn: Okay. The first two I can do. My name is Karyn Kelbaugh and I help small businesses with customer research, particularly interviews and I help nonprofits with evaluation and other fun nerdy things and you can find me at Heykaryn.com and I will get back to you on the purpose part. Who are you and where can we find you?

Maggie: Thank you for not asking about my purpose. My name is Maggie Hodge Kwan and like you I work with nonprofits particularly around evaluation and other services as required and you can find out more about that and about me at Creativeclarity.ca.

What are the differences?

Karyn: So we’re going to try and keep this one short today because it’s not that long of a question to answer, but I’ll just go ahead and ask you, what is the main difference or what would be the purpose of an open-ended question? Choosing an open ended question over a multiple choice question.

Maggie: Well, I think you alluded to it already in your why does this matter? Um, but it is finding that right tool for the job in front of you. So, of course when you ask a multiple choice question, you are prescribing answers for your survey taker to pick from and this certainly has its place and then when you’re asking an open ended question, you’re sort of saying, hey, there could be so many perspectives or so many different responses to this question that I need to just give you this box to blurt your thoughts into because I don’t know what all of the different possibilities are. And so again, both have their place. A couple of things that stick out to me are that in survey design, you want your survey to be as easy to respond to as possible and generally asking people a lot of open ended questions turns them off, because they have to do more thinking.

Maggie: And particularly when you’re asking those big, broad questions like name all of the strengths and the things we should celebrate about the community that you live in. You’re asking for people to give some pretty deep thought and to provide some time to think through their answer. So, yeah, both have their place. If you’re asking someone a frequency question, for example, how often do you go to the gym? Multiple choice is perfect because there are only so many different options and it’s easy for someone to say, Oh, 1-3 times a week. On the flip side, if you’re asking someone to describe maybe the three biggest pain points of a current problem they’re having, it might make more sense to leave it open ended so that, their perspective and their circumstance can really come through.

Karyn: So it’s like when you’re thinking about it, if you already, if the potential answers already have some boundaries or parameters, then it might be a better choice to go ahead and pick the multiple choice because you’ve got, like you said, like with the time, if it’s an how often, when. Things that already have a known quantity and are easy answers. But things that you’re doing that are more exploratory. Like I’m just restating what you stated another way, but you know, things that you don’t know where the edges are. I think this is probably the kind of things that people would put, but I don’t know yet. Another way I like to think about it is, especially if you’re kind of in a project where you’re trying to figure something out, like your end goal might be to create a survey you could ask a lot of people. Starting with those open ended exploratory questions with a small number of people to give you your top five answers for things. Let’s find out what everybody says, oh look, all of these people gave me consistently these five things came up as options and then you’re like, okay, so now I can make a multiple choice question was an other box. Me and other boxes. I can’t let them go. I’m just like, but what if they have a different response?

Maggie: Well, best of both worlds, right? You’re finding a lot of people who will use one of the answers that you’ve provided and you’re still giving space for people who come at it from a different angle.

Pros and Cons

Karyn: Trying to think about pros and cons of each. So if I were going to say the pros and cons of open ended, pros being it’s discovery, you can ask, now, obviously you have to have a good question, but we’ll get to that later. But in your questions, you’re giving people room to write whatever they think however they want without your limitations being imposed on them so you can learn more that you didn’t even know you didn’t know. The downside is that you then have to read all of those responses and process that information and make it useful, not necessarily to quantify it, but to make sense of it. And so that’s just more work.

Maggie: Yes. So if you’re designing a survey and you need 500 responses to it. I’ve been here. I had a survey monkey survey that had a number of open ended questions and printed off that is 493 pages.

Maggie: So if you are going for a large number of respondents, yes, that’s something that you’d want to plan for in terms of time, is that even reading the responses takes time. Then adding onto that, coding them somehow making sense of them and then preparing them or presenting them in a way that is meaningful and actionable. So there is a con.

Karyn: And then for multiple choice, obviously the pro is that you get this neat and tidy dataset. You’re like, oh, okay. You know, 10 percent of people said this, 20 percent of people said this, and the analysis part is literally just okay, let’s look at my report and see what everybody said. The cons, to me are that if you have used it for something that might, there might be other answers out there that you didn’t account for, you are looking at this little narrow window of possible choices. And if you didn’t do the work upfront and you’re just saying, oh, I’ll just pick the ones I think everyone will answer, then you’re just going to get confirmed what you thought everyone would answer. People will say, well, none of those are what I would do. Bye bye. And they leave.

Karyn: Obviously, if it’s what time do you go to the gym, that one’s going to be an easy yes, but if there’s one where you had to make a decision on what choices were included, then you can be building in bias to your question so that you’re thinking, well, I was right. Those are all the choices that everyone selected. And really that’s the only ones you gave them.

Maggie: So for example, if you ask a multiple choice question that says how many minutes a day or how many times a day do you check instagram? It’s scary and people are probably confronted by the responses. But I think people could say, you know, seven times or I look at it 18 times a day or whatever it is. There is there some kind of defined range.

Maggie: But if you create a multiple choice question that says, choose which of the following values are most important from a small business that you would support. And then you list honesty, transparency, community, whatever it is, you’re making a huge assumption that someone’s most important value is even on that list. In other words, unless you give them 200 choices, which is going to be a horrible experience, you are putting your own bias in the options.

The Best of Both

Karyn: So the way to find that happy medium then is to use the open ended piece or other more discovery based or exploratory way of finding that out and talking to people and learn what the most common values are that keep coming up and then you have an educated list of choices that you can ask people and feel more confident that you’re including the ones that are the most popular.

Maggie: Yes. And throw in that other box for good measure.

Karyn: Other boxes are your friends! Then you’ll get random comments. I love surveys, they’ll answer it, but then they also are, okay, I just wanted to tell you this survey was really long.

Maggie: Yeah, I did a demographic survey once that asked people about their sexual orientation and knowing that there are many ways to define orientation, put a comment box and boy, lots of pickup lines inserted in that comment box.

Final Thoughts

Maggie: So in summary, multiple choice questions are really helpful when you are surveying lots of people, when you might have a limited amount of time to analyze and make sense of the data and when you know the subject or know the quantities really well and you can confidently provide options that meet the potential responses of the people taking your survey. And on the flip side, open ended questions are great when you don’t know and you need to do that discovery and they’re helpful when you’ve been able to make the time to really analyze them and make sense of them.

 

Summary

Why should you care about double-barrelled questions? They confuse people and make you look like you are either being manipulative or just plain incompetent. They waste EVERYONE’s time, including yours.

Video Transcript

Why Should You Care About Double-Barrelled Questions?

Karyn: Today we are talking about double-barrelled questions and how horrible they are and that sort of thing. So just to get started and so everybody knows who might be watching. Maggie, why should we care about double-barrelled questions?

Maggie: You should care about double-barrelled questions for a number of reasons, but most importantly, double-barrelled questions can confuse respondents and when respondents are confused, they are questioning your authority and legitimacy and it’s going to weaken your position and not lead to any useful data or valuable insights.

Karyn: That’s the best answer I’ve ever heard to that question. That is why it matters. When people get confused, they quit, they stop. If you don’t understand the question, you skip it or just say I’m done. Thank you. Close window.

Maggie: Yes. And they totally don’t trust you because they’re like, wait, this person’s an expert in this and they don’t know how to ask me a question about it? No.

Introductions

Karyn: Alright, well just so you know who we are. My name is Karyn Kelbaugh and I do customer research, and evaluation for nonprofits and you can find me at heykaryn.com.

Maggie: I am Maggie Hodge Kwan and I do evaluation and research work for nonprofits and philanthropic foundations. And you can find me at creativeclarity.ca.

What is a Double-Barrelled Question?

Karyn: So today we’re going to try to keep it shortish, and get into our series of nerding out about questions. And today, as we said, we are talking about double-barrelled questions. And so let’s just talk about defining what is a double-barrelled question. Like to me, it is when you are asking more than one question but you’re trying to jam it into one question. Like what’s your favorite color? And do you like strawberries? A very simple version, but they can be a lot sneakier to where you don’t really think it’s a double-barrelled question. You had a really good example, like when you’re trying to connect two ideas together that don’t necessarily go together.

Examples

Maggie: Or even when they do. And the example that I can think of here is from a quality of life survey put out with the best of intentions, but asking many questions within questions. And one of the things that stood out to me – it said something like, “My municipality encourages and supports small business”. And off the top you think, oh, that’s interesting. This is gaging my perception around whether or not the place that I live support business, but you look at those two separate concepts and encouraging small business is one thing and it probably has to do with it with the attitude. Supporting small businesses is another thing and that could be with bylaws, policies, incentives, all those kinds of things. So as a respondent, I feel like I’m in a quandary because I might have two different opinions. I might think yes, they’re very vocally encouraging of small business, but no, there aren’t really a lot of actions that support small business.

Karyn: Yeah. It’s like if you can give two different answers to a question, it might be a double-barrelled question.

When you are saying “Yes, And” or “Yes, but”, that means that it’s probably a double-barrelled
question. So that’s kind of a good way to, to think about it, to recognize it. If you could break it down into separate questions, it’s probably a double-barrelled question.

I’m trying to think of another good example. They’re not just questions. You had one that was a statement. So it’s do you agree or disagree with this? And I think there was another example where we were talking earlier, the climate one. Yeah. They, sounded like they were trying to, and I’m going to be nerdy here, so just get over it, put causation into the statement. Like this happened because of this or something like that. Where they’re just trying to bake as much as possible into the question or statement. Do you remember that statement?

Maggie: Yes. And the statement was something like, “climate minded citizens support active transportation”. So, you know, walking and cycling routes and public. Sometimes public transportation can be related to that, but very much seeding the idea that people who care about climate change and people who want to mitigate climate change would automatically support things like sidewalks and cycling infrastructure. So just a lot going on. There are many issues.

Karyn: That one has a whole lot of problems. Yes, let’s fill a sentence full of jargon and see if anybody can tell what we’re talking about. And also the fact that you get to the end of the question, and you think, so what? We could go all day (haha) which is why where we’re making more than one video. But the idea of trying to shove two ideas into a statement and then asking do you agree or disagree with this? And you’re thinking “well, sort of”. If you can’t give a clear answer. If people can’t have an answer pop into their head immediately, then you’re confusing them and they’re going to leave.

Maggie: Yes. So, because that’s the kind of, I mean, Karyn, I think you know this probably better than anyone, but in a survey specifically, you want the experience to be as easy as possible for people. You want to use plain language so that people understand what you’re asking and you want them to be able to respond without having to take 10 minutes to think about exactly what’s being asked. So those darn double-barrelled questions really throw a wrench into that and give people pause and if you’re anything like me, make me kind of paranoid that they’re going to use the data in a weird way or that they’ll just have to throw it out and then they’re wasting my time asking me about it.

Not Always Better Together

Karyn: Yeah. I could just picture this scenario where somebody is tempted to do it and they don’t realize they’re doing it when they’re thinking, well, you know, I wrote this survey and it’s got 15 questions. I feel like that’s too long, so I’m going to make it 10. But I still want 15 questions worth of answers. So instead of actually narrowing down their questions based on the best questions, they just combine them thinking it will work.

And there’s a difference between asking a question and then having an immediate follow-up question. Literally a separate question, but it’s together. That’s one thing. Like you can kind of combine that to where I’m clearly asking you two questions, but I put it on one line and I made it a follow up question. That’s different than I’m going to ask what is your opinion about this and your opinion about this? Like I can’t just shove them together, but it can be really tempting when you’re trying to make it shorter to just say, well we’ll just jam them together and it’ll be fine. You’re better off just having a longer survey. I mean, you’re better off having a longer survey that asks clear individual questions than you are trying to jam them together just for brevity.

Maggie: Absolutely. And so, you know, hypothetically a grocery store wants to know what its customers are buying and let’s just ignore for now the fact that of course, they would know because they have all the sales data, but you know, say that they’re thinking, okay, we only have five minutes of our customer’s time. We need to ask you know, we need to cram in as much as we can. And so a customer gets a question that says “I buy apples and oranges” and the responses are always, frequently, sometimes, or never. There’s really no winning there. Right? And that would be a case where you would, it would make much more sense to break them into two questions and figure out how often people are buying apples and how often people are buying oranges by asking about those things separately.

Karyn: Or we could change the whole context of the question. Like how often do you buy fruit?

Maggie: Love it.

The Why Matters

Karyn: You’re going to hear us say this over and over again in every single video so just start putting it in your head. Why you’re asking the question matters. What are you trying to find out? What is the purpose of the question so that you can come back to that every single time. Like, why do they care how many apples and oranges they’re getting? And kind of just getting to the point of that. That can always bring you back when you feel like your questions might be getting a little out of control.

Is there anything else on double-barrelled questions that we should cover? Do you think that pretty much hits it?

Check Your Beliefs

Maggie: There’s just one more thing that I wanted to add, and that is that double-barrelled questions, as you catch yourself writing them, are a really good opportunity to check your own attitudes and biases that you might be bringing into a piece of research. So as you’re writing a survey, if you are that municipality and you’re writing, “my municipality supports and encourages small businesses”, that might be an opportunity to figure out, am I just trying to find out as much information as possible within a small survey or as a survey writer, is my own belief sort of bleeding in here? And how could I take a look at this question and reword it so that it’s asking one thing in an objective manner that isn’t biased by the survey writers experience or attitudes?

Karyn: Which totally touches on the topic that we’re going to talk about in another video, which is Leading Questions, which I’m just. There’s so much to talk about with leading questions, but for now, thank you.

That was awesome. Thank you for joining me. So that’s it for now for double-barrelled questions, but if you have any follow up questions for us, wherever you are watching this video, just send us a comment and we would be happy to get back to you on that. And if you found the content valuable, we accept exposure for payment so you can click like or share it with anyone that you think would find it valuable. And if you want to follow along with our question series, you can subscribe to the Youtube Channel or whatever page you are watching this on. There will be quite a few more. We’re going to get into other stuff like leading questions. And Maggie had a fabulous one which we’re totally going to do, which is developing the shortest most effective survey which totally ties into today. And other fun stuff like when it’s better to use an open-ended question versus a multiple choice question and all kinds of other survey format stuff. So thank you very much and we’ll talk to you later.