Will at Work Learning

GUEST POST by LAUREL NORRIS: Robust Responses to Open-Ended Questions: Good Surveys Prime Respondents to Think Critically

2017-08-31T11:38:59-04:00

This is a guest post by Laurel Norris (https://twitter.com/neutrinosky).

Laurel is a Training Specialist at Widen Enterprises, where she is involved in developing and delivering training, focusing on data, reporting, and strategy.

--------------------------------------------------

Robust Responses to Open-Ended Questions: Good Surveys Prime Respondents to Think Critically

By Laurel Norris

--------

I’ve always been a fan of evaluation. It’s a way to better understand the effectiveness of programs, determine if learning objectives are being met, and reveal ways to improve web workshops and live trainings.

Or so I thought.

It turns out that most evaluations don’t do those things. Performance-Focused Smile Sheets (the book is available at http://SmileSheets.com) taught me that and when I implemented the recommendations from the book, I discovered something interesting. Using Dr. Thalheimer’s method improved the quality and usefulness of survey data – and provided me with much more robust responses to open-ended questions.

By more robust, I mean they revealed what was helpful and why, talked about what they thought their challenges would be in trying it themselves, discussed what areas they thought could use more emphasis, and shared where they would have appreciated more examples. In short, they provided a huge amount of useful information.

Before using Dr. Thalheimer’s method, only a few open-ended responses were helpful. Most were along the lines of “Thanks!”, “Good webinar”, or “Well presented”. While those kinds of answers make me feel good, they don’t help me improve trainings.

I’m convinced that the improved survey primed people to be more engaged with the evaluation process and enabled them to easily provide useful information to me.

So what did I do differently? I’ll use real examples from web workshops I conducted. Both workshops ran around 45 minutes and had 30 responses to the end of workshop survey. They did differ in style, something that I will discuss towards the end of this article.

The Old Method

Let’s talk about the old method, what Dr. Thalheimer might call a traditional smile sheet. It was (luckily) short, with three multiple choice questions and two open-ended. Multiple choice questions included:

How satisfied are you with the content of this web workshop?
How satisfied are you with the presenter's style?
How closely did this web workshop align with your expectations?

Participants answered the questions with options on Likert-like scales ranging from “Very Unsatisfied” to “Very Satisfied” or “Not at all Closely” to “Very Closely”. Of course, in true smile-sheet style, the multiple choice yielded no useful information. People were 4.1 level satisfied with the content of the webinar, “data” which did not enable me to make any useful changes to the information I provided.

Open-ended questions invited people to “Share your ideas for web workshop topics” and offer “Additional Comments”. Of the thirteen open-ended responses I got, five of them provided useful information. The other seven were either a thank you or some form of praise.

The New Method

Respondents were asked four multiple choice questions that judged effectiveness of the web workshop, how much the concepts would help them improve work outcomes, how well they understand the concepts taught, and whether or not they would use skills they learned in the workshop at their job.

The web workshop was about user engagement, in particular, administrators increasing engagement with the systems they manage. Questions were:

In regard to the user engagement, how able are you to put what you’ve learned into practice on the job?
From your perspective, how valuable are the concepts taught in the workshop? How much will they help improve engagement with your site?
How well do you feel you understand user engagement?
How motivated will you be to utilize these user engagement skills at your work?

Responses were specific and adapted from Dr. Thalheimer’s book. For example, here were the optional responses to the question “In regard to the user engagement, how able are you to put what you’ve learned into practice on the job?”

I'm not at all able to put the concepts into practice.
I have general awareness of the concepts taught, but I will need more training or practice to complete user engagement projects.
I am able to work on user engagement projects, but I'll need more hands-on experience to be fully competent in using the concepts taught.
I am able to complete user engagement projects at a fully competent level in using the concepts taught.
I am able to complete user engagement projects at a expert level in using the concepts taught.

All four multiple choice questions had similarly complete options to choose from. From those responses, I was able to more appropriately determine the effectiveness of the workshop and whether my training content was performing as expected.

The open-ended question was relatively bland. I asked “What else would you like to share about your experience during the webinar today?” and received twelve specific, illuminating responses, such as:

“Loved the examples shown from other sites. Highly useful!”

“It validated some of the meetings I have had with my manager about user engagement and communication about our new site structure. It will be valuable for upcoming projects about asset distribution throughout the company.”

“I think the emphasis on planning the plan is helpful. I think I lack confidence in designing desk drops for Design teams. Also - I'm heavily engaged with my users now as it is - I am reached out to multiple times per day...but I think some of these suggestions will be valuable for more precision in those engagements.”

Even questions that didn’t give me direct feedback on the workshop, like “Still implementing our site, so a lot of today's content isn't yet relevant”, gave me information about my audience.

Conclusion

Clearly, I’m thrilled with the kind of information I am getting from using Dr. Thalheimer’s methods. I get useful, rich data from respondents that helps me better evaluate my content and understand my audience.

There is one positive aspect of using the new method that might have skewed the data. I designed the second web workshop after I read the book, and Dr. Thalheimer’s Training Effectiveness Taxonomy influenced the design. I thought more about the goals for the workshop, provided cognitive supports, repeated key messages, and did some situation-action triggering.

Based on those changes, the second web workshop was probably better than the first and it’s possible that the high-quality, engaging workshop contributed to the robust responses to open-ended questions I saw.

Either way, my evaluations (and learner experiences) are revolutionized. Has anyone seen a similar improvement in open-ended response rates since implementing performance-focused smile sheets?

Another Reason to Learn About Performance-Focused Smile Sheets

2017-08-14T16:38:10-04:00

This has been a great year for the Performance-Focused Smile Sheet approach. Not only did the book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, win a prestigious Award of Excellence from the International Society of Performance Improvement, but people are flocking to workshops, conference sessions, and webinars to learn about this revolutionary new method of gathering learner feedback.

Now there's even more reason to learn about this method. In the July 2017 issue of TD (Talent Development), it was reported that the Human Capital Institute (HCI) issued a report that said that measurement/evaluation is the top skill needed by learning and development professionals!

Go to SmileSheets.com to get the book.

Using Performance-Focused Smile Sheet Questions -- Even One!

2017-08-14T15:26:45-04:00

Mark Jenkins, a long-time technical trainer and forward-thinking learning professional, now providing concierge service on learning technologies, audio-video, and OneNote at inforivers, recently used a performance-focused smile-sheet question at the end of one of his public training sessions. He used just one question! And one follow-up open-ended question.

Mark loves the results he's getting. Now he gets very clear feedback on whether his workshop helps people actually do what he wants them to be able to do. And, he is able to judge their interest in another session on the same subject matter, all on a one-page smile sheet.

"I was surprised the liberation I felt not being shackled down by Likert scales, while still getting good and easily understandable analytics. The results are less ambiguous than a Likert scale. It also helps me to figure out how to follow up with each person that took the smile sheet."

His feedback form, shared with his permission:

Major Research Review on eLearning Effectiveness

2017-08-10T16:44:25-04:00

Is elearning effective? As effective as classroom instruction -- more or less effective? What about blended learning -- when elearning and classroom learning are combined?

These critical questions have now been answered and are available in the research report, Does eLearning Work? What the Scientific Research Says!

In this research review, I looked at meta-analyses and individual research studies, and was able to derive clear conclusions. The report is available for free, it includes an executive summary, and research jargon is kept to a minimum.

Click here to download the report...

Note that the August 10, 2017 version of this report incorrectly cited the Rowland (2014) study in a footnote and omitted it from the list of research citations. These issues were fixed on August 11, 2017. Special thanks to Elizabeth Dalton who notified me of the issues.

Research Reflections -- Take a Selfie Here; The Examined Life is Worth Living!

2017-07-11T13:17:17-04:00

As professionals in the learning field, memory is central to our work. If we don't help our learners preserve their memories (of what they learned), we have not really done our job. I'm oversimplifying here -- sometimes we want to guide our learners toward external memory aids instead of memory. But mostly, we aim to support learning and memory.

You might have learned that people who take photographs will remember less than those who did not take photographs. Several research studies showed this (see for example, Henkel, 2014).

The internet buzzed with this information a few years ago:

The Telegraph -- http://www.telegraph.co.uk/news/science/science-news/10507146/Taking-photographs-ruins-the-memory-research-finds.html
NPR -- http://www.npr.org/2014/05/22/314592247/overexposed-camera-phones-could-be-washing-out-our-memories
Slate -- http://www.slate.com/blogs/the_slatest/2013/12/09/a_new_study_finds_taking_photos_hurts_memory_of_the_thing_you_were_trying.html
CNN -- http://www.cnn.com/2013/12/10/health/memory-photos-psychology/index.html
Fox News -- http://www.foxnews.com/health/2013/12/11/taking-pictures-may-impair-memories-study-shows.html

Well, that was then. This is now.

Research Wisdom

There are CRITICAL LESSONS to be learned here -- about using science intelligently... with wisdom.

Science is a self-correcting system that, with the arc of time, bends toward the truth. So, at any point in time, when we ask science for its conclusions, it tells us what it knows, while it apologizes for not knowing everything. Scientists can be wrong. Science can take wrong turns on the long road toward better understanding.

Does this mean we should reject scientific conclusions because they can't guarantee omniscience; they can't guarantee truth? I've written about this in more depth elsewhere, but I'll say it here briefly -- recommendations from science are better than our own intuitions; especially in regards to learning, given all the ways we humans are blind to how learning works.

Memory With Photography

Earlier studies showed that people who photographed images were less able to remember them than people who simply examined the images. Researchers surmised that people who off-loaded their memories to an external memory aid -- to the photographs -- freed up memory for other things.

We can look back at this now and see that this was a time of innocence; that science had kept some confidences hidden. New research by Barasch, Diehl, Silverman, and Zauberman (2017), found that people "who could freely take photographs during an experience recognized more of what they saw" and that those "with a camera had better recognition of aspects of the scene that they photographed than of aspects they did not photograph."

Of course, this is just one set of studies... we must be patient with science. More research will be done, and you and will benefit in knowing more than we know now and with more confidence... but this will take time.

What is the difference between the earlier studies and this latest set of studies? As argued by Barasch, Diehl, Silverman, and Zauberman (2017), the older studies did not give people the choice of which objects to photograph. In the words of the researchers, people did not have volitional control of their photographing experience. They didn't go through the normal process we might go through in our real-world situations, where we must decide what to photograph and determine how to photograph the objects we target (i.e., the angles, borders, focus, etc.).

In a series of four experiments, the new research showed that attention was at the center of the memory effect. Indeed, people taking photographs "recognized more of what they saw and less of what they heard, compared with those who could not take any photographs (I added the bold underlines).

Interestingly, some of the same researchers, just the year before had found that taking photographs actually improved people's enjoyment of their experiences (Diehl, Zauberman, & Barasch, 2016).

Practical Considerations for Learning Professionals

You might be asking yourself, "How should I handle the research-based recommendations I encounter?" Here is my advice:

Be skeptical, but not too skeptical.
Determine whether the research comes from a trusted source. Best sources are top-tier refereed scientific journals -- especially where many studies find the same results. Worst sources are survey-based compilations of opinions. Beware of recommendations based on one scientific article. Beware of vendor-sponsored research. Beware of research that is not refereed; that is, not vetted by other researchers.
Find yourself a trusted research translator. These people -- and I count myself among them -- have spent enough substantial time exploring the practical aspects of the research that they are liable to have wisdom about what the research means -- and what its boundary conditions might be.
Pay your research translators -- so they can continue doing their work.
Be good and prosper. Use the research in your learning programs and test it. Do good evaluation so you can get valid feedback to make your learning initiatives maximally effective.

Inscribed in My High School Yearbook in 1976

Time it was, and what a time it was, it was
A time of innocence, A time of confidences
Long ago, it must be, I have a photograph
Preserve your memories; They're all that's left you

Written by Paul Simon

The Photograph Above

Taken in Glacier National Park, Montana, USA; July 1, 2017
And incidentally, the glaciers are shrinking permanently.

Research Cited

Barasch, A., Diehl, K., Silverman, J., & Zauberman, G. (2017). Photographic Memory: The Effects of Volitional Photo Taking on Memory for Visual and Auditory Aspects of an Experience. Psychological Science, early online publication.

Diehl, K., Zauberman, G., & Barasch, A. (2016). How taking photos increases enjoyment of experiences. Journal of Personality and Social Psychology, 111, 119–140.

Henkel, L. A. (2014). Point-and-shoot memories: The influence of taking photos on memory for a museum tour. Psychological Science, 25, 396–402.

Free Online Diagnostic for Your Organization's Smile Sheet (Learner-Feedback Questions)

2017-04-04T22:56:36-04:00

In honor of April as "Smile-Sheet Awareness Month," I am releasing a brand new smile-sheet diagnostic.

Available by clicking here:
http://smilesheets.com/smile-sheet-diagnostic-survey/

This diagnostic is based on wisdom from my award-winning book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, plus the experience I've gained helping top companies implement new measurement practices.

The diagnostic is free and asks you 20 questions about your organization's current practices. It then provides instant feedback.

Prompting Learning When Our Learners Play Games

2017-03-08T10:23:15-05:00

Another research brief. Answer the question and only then read what the research says:

In a recent study with teenagers playing a game to learn history, adding the learning instructions hurt learning outcomes for questions that assessed transfer, but NOT recall. The first choice hurt transfer but not recall. Give yourself some credit if you chose the second or third choices.

Caveats:

This is only one study.
It was done using only one type of learner.
It was done using only one type of learning method.
It was done with teenagers.

Important Point:

Don't assume that adding instructions to encourage learning will facilitate learning.

Research:

Hawlitschek, A., & Joeckel, S. (2017). Increasing the effectiveness of digital educational games: The effects of a learning instruction on students’ learning, motivation and cognitive load. Computers in Human Behavior, 72, 79-86.