The meta-analysis section has a useful table providing a quick summary of the major findings of each meta-analysis reviewed. Here’s a few points from that research:

• “Game-based approach produced significant knowledge-level increases over the conventional case-based teaching methods.” (Wolfe, 1997)
• “An instructional game will only be effective if it is designed to meet specific instructional objectives and used as it was intended.” (Hays, 2005)

In the elements of games section, Karl summarizes several individual studies and their findings in the following areas:

• Reward structures
• Player motivation (both intrinsic and extrinsic)
• Avatars
• Player perspective

Gamification in learning is often viewed very superficially as just adding extrinsic motivators like badges and leaderboards. In this book, Karl recommends going beyond that shallow understanding to look at the ways that games can be effective and to use those elements to enhance learning.

If you’re interested in more information about the book, check out the other posts in the blog book tour.

References (as cited in The Gamification of Learning and Instruction):

Hays, R.T. (2005). The effectiveness of instructional games: A literature review and discussion. Naval Air Warfare Center Training Systems Division (No 2005–004).

Wolfe, J. (1997) The effectiveness of business games in strategic management
course work. Simulation & Gaming, 28(4), 360–376.

She started with her personal story, told mostly with old photos on the slides and very little text

Story: likeable hero, encounters roadblocks, emerges transformed

Why are so many presentations bad? We use presentations to create reports–dense “slide-uments”

When you need to persuade, use a story

Every story should have a beginning, middle, and end, with a turning point to move between sections

The presenter is not the hero of the story: the audience is the hero. They are the ones who have the power and must decide to take action. You are the mentor (she showed Yoda on Luke’s back while talking about mentors)

Joseph Campbell story structure

• Ordinary world
• Call to adventure
• Refusal of call
• Meeting with the mentor–this is a turning point

Freytag’s dramatic story structure; has a shape.

She wondered if great presentations had a shape like this

• What is
• What could be (the gap between this and what is is the “call to adventure”)
• Keep going back and forth between these two

An image of this shape is found in this summary of Duarte’s book

This shape can be used as an analysis tool She analyzed a 90-minute speech by Steve Jobs, who kept the audience riveted, laughing or clapping about every 30 seconds.

Jobs was passionate about his product and constantly marveled at it during the speech

STAR moment: Something They’ll Always Remember

Same kind of analysis for the I Have a Dream speech. Lots of pauses, more like poetry than a traditional speech. King had a rhythm to his speech.  Color coded analysis for the words: repetition; metaphor, visual words; familiar songs, scripture, literature; political references. He moved back and forth between what is and what could be at the phrase level at “I have a dream”; makes more excitement. Familiar references touch something that already resonates within the audience.

The stakes are higher now. It used to be that you could get away with crappy presentations because everyone else is crappy too. Now, there are books and best practices, and TED presentations set the bar higher. Twitter also sets the bar higher; the audience no longer has to suffer alone. They have a back channel and can revolt against a presenter. The audience can say cruel things. (example tweets from the disastrous #heweb09 keynote). Back channel can be good too; people may move to a good presentation they hear about on a back channel at a conference.

Don’t stay trapped in the roadblocks section of your own story. Push through and emerge transformed.

We need to find what we are passionate about to change the world.

Question: What do you do when you’re not fighting for human rights or a product that can’t be marveled at like the iphone?

Answer: some people really need to have passion and some don’t. Everyone needs to be passionate about something, but it may not be work related. People won’t invest in their communication skills if they aren’t passionate.

Question: How much time do we need to invest in our communication?

Answer: If you are given something you need to present in 3 days, it’s probably not high stakes. Categorize what is really important and what isn’t, and fight for the ones that are important. When you are launching your new 5-year vision, or making a big sale, you need to put a lot of time in.

Question: Going back to your “present in person” idea from the beginning, what about globally dispersed teams that don’t meet in person?

Answer: Plan and prepare. She stood up in front of pictures of people to practice so she would talk more like face to face in this online format. Your biggest competitor with virtual presentations is their inbox; if you aren’t more interesting than their inbox, they’ll be reading email. Think about getting their attention back. Break it into very small “Halloween candy size” bites to keep them engaged.

Question: You mentioned investing time in improving communications. What are ways people can invest in their skills?

Answer: Be a consumer of good information. You also need to practice it. They have workshops, other people do too–toastmasters

Question: Is there a time limit on keeping interest?

Answer: Depends on the speaker. Some can hold it for much longer. Emotionally charged content can engage people for longer.

Question: Who is your favorite storyteller?

Answer: Several favorites: Cheryl Sandberg (COO Facebook) is one

Question: Are there differences between people in how interested they are in stories? Are women more interested in stories than men?

Answer: Women may have a higher capacity for emotional content. There are stories as little anecdotes, overall themes, or story structure. You need to know your audience. Emotionally charged content may not work with biochemists. Everyone is human though, and everyone responds to story if it applies.

Question: How many slides should you use?

Answer: It depends. Keep one idea on a slide. If you have 5 ideas on a slide, the audience will read ahead and think you are slow. Slide count doesn’t really affect presentation length; if you click fast, you may have a lot of slides. This was about 75 slides for about 35 minutes of presentation.

Question: What do you do with SMEs who want to include everything in their presentation? How do you help them chunk content into smaller bits?

Answer: Slides are free. It’s not like you’re printing and more slides is more money to print. Sometimes a slide does need more information. They usually do printouts for dense information so they walk away with it rather than trying to cram it on a slide. Put a picture of the handout on the screen and tell people to look at the handout instead of looking at dense text on a slide.

Question: What is the greatest lesson you have learned from a webinar that didn’t go well?

Answer: Technology glitches. She had 25 people in the room, 200 online. It was distracting. She didn’t do a technical walkthrough first. Energy is really hard when you are the speaker and everyone else is muted. You have to keep your own energy very high.

Question: Back to the sailing analogy: how do we use the wind resistance idea to catch the audience’s attention?

Answer: The best way is to grab a few coworkers or the potential audience members. Let them think about ways people might resist. Get people who are comfortable being honest about resistance and reactions.

Question: How do your in person presentations differ from what you do in a webinar?

Answer: She really feeds on audience energy, but she tries to not have much gap. She describes things more visually when presenting online to make up for physical presence.

Question: How do you build this in written materials? Can we use this storytelling in emails or other communication?

Answer: Yes, this can work in other forms of persuasion. Her book resonate follows this form on every page, and then the book follows the form.

Question: Best practices for hybrid live/virtual audiences?

Answer: Make sure the technology works. Acknowledge that people who are calling in are humans too to make them not feel like they are outside looking in.

Before I continue, I want to say that I enjoyed her book, eLearning and the Science of Instruction, and I have found some of the research there valuable. I respect her past contributions to the field.

However, I think Clark didn’t do a very careful review of the literature before writing her post, and I don’t think that one study is enough for her to make such a broad claim dismissing games for learning.

According to Ruth Clark, you didn’t learn anything playing Oregon Trail, Carmen Sandiego or Lemonade Stand

Let’s look at her summary of the research:

The goal of the research was to compare learning efficiency and effectiveness from a narrative game to a slide presentation of the content. Students who played the Crystal Island game learned less and rated the lesson more difficult than students who viewed a slide presentation without any game narrative or hands on activities. Results were similar with the Cache 17 game. The authors conclude that their findings “show that the two well-designed narrative discovery games…were less effective than corresponding slideshows in promoting learning outcomes based on transfer and retention of the games’ academic content” (p. 246).

The research is behind a paywall, of course, but the abstract is online.

Adams, D.M., Mayer, R.E., MacNamara, A., Koenig, A., and Wainess, R. (2012). Narrative games for learning: Testing the discovery and narrative hypotheses. Journal of Educational Psychology, 104, 235-249.

Next, let’s look at how the authors summarize their own work and see how it compares to Clark’s summary (emphasis mine).

Overall, these results provide no evidence that computer-based narrative games offer a superior venue for academic learning under short time spans of under 2 hr. Findings contradict the discovery hypothesis that students learn better when they do hands-on activities in engaging scenarios during learning and the narrative hypothesis that students learn better when games have a strong narrative theme, although there is no evidence concerning longer periods of game play.

Gee, that “under two hours” point seems like an important limitation of the research, maybe one that should have been mentioned by Clark when claiming that games have no value and “don’t teach.”

It’s also possible that there are flaws in the research.

• The research says that the games were “well designed,” but maybe they actually weren’t. Maybe they were “well designed” by the standards of traditional courses, but not by the standards of games. Without seeing the full article, I can’t tell.
• The learners did worse at “retention,” but honestly, I wouldn’t expect a narrative game to be all that effective at helping people memorize content. If retention was the goal, a narrative discovery style game probably was the wrong approach, which brings us back to the previous point about whether the course was well designed for the goals.
• One of the benefits of games for learning is application and behavior change, something this research didn’t measure. I’m not terribly surprised that a game with hands-on practice didn’t help people simply recall information that well. I would have liked to see some measure of how well the learners could apply the concepts. But, as is also typical of Clark’s work, the focus is on whether people recall content, not whether they can apply it. This, strictly speaking, is a limitation of the research and not a flaw, but it is something we should consider when looking at how we apply this research to our work.

I think there’s a case to be made that the games themselves weren’t actually “well designed” as claimed. They didn’t allow for real practice, just a different format for receiving content. In the discussion on this post in the eLearning Guild’s LinkedIn group, Cathy Moore made this observation:

I don’t have access to the full study cited in Ruth’s article, but based on the description of the games in the abstract, (1) they don’t simulate a realistic situation that’s relevant to the learners and (2) they teach academic info that learners aren’t expected to apply in real life. The material was tested on college students in an academic setting, not adults on the job.

By requiring learners to explore (or slog though, in my opinion!) an irrelevant treasure hunt, you’re adding cognitive load or at the least distracting the brain from the content. It seems likely to me that putting the material in a more relevant context, such as using your knowledge of pathogens to protect patients in a hospital, would have changed the results of the study.

As Ruth herself says in the comments to the article, “I think it’s about designing a simulation (which I don’t equate directly to games) in a manner that allows learners to practice job-relevant skills.” Neither of those games let students practice job- or life-relevant skills. They were entertaining and distracting ways of presenting information for a test.

Another limitation is that this research can’t address the question of engagement and completion rates. In the real world, getting people to complete online learning is often a challenge. If your traditional text-based click next slide presentation course has a less than 20% completion rate, then a game that is engaging enough to make people want to finish and gets completion rates above 90% is a big improvement—even if that game technically produced lower retention rates in a controlled lab environment. Learning doesn’t always have to be drudgery, although sometimes we equate “worthwhile” with “unpleasant.” There is value in making it interesting enough to keep people’s attention, and maybe even an enjoyable experience.

In the previously mentioned discussion, Tahiya Marome made this point:

For the brain, play is learning and learning is play. That traditional educational structures have sucked that dry and replaced it with a grim Puritanical work is learning and learning is work structure doesn’t mean we have to leave it that way. It may take us a while to figure out exactly how, but we can make educating oneself playful and a great, life long game again. We can. Our brains are wired for it.

Clark has some legitimate points about the definition of games being fuzzy and that the design of the game should match the learning outcomes. For example, I agree with her that adding a timer to critical thinking tasks can be counterproductive. Adding a timer to skill practice for skills that really do need to be timed is good practice though. Think of help desk agents who are evaluated both on the quality of their service and how quickly they can solve problems; timed practice matches the learning outcomes.

If Clark is going to make the claim that “games don’t teach,” she needs to address all the research that contradicts her point. She makes this claim without even mentioning any of the other research.; she just pretends nothing else exists beyond the one study cited. That is, frankly, an extraordinary claim, and extraordinary claims require extraordinary evidence. One study doesn’t discount the dozens of successful examples out there. It’s bad use of research to treat any individual study as applying in all situations, regardless of the limitations of the study. What we need to look at is the trends across the bulk of research, not a single data point. There are definitely bad games out there, and games aren’t the solution in every situation, but that doesn’t mean games shouldn’t be one tool in our toolbox like Clark claims.

Here’s a cursory review of a few examples of successful games for learning. This is by no means a comprehensive review, but this would be a good place for Clark to start refuting evidence if she wants to dissuade people from using games. Again, the point is not to look at any single study as being the end of the discussion, but to look at the overall findings and the types of strategies that have repeatedly been shown to work.

• Immersive games beats classroom in maths, summarized by Donald Clark
• Via Karl Kapp, from a past presentation:
  • “Trainees learn more from simulations games that actively engage trainees in learning rather than passively conveying the instructional material.”
  • “Trainees participating in simulation game learning experiences have higher declarative knowledge, procedural knowledge and retention of training material than those trainees participating in more traditional learning experiences.”
• Eduweb has a collection of research related to games for learning. Here’s a highlight from the findings of one paper: “Summative evaluation of our WolfQuest wildlife simulation game finds that players report knowledge gain, stronger emotional attachment to wolves, and significant behavioral outcomes, with large percentages of players following their game sessions with other wolf-related activities, including such further explorations of wolves on the internet, in books and on television.”
• Kurt Squire has done extensive research in games for learning. Clark basically needs to disprove all of his work to support her claim.
• Clark Aldrich has created a number of successful games and simulations, such as Virtual Leader. “Using practiceware significantly increased retention and application, not just awareness of learned content.”
• James Paul Gee has published a number of articles on games and learning.
• Mark Wagner’s dissertation on MMORPGs in education found that “MMORPGs may help students develop difficult to teach 21st Century skills and may be used to support student reflection.”
• The Educational Games Research blog features exactly what you would think it does based on the title.

Thanks to Cathy and Tahiya for giving me permission to quote them here!

I’d love to hear if any of you out there have designed games for learning and found them to be effective or not. I’ll have more to say about this topic next week in my post for the blog book tour for the Gamification of Learning and Instruction.

I ended up putting all the “chunks” of content into boxes on a PowerPoint slide and dragging and dropping until I was happy with it. The orange blocks are videos; the blue blocks are content pieces. The one white box was an optional piece I debated whether to cut. (Note that the specific content labels here are unlikely to make much sense, since I removed a number of identifying details for this post. Ignore the specific content and just think about the development process.)

This worked really well for me, and got me “un-stuck.” I could have done the same sort of organization with index cards, but PowerPoint was handy. It also has the advantage of being easily saved and edited at a later date. I suppose with index cards you could take a picture or just transcribe everything, but that seems like too much hassle to me. This was quick and dirty, but it got the job done. I have also found this technique useful when working remotely with SMEs. Bring up a PowerPoint slide in your web conferencing software and drag and drop live while you’re on the phone.

However, I know sometimes the tactile experience can be helpful. When I wrote the branching video at the end of the above plan, I ended up writing my first draft in a notebook instead of on the computer. I’m very comfortable composing at the keyboard, but sometimes for creative writing like that storyline, I still want that physical sensation of a pen in my hand. I know a local author who recently tried and then abandoned software for planning a novel. She has returned to organizing her work with sticky notes on a large storyboard. That tactile work is part of her process.

I’m curious what other instructional designers do to organize content. Do you just reorder the text in Word? Do you use something visual like PowerPoint or a mind map? Do you use something physical like index cards? Is there another method for this process that I haven’t thought of? Please take a few seconds and answer this one-question poll. (If you’re reading this in email or RSS, you may need to visit my site to answer the poll.) If you have another process, please share!