The first paper by Douglas Clark and team does an excellent job reviewing the “state of the state” of games and simulations. A read through this will give you a good grounding in where we are with the use of simulations and gaming in education. My only quibble is that it appears to leave out the medical and military advances in simulations and gaming, which are extensive. It may be that the charter for this paper was to focus on K-12, which it covers very well. Just keep in mind that there is a little more out there when it mentions the dearth of options in post-secondary education.

The paper by Quellmalz et al is also a good introduction to both the assessment of learning in games and simulation and the use of games and simulation as assessment.

The final group on the list addresses the problems of bringing these games and simulations to scale. there has been work going on in this area for at least the last decade, but there really have only been a few examples of games that have really reached scale. And of those, the impact on learning is debatable (are they just pleasant diverisons?). The authors of the last papers argue that too often academics are not equipped with the skills necessary to bring NSF-funded prototypes to market, and planning for this move is not done up front.

All-in-all, a good group of papers if this is an area of interest for you.

Jaakkola & Nurmi (2007) looked at the possibilities of combining simulation and laboratory. They placed elementary students in one of three groups: computer simulation, lab exercise, or a simulation-laboratory combination. The groups completed an electricity lab along with pre- and post-tests. Controlling for pre-test performance, the combination group had both the highest post-test scores and the smallest variability around those scores.

One reason the authors believe the combination group had better outcomes was that the simulation allowed students to visualize electrical processes that are otherwise not observable. On the other hand, it was only semi-realistic in representing circuits, so the hands-on lab filled in those gaps. The authors state:

A simulation can help students to first understand the theoretical principles of electricity; however, in order to promote conceptual change, it is necessary to challenge further students’ intuitive conceptions by demonstrating through testing that the laws and principles discovered through a simulation also apply in reality.

Jaakkola, T., & Nurmi, S. (2008). Fostering elementary school students understanding of simple electricity by combining simulation and laboratory activities Journal of Computer Assisted Learning, 24 (4), 271-283 DOI: 10.1111/j.1365-2729.2007.00259.x

I’ve been a proponent of the open source model of journal publishing since serving as an editor to Current Issues in Education as a graduate student. However, text book writing is hard work, and authors of textbooks do usually get paid (as opposed to authors of journal articles). Just like the music and movie industries, however, it looks like experimenting with new revenue models may be in order. However, the textbook companies that sell e-textbooks don’t seem to be reducing the costs for these books; a study cited in the report indicates that the e-textbooks cost the same, on average, as new books purchased in the bookstore and sold back, and twice a used book bought and sold back. Good grief! That’s not going to do…

Texas is allowing professors at post-secondary institutions to create open source texts, California (deep in its own budget crisis) has indicated that 11 open source texts met 90% of the state math and science standards.

If you’ve never seen it before, Connexions is a big commons of open source educational material… definitely worth a look.

The times, they are a changin’…

Ionna Lykourentzou and collegues, working in Greece, reported in Computers & Education their attempt to use machine learning to do just that. The hope is that this capability would then lead to the ability to intervene with the at-risk students. They also analyze their results in terms of accuracy, sensitivity, and precision so that the analysis takes into account both false positives and false negatives.

Machine learning generally involves the creation of algorithms that allow computers to learn from data. In this context, computers are fed data from existing classes with student variables and eventual completion or dropout of the students known. The computer then uses its algorithms to create a model that will best predict these known outcomes. Following this, new data, without outcomes is entered into the computer. The computer is then asked to predict, based on its previous learning, whether students will dropout or complete. These predictions can be compared to the actual results to understand the effectiveness of the computer model. How good was it at predicting?

Lykourentzou et al looked at three different methods of machine learning. Then, they looked at ways to combine the results of those methods, since each by itself may not be perfect. This asked the question, is it better, for example, to classify a student as a dropout if at least one of the methods classified them as such, or should we require two of the three methods to classify them as a dropout to code them as such?

The authors included gender, geographical location, work experience, education level, English language literacy, test grades, project grades, project submission date (e.g., was it late?), and number of sessions in the online environment. The accuracy of the predictors was judged at each of the seven sections of the course (more information, such as grades, was available at the end of each section). Each of the predictors ended up being important in the models, but time invariant data (such as demographics; data that does not vary through the course) was less accurate than time varying data (such as grades).

With all of the variables and the machine learning techniques, it was determined that the best way to predict dropout was to run all three machine learning techniques and classify a student as a dropout if any of the three indicated that they were classified in that category. This technique achieve accuracy ratings between 97 and 100%, sensitivity rates between 95 and 100%, and a precision rate of 100%.

There are some drawbacks to these methods. Predictions are dependent on the quality and representativeness of the training data. So, if a previous semester’s data is used to train the algorithms and there were some very idiosyncratic events in either that semester or this semester, prediction will not be as good. However, with these complex techniques running in the background, fairly simple interfaces could be created for teachers to highlight at-risk students for intervention.

Perhaps this is another way we can begin to address the dismal dropout numbers at post-secondary institutions, particularly in online courses.

Lykourentzou, I., Giannoukos, I., Nikolopoulos, V., Mpardis, G., & Loumos, V. (2009). Dropout prediction in e-learning courses through the combination of machine learning techniques Computers & Education, 53 (3), 950-965 DOI: 10.1016/j.compedu.2009.05.010

The use of help features in computer-based learning has been an issue of recent research. Learning outcomes appear to be at least partially dependent on available support, and help-seeking on the part of students is seen as a positive sign of self-regulated learning. How do students ask for help? Are there developmental changes in this asking?

These were the questions explored by Minna Puustinen and colleagues in a recent article in Computers & Education. They examined 206 messages sent by middle school students via an online forum to math tutors.

The questions were coded based on whether they included:

• A description of the problem the student was trying to solve
• Explicit request for help
• Signs of preliminary personal work
• Openings
• Closings
• Student introduction/ identity
• Context (e.g., “I’m working on homework”)
• Politeness markers

The results indicated clear differences in the complexity of the help requests across grades. More than half of 6th graders’ messages only included one of the above categories, while 70% of the 9th graders’ messages contained three to six of the categories. Examination of the frequency of categories indicated that older students’ messages were more likely to contain context-related information and explicit requests for help. These results indicate that the messages of younger students do not seem to contain enough information to receive help (a description of the problem and explicit request for help would appear to be a minimum).

The implications of these findings are interesting. First, when designing computer-based lessons for the middle school age group, the changing nature of their help requests must be taken into account. They are not, in fact, a homogeneous group in this arena. For younger students, specific prompts might be given for the types of information needed in order to get the most from a help session. In a computer-based help situation, younger students may need scaffolding about the kinds of information they need to search for. I think the developmental implications are interesting here… is this a phenomenon related to taking another’s point of view?

I just used an online help chat feature with a problem I was having with the Ebay site, at least I would get marks for opening, closing, and politeness. I wonder what analysis of a bunch of those transcripts would look like for adults!

Puustinen, M., Volckaert-Legrier, O., Coquin, D., & Bernicot, J. (2009). An analysis of students’ spontaneous computer-mediated help seeking: A step toward the design of ecologically valid supporting tools Computers & Education, 53 (4), 1040-1047 DOI: 10.1016/j.compedu.2008.10.003

Is statistics funny? Neumann, Hood, & Neumann (2009) think they can make it funny, and sought to find out how students reacted to including humor in statistics classes. (It should be noted that this article is from a Journal of Statistics Education “Research to Practice” article in which the authors took another research article, tried it in their class, and are reporting on the results… pretty neat.) They looked at two semesters of classes with over 200 students and interviewed approximately 20 students from a stratified sample.

The authors give examples of the kinds of humor used in the course, consisting primarily of cartoons and visual images. For example, “Introducing the first statistical formulas can often be a daunting experience for non-mathematical students. Students were encouraged to be brave when dealing with formula by showing a picture of a squirrel “undressing” its fur to show the S of the Superman suit.”

What did the students think? They were asked about the positive and negative aspects of humor in the classroom. In general, responses fell into affective effects of humor (amuses, lightens mood, motivates to attend class) and cognitive (helped learning, helped maintain attention, provided mental break, reduces monotony). Comments were more positive than negative (although the questions asked appear to be more likely to yield positive responses).

It’s probably not surprising that students reported enjoying humor. It would be interesting to try some more controlled studies, but humor is so individual that it would be hard to tell a teacher who uses it not to use it or vice versa. But it would be interesting if teachers who were independently rated as “funnier” (by humor experts??) had better class attendence.

It’s also interesting that some students saw humor as a way to bring them back to attention while others saw it as providing a mental break from the content. Those seem like opposing results, I suppose depending on the students’ state prior to the joke.

Finally, are some types of humor more effective? This study focused on visual humor and unexpectedness, what about other types of humor? (Oh, and hit me up if you’ve got any jokes about statisticians.)

Neumann, D. L., Hood, M., & Neumann, M. M. (2009). Statistics? You Must Be Joking: The Application and Evaluation of Humor when Teaching Statistics Journal of Statistics Education, 17 (2)

Currently, international testing is done by random sampling nationally. No effort is made to get representative samples within a state and conclusions cannot be made about performance in a given state. So, is it really worth the estimated $700,000 to be able to say Massachusetts students perform better than students in Spain? (OK, so maybe that’s a loaded question, but that’s an estimate of what it would take to test a valid sample of 1,500 students in Massachusetts on the PISA.)

Mark Schneider (now a VP at American Institutes for Research, formerly commisioner of the National Center for Educational Statistics) says this is probably not worth the effort. First, the gains in terms of policy knowledge for a given state would probably be limited. I agree, this is not the kind of test by which to evaluate the effectiveness of policy changes.

Dr. Schneider suggests a much cheaper solution would be to use statistical methods to link the international exams to existing state exams. A sample of PISA items could be added to a state assessment and then used to link the state assessment to the PISA scale. Once this is done, a state could compare itself to PISA results. Schneider notes:

While the alternatives [statistical linking] would not produce all the details that might come from the full assessment (and mercifully avoid the temptation to use these data for unwarranted policy analysis), they could produce reliable estimates of state performance relative to international performance.

There are other, more statistically complex, procedures that could also be used that are also further away from having all of the students take the test. While many of us in research think these alternatives make sense if states really want that comparison, I think there is still the general distrust of statistics to deal with. I know when I worked in a local school district, teachers were very skeptical of the state average yearly progress scores for the school because they didn’t understand the regression modeling that went into it. We need to be better at 1) communicating complex statistics simply and 2) building trust in statistical methods.

At a high level, they have groups of students randomly assigned to work on physics problems in triads on either well-structured or ill-structured problems. The ill-structured problems had more problem parameters (i.e., many variables included in the problem statement that were irrelevant to the problem solution), variables with less specificity (e.g., ranges that require estimation or opinion), and interactions among variables in the problem.

The study is a tightly designed with random assignment experiment. Students took preassessments individually, then worked on either ill-structured or well-structured problems in triads. They then individually completed post-tests of both ill-structured and well-structured problems.

The researchers examined the chat logs and performance on the group worked problems, as well as performance on the individual post-tests. Analysis of the group problems revealed that the solutions of the well-structured groups  were significantly better than the solutions of the ill-structured groups. Analysis of the interactions in the groups revealed that the ill-structured groups spent significantly more time in problem definition activities while the well-structured groups spent more time on solution generation. The authors write, “IS [ill-structured] groups found it difficult to converge on the causes of the problem, set appropriate criteria for a solution, and actually develop a solution, resulting in poor group performance.”

Now, the interesting part: on the individual post-tests, the students who were members of the ill-structured groups performed better on both ill-structured and well-structured problems. So what is the explanation? The authors write:

Indeed, what separated the interactional dynamics of IS from those of the WS groups was a focus on problem analysis and criteria development, as well as sustained problem critique and solution evaluation with a number of transitions and feedback loops. Although seemingly unproductive and leading to failure in the shorter term, a more complex and divergent exploration of the problem and solution spaces as evidenced by the emergence of a diversity of interactional sequences was what differentiated the interactional dynamics of IS groups from those of WS groups.

In other words, working through the ill-structured problems taught the students problem-solving strategies that then transferred to other problems, leading to greater success on those problems. Very interesting! A couple things next: can we show improvement in problem solving skills? Make the explanation explicit? If this is a solid finding, as it seems to be, how frustrating is too frustrating? Is there some point at which students are given problems that are so ill-structured that they give up? The authors emphasize that they do not support giving only completely ill-structured problems to students, but they certainly suggest that work with these problems is beneficial. 

Kapur, M., & Kinzer, C. (2008). Productive failure in CSCL groups International Journal of Computer-Supported Collaborative Learning, 4 (1), 21-46 DOI: 10.1007/s11412-008-9059-z

A group of researchers studying teachers in Switzerland (hey! a non-US study!) conducted a survey of 93 teachers of French as a second language. Their survey included scales measuring endorsement of homework as: a drill and practice activity, a motivation activity, a way to particularly help low-achieving students, and a way to establish a school-home link. In general, teachers more strongly endorsed the Drill and Practice scale and endorsed the school-home link the least. Interestingly, teachers did not appear to find the different objectives to be antagonistic, with many endorsing, for example, both drill and practice and motivation.

The survey also asked about parent participation in homework and found the majority of teachers showed support for student homework autonomy. Interestingly, there was a .29 correlation between drill and practice and endorsement of parent homework completion control (i.e., having parents monitor completion), while the correlations between parent homework completion control were negative (achievement gap r=-.12) or near-zero (school-home link and motivation) for other homework purposes. That seems a little counter-intuitive to me… wouldn’t drill and practice be most beneficial if controlled by the student?

To continue, the researchers also had students complete surveys about their effort and achievement tests, and conducted multilevel analyses to examine the effects of teacher homework attitudes on student effort. Go ahead, guess which purpose was associated with negative homework effort and achievement. Yup, drill and practice. Also, the more teachers saw homework as a school-home link and tried to involve parents, the lower their students’ homework effort and achievement.

This study definitely suggests a link between teacher homework attitudes and student outcomes. I struggle with the drill and practice debate. On one hand, I really see the need to increase students’ automaticity with basic skills… and drill and practice is a way to do that. On the other hand, does this practice actually decrease effort and achievement? We need a way to practice skills to the point of automaticity that is also motivating and interesting… educational technology anyone?
Trautwein, U., Niggli, A., Schnyder, I., & Lüdtke, O. (2009). Between-teacher differences in homework assignments and the development of students’ homework effort, homework emotions, and achievement. Journal of Educational Psychology, 101 (1), 176-189 DOI: 10.1037/0022-0663.101.1.176

Diane Schallert and her colleagues have been investigating politeness on discussion boards, and in an article in Computers & Education look at relationships between discourse functions, synchronous/ asynchronous sessions, and politeness. They analyzed 3 synchronous and 3 asynchronous discussions of a class of 24 students. Of some note, the class was a graduate level psycholinguistics class and the students all had 12 face-to-face meetings. The psycholinguistic class content is important because these students may be more aware of their language use than the average student. The in-person meetings are important to note because these conversations all happened between people who also have face-to-face contact, which may influence how polite they are on discussion boards. The researchers examined a total of 1475 messages.

Before looking at politeness, the researchers looked at discourse functions in synchronous versus asynchronous discussions. They report that communication in the asynchronous discussions was more formal and serious. 65% of their moves focused on content, versus 58% for the synchronous sessions. The asynchronous chunks also tended to be longer. The synchronous sessions contained more social comments (14% versus 5% for synchronous).

Interestingly, when the researchers turned to politeness, they found that students were equally polite, using similar numbers of polite moves in both synchronous and asynchronous sessions. Approximately 56% of the discourse chunks across both types of sessions contained politeness moves. That seems to confirm my impression that students are relatively polite on classroom boards.

Like a lot of research, this certainly raises more questions:

• Would similar results hold in different content areas?
• Would you get similar results if students did not meet face to face? 
• What is the opposite of politeness? Harshness? What was the frequency of those comments?
• Is there any relationship between politeness and learning? Yang et al (2006) in the Yearbook of the National Reading Conference suggested politeness might interfere with learning.

Schallert, D., Chiang, Y., Park, Y., Jordan, M., Lee, H., Janne Cheng, A., Rebecca Chu, H., Lee, S., Kim, T., & Song, K. (2009). Being polite while fulfilling different discourse functions in online classroom discussions Computers & Education, 53 (3), 713-725 DOI: 10.1016/j.compedu.2009.04.009