1. Thinking it could be a small shift or change in the way we did things. It took time to explain what we were doing, why, and how. But that time was well spent. The failure here was a failure in scaffolding for using the system. It requires some dedicated time, just as any challenges to existing structure do. Now I devote a good part of the first week to using the system.

2. Too many little badges. I started out with a pattern seen on a lot of websites: make the first badges easy. I still have a “how to get a badge” badge. (Actually, it is the “Human” badge–pictured above and fairly easy to get.) But I’ve moved to fewer, more substantial badges. Several reasons for this. First, a more substantial badge is valued differently. It carries more weight. Second, because there is a certain amount of overhead for earning a badge, it makes sense to chunk things a bit larger. Getting that balance right is key, and not easy.

3. Conflated badges. This is the other side of the above. In one course I had a badge for basic blogging, in which you were required to post on a self-hosted blog (not, e.g., Blogger or WordPress.com), because I wanted to know they could get hosting, set up a domain, and install the software. In another course, this wasn’t as essential, but I had to do another “basic blogging” badge. In other words, I ended up with badges that were bound to the course, when I wanted to avoid that. Again, hard to get away from rolling too much in if you are doing substantial badges.

4. Identification issues. This is both technical and a policy issue, but I’ve gone through a number of authentication processes. I’d love to federate with the university’s system, but also want to let others in. I’ve considered Facebook Connect, or OpenID, or something, but just need to settle on something. Because it will aid with the Open Badge Infrastructure, that will probably be Persona (née BrowserID). There is a part of me that would like to see this integrated into Blackboard, and a much, much larger part urging me to resist the dark side.

5. Bad peer reviews. I actually talked a bit about this in the previous post. It’s really hard to get people to post more than a word or two. Need to figure out some way to reward those who do, and encourage the practice. (Yes, extrinsic reward, but how else will they come to understand the intrinsic value of critique?) Maybe a system by which the recipient meta-judges the critique? Perhaps the number of words of critique you type leads to a bank of credit for getting your own critiques and endorsements? I don’t know.

6. Badly written rubrics. I’ve always hated rubrics, thusly I’m bad at writing them. I’ll need to find some great examples to use.

7. End of the semester rush. I also have no deadlines. I like it that way, but I don’t like the mad rush at the end of the semester. I’ll have grades moving forward, and they will be based on maintaining steady progress toward the end of the semester.

There’s a lot wrong in the details of the technical implementation (and even more in terms of the actual code) but these are some of the issues I had at the broader design level.

Related posts:

1. Brief Introduction to BadgePost Prototype
2. BlogPost Progress Report: peer assessment
3. What makes up a badge?

It doesn’t really work, but it works well enough, and thanks to some patient and very helpful students, I now know a great deal more about how badges can work in higher education. I make no claim to my successes being best practices, but I at least know more now than when I started, and figured I would share some of this experience.

Why did you do that?

More than a decade ago, I coded my first blog system for a course, though the term was not widely used then. I did it because there were particular kinds of interactions I wanted to encourage, and existing applications didn’t do quite what I wanted them to. I created my BadgePost system for the same reason. I am not really a coder (I dabble) but what I wanted did not exist, and so I took a shot at prototyping something that might work. (As an aside, I also hope that what happened with blogs happens with badges, and I can download the equivalent of WordPress soon instead of having to roll my own.) I knew I wanted:

Peer assessment. I wanted to get myself out of the sole role of sole reviewer. In many cases peers can give better advice than I can. One of the main difficulties of teaching is rewinding to the perspective of the student, and that can be easier, in some cases, for those who have just learned something. I wanted to enable that kind of open peer review in both hybrid courses and those taught entirely online.

Mastery. I also wanted desperately to get away from letter grades, as they seemed like a plague, not just for undergrad courses, but for grad as well. Students seemed far more interested in the grade than they were in learning something, a refrain I’ve heard frequently from a lot of my colleagues. I wanted to move the focus off of the grade.

Peers as cases. Students often ask me for models of good work, and because I change assignments so frequently, I rarely have a “model.” The advantage to open assessment that travels beyond a single course is that there are exemplars to look at, and (hopefully) they are diverse enough not to stifle creative interpretations by new students.

Unbundling the credential from the course. I had a number of problems that seemed to swirl around the equation of course time to learning objectives. For one, in the required technical courses, some people came in with nothing and others with extensive knowledge, and I wanted to try to address the issue of not all students moving through a program in lock-step. I wanted a back door to reduce redundancy and have instructors know that their students were coming into a course with certain skills. Finally, I wanted to give students a range of choices so that they could pursue the areas they were most interested in.

I also wanted non-paying non-Quinnipiac students participating in my courses to have a portable credential to show for it. And I wanted paying, matriculating students to have an easier way of communicating the kinds of things they had learned in the program.

I won’t cover all of these in detail, but will expound a bit more on the assessment and assessing piece…

Peer Assessment

There have been suggestions that the credentialing aspect of badges is separate from the process of assessment that leads to the badge, but in practice I think it’s both likely that they get rolled together, and beneficial when they are. Frankly, students don’t see the distinction, and they can reinforce each other in interesting ways. So, while I have done peer critique in the past, from the outset here, I wanted to get students involved in the process of granting badges via peer critique.

A lot of this was influenced by discussions with Philipp Schmidt and the application of badges in Peer2Peer University. I have long stated the goal of “disappearing” as an instructor in a course, and the place where that appearance is most obvious is when it comes to grading. (And assessment, not the same thing, but bound together.) From the outset, I saw the authority of a badge as vested in the material presented as evidence of learning, and the open endorsement/assessment of that work by peers.

Lots of reasons for this, but part was as a demotivator. That is, my least favorite question on the first day of classes is “how do I get an A?” I am always tempted to tell the truth: “I don’t care, and I wish you didn’t either.” So, I wanted badges to provide a way of getting away from that linear grading scale. I went so far as to basically throw grades out, saying that if you showed up on something approaching a regular basis, you’d get an A.

I should say that this was a failure. If anything, students paid more attention to grades because the unique system made them have to think about it. It wasn’t onerous, but a lot more of the course became about the assessment process. And it’s funny, my desire to escape grading as a focus and process turned a 180, and I am now all about assessment. I should explain…

I hate giving traditional tests (I don’t think they show anything), and hate empty work. And while I now know I like ideas around authentic assessment, from the outside these seemed a lot like more of the same. Now, not only do I think formative assessment is the key element of learning, but that the skill of assessing work in any field is what essentially defines expertise. Being able to tell what constitutes good work allows you to improve the work of others, and importantly, of yourself. At the core of teaching is figuring out what in a piece of work is good, what needs improvement, and how the creator can improve her work.

Beyond Binary

I had expected students to do the work, apply for a badge, and then either get it or not. A lot of other people new to badges seem to have a similar expectation. Just the opposite occurred, and a lot of the changes to my badge system have been to accommodate this.

First, a lot of work that really was not ready for a badge was submitted. I kind of expected students to be very sure of the work that they submitted for a badge, in part because of my experience with blogging in classes, and seeing that students were more careful about their writing when it was for a peer audience. Instead, students often presented work that was not enough for a badge, or barely enough for a badge. I was pleasantly surprised by how much feedback, and in what detail, students gave to their peers.

One of the more concrete changes I made to the system was to move from a binary endorsement (qualified or not, on a number of factors), to a sliding scale, with the center point being passing, and the ability of reviewers to come back and revise their “vote.” As a result, you can see from the evidence of a badge not just what the student has done, but whether their peers thought this was acceptable or awesome.

I’ve also been surprised by how many nominated themselves for “aspirational” badges. When a user selects a badge, it is moved into their “pending” category, and I was confused by so many pending badges that had no evidence uploaded. But students seem to click on these as a kind of note to themselves that this is what they are pursuing. This, incidentally, leads to a problem for reviewers who look at a pending badge before it is ready, and find that process frustrating, but one of the things that needs to improve in the system is communicating such progress. I didn’t plan to need to do that, since I saw badges as an end point rather than a process.

The Reappearing Teacher

The other surprise was just how interested students were in getting my imprimatur. But the reason, in this case, was not the grade–they had that. They actually valued my response as an expert a bit more, I think. This was a refreshing change from students turning to the back page of graded paper to see the grade, and then throwing it out before reading any of my comments. No doubt, some of this comes from a lack in confidence in their peers as well, and I’ve found that in some cases this lack is reasonable.

In some ways, I’m trying to encourage the sempai/kohai relationship, of those who have “gone before” and therefore have more to say about a particular badge. I’ve been reluctant to limit approval to only those who actually have the badge (in part for reasons I’ll note below regarding encouraging reviews), but I may do more of that. There are some kinds of assessment, though, that don’t require having the badge. I don’t need to know how to create a magic trick to be amazed by it, for example. So I don’t want to rule out this kind of “audience assessment.” There is also space for automated assessment. For example, for some badges you need to show a minimal number of tweets, or comments, or responses to comments, or (e.g.) valid HTML. There is no reason to have a human do these pieces of the assessments, though I would hate to see badges that did not involve human assessment, in large part because, again, I think building the capacity to do assessments is an important part of the system.

The Other Motivation

I began by hoping students would ignore the grading process, and have evolved to think that they should pay a lot of attention to assessment. In some courses, students have jumped into peer assessment. In others–and particularly the undergraduate course I’m teaching this semester–they were slow to get started. I want to think about why people assess, and how to motivate them to be involved.

When I did peer assessments in the pre-badge world, I assigned a grade for the quality of the assessment provided. I want to do something similar here, and a lot of this comes of a discussion with Philipp Schmidt in Chicago last year. The meta-project here is getting students to be able to analytically assess work and communicate that. Yes, you could do an “expert assessor” badge, or something similar, but really it is more essential to the overall project.

One way to do this is inter-coder reliability. If I am considered an expert in the area (and in the current system, this is defined as having badges at a higher level than the one in question, within the same “vertical”), those with less experience should be able to spot the same kinds of things I do, and arrive at a similar quantitative result on the assessments.

So, for example, if someone submits the write-up of a content analysis, two of her peers might look at it and come up with two very different assessments of the methods section of the article. Alice may say that it is outstanding, 90/100 on the scale of a particular rubric. Frank might disagree, putting it at 25/100. Of course, both would provide some textual explanation for why they reached these conclusions. Then I come along and give it a 30/100, along with my own critique.
The dynamics of getting students to do peer assessments (some courses they did a lot, some they have not), and my involvement in the assessment, is an interesting piece for me. In this case, Frank should receive some sort of indication within the system that he has done a good job of performing the assessment.

I’m still working out a way to do this that isn’t unnecessarily complex. Right now there is a karma system that gives users karma for performing assessments, with multipliers for agreeing with more experienced assessors, but this is complicated to “tune” and non-intuitive.

There is also the issue of when various levels perform the assessment. For the above process to work, Alice and Frank both need to get their assessments in before I do, and shouldn’t get the same kind of kudos for “me too” assessments after the fact.

Badges

None of this is necessarily about badges, but it leaves a trail of evidence, conversation, and assessment behind. One of the big questions is whether badge records should be formative or summative. As I said, the degree to which students have engaged in badges as a process rather than an outcome came as a bit of a surprise to me. Right now, much of that process happens pretty openly, but I can fully understand how someone well on in their career may not want to expose fully their learning process. (“May” is operative here–I think doing so is valuable for the learning community!)

On the other hand, I think badges that appeal to authority undermine the whole reason badges are not evil. Badges that make an authoritative appeal (“Yale gave me this badge so it must be good.”) simply reinforce many of the bad structures of learning and credentialing that currently exist. Far better is a record of the work done to show that you understand something or can do something, along with the peers that helped you get there, pointed to and easily found via a digital badge.

Balancing the privacy needs with the need to authentically vest the badge with some authority will be an interesting feat. I suspect I may provide ways of hiding “the work” and only displaying the final version (and final critiques) to the outside world, while preserving the sausage-making process for the learning community itself. But this remains a tricky balance.

Related posts:

1. Badges: The Skeptical Evangelist
2. Badgepost Failures
3. Free Range Assessment

So begins an article in what has become one of the truthiest sources of news on the web. But it is no longer time for mid-semester evals. In most of the US classes are wrapping up, and professors are chest-deep in grading. And the students–the students are also grading.

Few faculty are great fans of student evaluations, and I think with good reason. Even the best designed instruments–and few are well designed–treat the course like a marketing survey. How did you feel about the textbook that was chosen? Were the tests too hard? And tell us, were you entertained?

Were the student evals used for marketing, that would probably be OK. At a couple of the universities where I taught, evals were made publicly available, allowing students a glimpse of what to expect from a course or a professor. While that has its own problems, it’s not a bad use of the practice. It can also be helpful for a professor who is student-centered (and that should be all of us) and wants to consider this response when redesigning the course. I certainly have benefited from evaluations in that way.

Their primary importance on the university campus, however, is as a measure of teaching effectiveness. Often, they are used as the main measure of such effectiveness. Especially for tenure, and now as many universities incorporate more rigorous post-tenure evaluation, there as well.

Teaching to the Test

A former colleague, who shall remain nameless, noted that priming the student evals was actually pretty easily done, and started with the syllabus. You note why your text choice is appropriate, how you are making sure grading is fair, indicate the methods you use to be well organized and speak clearly, etc. Throughout the semester, you keep using the terms used on the evals to make clear how outstanding a professor you really are. While not all the students may fall for this, a good proportion would, he surmised.

(Yes, this faculty member had ridiculously good teaching evaluations. But from what I knew, he was also an outstanding teacher.)

Or you could just change your wardrobe. Or do one of a dozen other things the literature suggests improves student evaluations.

Or you could do what my car dealership does and prominently note that you are going to be surveyed and if you can’t answer “Excellent” to any item, to please bring it to their attention so they can get to excellent. This verges on slimy, and I can imagine, in the final third of the semester, that if I said this it might even cross over into unethical. Of course, if I do the same for students–give them an opportunity to get to the A–it is called mastery learning, and can actually be a pretty effective use of formative assessment.

Or you could do what an Amazon seller has recently done for me, and offer students $10 to remove any negative evaluations. But I think the clearly crosses the line both in Amazon’s case and in the classroom. (That said, I have on one occasion had students fill out evals in a bar after buying them a pitcher of beer.)

It is perhaps a testament to the general character of the professoriate that in an environment where student evaluations have come to be disproportionately influential on our careers, such manipulation–if it occurs at all–is extremely rare.

It’s the nature of the beast, though: we focus on what is measured. If what is being measured is student attitudes toward the course and the professor, we will naturally focus on those attitudes. While such attitudes are related to the ability to learn new material, they are not equivalent.

Doctor Feelgood

Imagine a hospital that promoted doctors (or dismissed them) based largely on patient reviews. Some of you may be saying “that would be awesome.” Given the way many doctors relate to patients, I am right there with you. My current doctor, Ernest Young, actually takes time to talk to me, listens to me, and seems to care about my health, which makes me want to care about my health too. So, good. And frankly, I do think that student (and patient) evaluation serves an important role.

But–and mind you I really have no idea how hospitals evaluate their staff–I suspect there are other metrics involved. Probably some metrics we would prefer were not (how many patients the doctor sees in an hour) and some that we are happy about (how many patients manage to stay alive). As I type this, I strongly suspect that hospitals are not making use of these outcome measures, but I would be pleased to hear otherwise.

A hospital that promoted only doctors who made patients think they were doing better, and who made important medical decisions for them, and who fed them drugs on demand would be a not-so-great place to go to get well. Likewise, a university that promotes faculty who inflate grades, reduce workload to nill, and focus on entertainment to the exclusion of learning would also be a pretty bad place to spend four years.

If we are talking about teaching effectiveness, we should measure outcomes: do students walk out of the classroom knowing much more than they did when they walked in? And we may also want to measure performance: are professors following practices that we know promote learning? The worst people to determine these things: the legislature. The second worst: the students. The third worst: fellow faculty.

Faculty should have their students evaluated by someone else. They should have their teaching performance peer reviewed–and not just by their departmental colleagues. And yes, well designed student evaluations could remain a part of this picture, but they shouldn’t be the whole things.

Buffet Evals

I would guess that 95% of my courses are in the top half on average evals, and that a slightly smaller percentage are in the top quarter. (At SUNY Buffalo, our means were reported against department, school, and university means, as well as weighted against our average grade in the course. Not the case at Quinnipiac.) So, my student evals tend not to suck, but there are also faculty who much more consistently get top marks. In some cases, this is because they are young, charming, and cool–three things I emphatically am not. But in many cases it is because they really care about teaching.

These are the people who need to lead reform of the use of teaching evaluation use in tenure and promotion. It’s true, a lot of them probably like reading their own reviews, and probably agree with their students that they do, indeed, rock. But a fair number I’ve talked to recognize that these evals are given far more weight than they deserve. Right now, the most vocal opponents to student evaluations are those who are–both fairly and unfairly–consistently savaged by their students at the end of the semester.

We need those who have heart-stoppingly perfect evaluations to stand up and say that we need to not pay so much attention to evaluations. I’m not going to hold my breath on that one.

Short of this, we need to create systems of evaluating teaching that are at least reasonably easy and can begin to crowd out the student eval as the sole quantitative measure of teaching effectiveness.

Related posts:

1. Media law student evals
2. More on ratings
3. Student evals for porn class

Fitzpatrick’s book is a clear and well-thought out response to what is widely accepted as a deeply dysfunctional form of scholarly dissemination: the monograph. In the introduction, Fitzpatrick suggests that modern academic publishing in many ways operates via zombie logic, reanimating dead forms, feeding off of the living. As a result, it is tempting to conclude that the easiest way to deal with academic publishing is similar to the best cure for zombies: a quick death.

Planned Obsolescence does not take this easy path, and instead seeks to understand what animates the undead book. For Fitzpatrick, this begins with questioning the place and process of peer review, and this in turn forces us to peel back the skin of what lays beneath: authorship, texts, preservation, and the university. The essential question here is whether the cure to the zombification of scholarly communication may be found in a new set of digital tools for dissemination, and explores what the side effects of that cure may be.

In what constitutes the linchpin of her argument, the first chapter takes a bite out of one of the sacred cows of modern academia: the flawed nature of peer review as it is currently practiced. Unlike those who have argued–perhaps tactically–that open access and online journals will keep sacrosanct peer review in its current form, Fitzpatrick suggests that new bottles need new wine, and draws on a wide-reaching review of the history and problems of the present system of peer review, a system driven more by credentialing authors than promoting good ideas.

Fitzpatrick does not offer an alternative as much as suggests some existing patterns that may work, including successful community-filtered websites. She acknowledges that these sites tend to promote an idiosyncratic view of “quality,” and that problems like that of the male-dominated discourse on Slashdot would need to be addressed if we do not want to replace one calcified system of groupthink with another. The argument would be strengthened here, I think, with a clearer set of requirements for a proposed alternative system. She presents MediaCommons, an effort she has been involved in that provides a prototype for “peer-to-peer review,” as itself a work in progress. It is not clear that the dysfunctional ranking and rating function of the current peer review system is avoided in many of the alternative popular models she suggests, in which “karma whoring” is often endemic. As such, the discussion of what is needed, and how it might be effectively achieved could have been expanded; meta-moderation of texts is important, but it is not clear whether this is a solution or a temporary salve.

If we move from peer review to “peer-to-peer review,” it will have a significant effect on what we think of as “the author.” In her discussion of the changing nature of authorship, Fitzpatrick risks either ignoring a rich theoretic discussion of the “death of the author” or becoming so embroiled in that discussion that she misses the practical relationship to authors working in new environments of scholarly discourse. She does neither, masterfully weaving together a history of print culture, questions of authorship, and ways in which digital technologies enable and encourage the cutting up and remixing of work, and complicate the question of authorship.

The following two chapters discuss texts and their preservation. As the process of authorship changes, we should expect this to be revealed in the texts produced. Naturally, this includes the potential for hypermedia, but Fitzpatrick suggests a range of potential changes, not least those that make the process of scholarly review and conversation more transparent. This discussion of the potential edges of digital scholarship provides some helpful examples of the variety of scholarly discourse that is afforded by new media forms–a set of potentialities that is richer than the future that is sometimes presented by academic publishers whose visions are clouded by models and technologies that require profitability. The following chapter on the processes of disseminating and preserving this work I found to be particularly enlightening. As in earlier chapters, Fitzpatrick manages to draw together a surprisingly broad set of experiments and analyses into an intriguing and concise synthesis.

The penultimate chapter of the book discusses the question of how to support and sustain the creation of these new texts. The chapter argues that university presses should not try to beat commercial presses at their own game, but should instead invent a new game. It presents a number of models and strategies through which this might be achieved, and suggests those that show the most promise: notably, providing for open access and drawing the university press more directly into the work of the university and its library.

Planned Obsolescence itself was born of the realization that Fitzpatrick’s previous book was rejected due not to the quality of its thought but by the potential for press profit. That her next book is of now in a bound physical volume, published by New York University Press, and that this review will itself appear in a bound journal, published by Sage, seems to suggest that in some ways this born-digital scholarly conversation has itself succomed to the slow-moving process of traditional scholarly publication, and as such, might appear as something of a rebuttal to the argument that the only good zombie is a dead one. On the other hand, any criticism I provide above represents a form of slow, printed conversation that is largely outmoded by digital scholarly communication.

In fact, this neatly reflects the complexity of the new structures of scholarly publishing, and the promise for its future; a future in which we stop hiding from zombie books and invite them to a more convivial scholarly conversation. Anyone who is serious about understanding the future of scholarly publishing–and anyone who cares about knowledge and society should share this concern–will find Fitzpatrick’s book an essential, thought-provoking, and highly approachable introduction to the conversation.

Kathleen Fitzpatrick, Planned Obsolescence: Publishing, Technology, and the Future of the Academy. New York: NYU Press, 2011, vii+245 pp. ISBN 0-8147-2788-1, $23 (pbk)

A version of this review is to appear in New Media & Society, which provided me with a review copy.

Related posts:

1. What do my colleagues know?
2. Super-expedited review
3. Peer review in courses

Earlier this week, in the context of discussing what the traditional university provided that crowdsourced and open options did not with my students, I got an interesting mix of the usual suspects and some answers that I hadn’t heard framed in exactly that way before. (And yes, I am always impressed when students are thoughtful about complex issues.) Here are some of the reasons to go to college despite the increased availability of alternatives:

Credentialing

This is no surprise, of course. One of the reasons to go to an accredited university is for the transcript and the diploma. Long after other structures do learning better (if, indeed, such structures or institutions emerge), the university will maintain a stranglehold on students because of its ability to print educational currency in the form of a transcript. For me, this serves as a good reason to loosen that grip.

When MIT jumps on the badge bandwagon and people start talking about Thrun credits, we might argue that this imperative has already been diminished. But does a MOOC that is not a Stanford course hold the same kind of value? I doubt it. Personal brads do matter: that’s why Howard Rheingold and Edward Tufte, among others, can draw paying students to their seminars. But I wonder how far their letters of completion or endorsement carry.

For the time being, if you want to brand yourself as a graduate, you have to go to a university. And completely regardless of the quality of the instruction at that university, the name must be recognized and valued.

Structure

A university tells you what you should be doing, and not just in the classroom. One of my students was open about the fact that if she didn’t have to go to class she wouldn’t: the university in some sense provides a structure of discipline. (Another student disagreed, saying she would get bored without being able to go to class, and was motivated to attend on her own.) This extended beyond the classroom, though, to “life skills.” For many people this is when they are becoming independent, both living on their own and becoming their own thinkers.

Now, I worry that in some places, universities do a poor job of this, extending adolescence well beyond what might be ideal. Many of our students are too scaffolded, and unwilling or unable to put themselves in the driver’s seat of their own learning career. But it was an interesting suggestion: the university provides a needed structure for learning, and frees the student from some of the “meta”: what’s important? when should I study? what are we doing in class today?

Expertise Curation

I thought one of the more interesting answers was that it was hard to find the right people to teach the right things. Yes, there were a lot of self-styled experts out there, but when you don’t know anything about a field, the university provides a faculty that is presumably made up of people who know what they are doing. Perhaps because I’m not as confident in the ways in which universities filter people, this was never one of my top picks for the reason universities exist, but it is an interesting one. For open alternatives to thrive, they need to present a compelling case that they are providing access to experts, and that can be a difficult thing to do.

In some ways, I’m interested in the model of the European Graduate School, which boasts a star-studded faculty. Where else do you find Derrida and John Waters in the same list? Or Peter Greenaway and Donna Haraway? But it is an interesting question: how do you distinguish expertise when you don’t know anything about a field? You don’t. You leave it up to an institution that can act as a filter, and you trust that they hire the cream of the crop.

Guaranteed Skepticism

As an institution, skepticism is built-in. One of the students noted that faculty are willing to expand the conversation by taking positions they may not agree with, by raising questions, by placing skepticism and inquiry in an exalted position. Now, I am sure there are other institutions that do this, but it was heartening to hear this from students: one of the values of the university is a professoriate that is not married to the status quo, that takes nothing for granted, and that encourages a community of inquiry.

Virtuous Community

Because you are–in many cases–sharing the same physical space and bound together in a community, there is some feeling that you are expected to serve other students. Showing up to class unprepared isn’t just a personal failure, but in some way a letting down others in the community. Of course, you can get virtual communities where similar social capital is built, but it is much harder to achieve in the one-off networked class, where dropping the ball (or the course) might have very little effect on other parts of your life. The investment of time in an undergraduate degree means that you are all in the same boat.

Alt-U

I’m a fan of efforts like P2PU. But I also am not quite ready to give up on the university. I don’t think our only choices are the university as it is today or no universities at all. In fact, those two may be exactly the same thing: the university that does not rapidly change to fit the new environment is likely to be buried by the forces of history. As I’ve said before, we are about to go through a sea change in the way universities work that will make the newspaper shakeup seem tame by comparison. The mountain of student loan debt (some of which I continue to carry) constitutes an educational bubble. When universities find themselves having to contract, the outmoded tenure system will make that difficult.

But I also think that this will force some universities to rapidly innovate their way away from failure. It will be a painful process, but part of that process is figuring out what the real value and strengths of a university are. I think relying on the current hold on the credential is a very short-sighted approach.

In the medium term, one of the best solutions is the liberal arts college / research university hybrid. I also suspect that a successful model exists in universities and university towns merging. The walls of the university are coming down, and with them the distinction between student and faculty and worker. I suspect one model of the future university feels a bit more like a small town with a really good library and really good schools, and the four-year program leading to a slip of paper will slowly fade away.

Related posts:

1. Social networking at the end of the university
2. University death watch
3. University visitors

First, there is a reason PACs buy TV ads. It’s the same reason retailers do: they work. On Super Tuesday alone, GOP candidates spent just shy of a $100 million, much of it on TV ads. It’s hard to know to what degree spending will accelerate during the general, but let’s say it comes out just short of a total of, say $4 per each household in the US. (That’s lower, by almost an order of magnitude, than some are predicting.) That’s not, in the whole scheme of things, that much money. It’s, say, a dozen B-1 bombers. It’s probably not much more than our daily burn in Afghanistan.

What I wonder, though, is how all this TV ad spending affects the cost of advertising. If we can take a guess at the total spend on TV ads during the campaign, it will almost certainly outstrip the annual spending on television advertising for soda, for example (pdf). As a result, this makes local TV advertising–particularly in contested markets–more scarce, and drives up prices.

Leaving aside whether we can spend our way out of the recession as consumers, it does seem like retail sales have an effect on the health of our economy. So it’s a double whammy. Some consumers are clearly donating to these super PACs–although it will be interesting to see how much of Obama’s ad buys are also being paid for by large donors this time around. And the businesses are presumably donating millions of dollars into these funds. They are then faced with increased costs for TV ad buys–and probably mitigate this by buying fewer ads and spending more on advertising. This works its way into their product pricing structures. So the consumer donates to these PACs, and then finds that they are paying for the TV ads they are seeing, but they also are (already) paying for the ads they are seeing for retailers. Although most Super PAC money is coming from Wall Street and various parts of the service industry rather than manufacturers or retailers, those donations also end up ultimately coming out of the consumer’s pocket.

Leaving aside the pernicious effect of election spending on public discourse, it’s a great way to put the brakes on economic recovery.

Related posts:

1. Super-expedited review

Related posts:

1. Badgepost Failures

Good? Bad?

My advisor one day related the story of a local news crew that came to interview him in his office. This would have been in the mid-1990s. The first question the reporter asked him was: “The Internet: Good? Or Bad?”

Technologies have politics, but the obvious answer to that obvious question is “Yes.” Just as when people ask about computers and learning, the answer is that technology can be a force for oppressive, ordered, adaptive multiple-choice “Computer Aided Teaching,” or it can be used to provide a platform for autonomous, participatory, authentic interaction. If there is a tendency, it is one that is largely reflective of existing structures of power. But that doesn’t mean you throw the baby out with the bathwater. On the whole, I think computers provide more opportunities for learning than threats to it, but I’ll be the first to admit that outcome was neither predestined nor obvious. It still isn’t.

Are there dangers inherent to the very idea of badges? I think there are. I’ve written a bit about them in a recent article on the genealogy of badges. But just as I can find Herb Schiller’s work on the role of computer technology in cultural hegemony compelling, but still entertain its emancipatory possibilities, I can acknowledge that badges have a long and unfortunate past, and still recognize in them a potential tool for disrupting the currently dominant patterns of assessment in institutionalized settings, and building bridges between informal and formal learning environments.

Ultimately, what is so confusing to me is that I agree wholeheartedly with many of the critics of badges, and reach different conclusions. To look at how some badges have been used in the past and not be concerned about the ways they might be applied in the future would require a healthy amount of selective perception. I have no doubt that badges, badly applied, are dangerous. But so are table saws and genetic engineering. The question is whether they can also be used to positive ends.

Over the last year, I’ve used badges to such positive ends. My own experience suggests that they can be an effective way of improving and structuring peer learning communities and forms of authentic assessment. I know others have had similar successes. So, I will wholeheartedly agree with many of the critics: badges can be poorly employed. Indeed, I suspect they will be poorly employed. But the same can be said of just about any technology. The real question is if there is also some promise that they could represent an effective tool for opening up learning, and providing the leverage needed to create new forms of assessment.

Gold Stars

One of the main critiques of badges suggests that they represent extrinsic forms of motivation to the natural exclusion of intrinsic motivation. Mitch Resnick makes the case here:

I worry that students will focus on accumulating badges rather than making connections with the ideas and material associated with the badges – the same way that students too often focus on grades in a class rather than the material in the class, or the points in an educational game rather than the ideas in the game.

I worry about the same thing. I will note in passing that at worst, he is describing a situation that does no harm: replacing a scalar (A-F letter grades) with a system of extrinsic motivation that is more multidimensional. But the problem remains: if badges are being used chiefly as a way of motivating students, this is probably not going to end well.

And I will note that many educators I’ve met are excited about badges precisely because they see them as ways of motivating students. I think that if you had to limit the influences of using badges to three areas, they would be motivation, assessment, and credentialing. The first of these if often seen as the most important, and not just by the “bad” badgers, but by many who are actively a part of the community promoting learning badges.

(As an aside, I think there are important applications of badges beyond these “big three.” I think they can be used, for example, as a way for a community to collaboratively structure and restructure their view of how different forms of local knowledge are related and I think they can provide a neophyte a map of this knowledge, and an expert a way of tracing their learning autobiography over time. I suspect there are other implications as well.)

Perhaps my biggest frustration is the ways in which badges are automatically tied to gamification. I think there are ways that games can be used for learning, and I know that a lot of the discussion around badges comes from their use in computer games, but for a number of reasons I think the tie is unfortunate; not least, badges in games are often seen primarily as a way of motivating players to do something they would otherwise not do.

Badges and Assessment

The other way in which I worry about computer gaming badges as a model is the way they are awarded. I think that both learning informatics and “stealth assessment,” have their place, but if misapplied they can be very dangerous. My own application of badges puts formative assessment by actual humans (especially peers) at the core. Over time I have come to believe that the essential skill of the expert is an ability to assess. If someone can effectively determine whether something is “good”–a good fit, a good solution, aesthetically pleasing, interesting, etc.–she can then apply that to her own work. Only through this critical view can learning take place.

For me, badges provide a framework for engaging effectively in assessment within a learning community. This seems also to be true for Barry Joseph, who suggests some good and bad examples of badge use here. Can this kind of re-imagination of assessment happen outside of a “badge” construct? Certainly. But badges provide a way of structuring assessment that provides scaffolding without significant constraints. This is particularly true when the community is involved in the continual creation and revision of the badges and what they represent.

Boundary Objects

Badges provide the opportunity to represent knowledge and power within a learning community. Any such representation comes with a dash of danger. The physical structuring of communities: who gets to talk to whom and when, where people sit and stand, gaze–all these things are dangerous. But providing markers of knowledge is not inherently a bad thing, and particularly as learning communities move online and lose some of the social and cultural context, finding those who know something can be difficult.

This becomes even more difficult as people move from one learning community to another. Georg Simmel described the intersection of such social circles as the quintessential property of modern society. You choose your circles, and you have markers of standing that might travel with you to a certain degree. We know what these are: and the college degree is one of the most significant.

I went to graduate school with students who finished their undergraduate degrees at Evergreen State College, and have been on admissions committees that considered Evergreen transcripts in making admissions decisions. Evergreen provides narrative assessments of student work, and while I wholeheartedly stand by the practice–as a great divergence if not a model–it makes understanding a learning experience difficult for those outside the community. Wouldn’t it be nice to have a table of contents? A visual guide though a learning portfolio and narrative evaluation? A way of representing abilities and learning to those unfamiliar with the community in which occurred?

I came to badges because I was interested in alternative ways of indicating learning. I think that open resources and communities of learning are vitally important, but I know that universities will cling to the diploma as a source of tuition dollars and social capital. Badges represent one way of nibbling at the commodity of the college diploma.

Badges, if done badly, just become another commodity: a replacement of authentic learning with an powerful image. To me, badges when done well are nothing more than a pointer. In an era when storing and transmitting vast amounts of content is simple, there is no technical need for badges as a replacement. But as a way of structuring and organizing a personal narrative, and relating knowledge learned in one place to the ideas found in another, badges represent a bridge and a pointer.

This is one reason I strongly endorsed the inclusion of an “evidence” url in the Mozilla Open Badge Infrastructure schema. Of course, the OBI is not the only way of representing badges, nor does it intend to represent only learning badges–there is a danger here of confusing the medium and the message. Nonetheless, it does make for an easier exchange and presentation of badges, and importantly, a way of quickly finding the work that under-girds a personal learning history.

All the Cool Kids Are Doing It

Henry Jenkins provides one of the most compelling cases against badges I’ve seen, though it’s less a case against badges and more a case against the potential of a badgecopalypse, in which a single sort of badging system becomes ubiquitous and totalizing. Even if such a badge system followed more of the “good” patterns on Barry Joseph’s list than the “bad,” it would nonetheless create a space in which participation was largely expected and required.

Some of this comes of the groups that came together around the badge competition. If it were, like several years ago, something that a few people were experimenting with on the periphery, I suspect we would see little conversation. But when foundations and technologists, the Department of Education and NASA, all get behind a new way of doing something, I think it is appropriate to be concerned that it might obliterate other interesting approaches. I share Jenkins’ worry that interesting approaches might easily be cast aside by the DML Competition (though I will readily concede that may be because I was a loser“unfunded winner” in the competition) and hope that the projects that move forward do so with open, experimental eyes, allowing their various communities to help iteratively guide the application of badges to their own ends. I worry that by winnowing 500 applications to 30, we may have already begun to centralize what “counts” in approaches to badges. But perhaps the skeptical posts I’ve linked to here provide evidence of the contrary: that the competition has encouraged a healthy public dialog around alternative assessment, and badges represent a kind of “conversation piece.”

Ultimately, it is important that critical voices of approaches to badges remain at the core of the discussion. My greatest concern is that the perception that there are badge evangelists and skeptics is in fact true. I certainly think of myself as both, and I hope that others feel the same way.

Related posts:

1. BlogPost Progress Report: peer assessment
2. Buffet Evals
3. Independent Study: Spring

The problem with ranking is that it suggests that excellence in teaching is a uni-dimensional construct, which I think even a cursory “gut-check” says is dead wrong. When I think back to my greatest teachers, they have little in common. One was cold, condescending, and frankly not a very nice human, but he was exacting in asking us to clearly express ourselves, and his approach led to a room full of students who could clearly state an argument, lead a discussion, and understand the effects of style on philosophical argument. Another was a little scattered, but brought us into his home and family, was passionate about the field, and taught us how important it was to care about our research subjects. Another had a bit of the trickster in him, and would challenge our assumptions by setting absurd situations. And I could name another half-dozen who were excellent teachers–but one of the things that made them excellent was the unique way in which they approached the process of learning.

And frankly, if you asked a number of my undergraduatepeers who the “best” teachers in our program were, there would certainly be some overlap, but it would be far from perfect. An essential question is “best for whom”? And just as our students are each unique, and we should approach them as whole people (the unfortunate fact is that we *do* rank them by grading them, but that doesn’t make the process right), we should approach faculty as… perhaps a box of chocolate. The diversity of backgrounds, styles, and approaches to teaching and learning are a strength, not a weakness. We shouldn’t all be striving to fit to the golden standard of the best among us.

Now, this is not an argument for absolute relativism: there are better and worse ways of fostering student learning. It is also not an argument against quantification or assessment: I think an essential tool for improving our teaching is operationalizing some of the abstruse concepts of “good teaching” to something measurable, and using qualitative AND quantitative assessments to help us develop as a group. But the problem with ranking faculty is that there isn’t a single scale for teaching effectiveness, nor even the three (or four, if you count “hotness”) that RateYourProfessor suggest, but dozens of different scales that we might be ranked on. And while some of us may be near the top of many of those scales, I doubt any of us are at the top of all of them.

Related posts:

1. Teacher’s aid
2. Ranking Schools
3. How rank

Related posts:

1. Do I. Talk. Like that?
2. Everybody talk about pop musik
3. Google Talk

Funding agencies and journals should require authors to submit and openly publish protocols as submitted to the IRB.

There are a bunch of reasons for this. First, IRB protocols should be public. Right now, it’s treated on the one hand a bit like the Napoleonic code: it doesn’t matter what others have decided, the board decides entirely on the basis of your submitted application. This has some real negative implications.

First, the same protocol may be accepted at one campus and rejected at another. Or research in the same stream (or seeking to replicate) may be rejected at a later date.

Second, the IRBs have to make an original determination each time wasting, in many respects, the efforts of other competent IRBs who have already made a determination. They don’t need to be bound by earlier determinations, but don’t you think it would be worthwhile to be at least aware of them?

Third, when applicants feel as though they are handled unfairly, a transparent system is better for all.

Fourth, the best way to learn to do ethical research is to be able to observe the process from the periphery, and listen to the queries of IRBs and the responses by researchers. Releasing the approved protocols may not get at that deeper conversation completely, but it at least provides a small window.

Fifth, the protocols are an excellent way of “indexing” open data. Open research data often is published with codebooks and other ancillary material, but IRB protocols in many ways are the ideal introduction to an open collection of data, explaining why it was collected, how it was collected, and how it might be used.

For these reasons, among others, there should be open repositories of IRB protocols. Now, we could just try to convince individual IRBs on campuses to open up their process and publish protocols they approve, and I hope that they might. But IRBs are by nature a conservative group, intended to protect, not to disrupt. In many cases, they are made even more conservative by the institutions that they are housed in, and concerns by that institution either that they might be sued by subjects or that they might be investigated by federal regulators. (Those regulators, naturally, have access to the protocols and the decision process of the IRB once an investigation or audit begins, but they might not want to provide any sort of “probable cause.”)

Individuals might be encouraged to submit their own protocols to a repository, and in fact, self-archiving has made an important impact on the way publishing happens. But there are enough open questions surrounding this that it’s a hard place to start the ball rolling.

Funding agencies, and more recently journals, either insist on or facilitate the sharing of research data. In many ways IRB protocols are an important part of those research data. If funders required that IRB protocols be shared just like any other research data, and if journals provided authors the resources to share these protocols, it would revitalize in some ways the role of scholarly publishers and it would make for ethical oversight that was more robust and transparent.

But what if you were not IRB approved? It may be you didn’t need pre-approval of research by the IRB, and as I argue in the article above, I think this should be the case for much of the research that is currently placed under some level of review by IRBs. But if you don’t have to have IRB approval, I think funders should still require you to talk about the ethical considerations of your research, and journals should require you to publish this online when you do not have an IRB-approved protocol to provide.

What this does is creates an environment in which ethical post-review is encouraged. Certainly, when it comes to drug trials–and even to invasive forms of research of vulnerable populations in the social sciences–there should be some sort of oversight before the research occurs. But even after it occurs, peer-reviewers and the reading public should be able to see how the researchers weighed the needs and rights of subjects against the importance of their research questions.

Related posts:

1. Rethinking the human subjects process
2. Exxon Secrets
3. A not so clean slate

What happened? Well, at least pedagogically, I was fine with it. To recap, I was concerned that students were more interested in grades than they were in the actual material. I speculated that replacing the grades with badges would at least move them from focusing on entirely arbitrary markers (letters) to markers that were more explicitly tied to learning objectives.

I still think grades suck, of course. Grades are grand for beef, and actually pretty handy for deciding what restaurants to avoid, but as a tool for learning I think they take away more than they add. Realistically, though, I can’t get away from grading on my own. Unless I can convince all my colleagues to move in that direction, my experiment threatens to be merely a distraction for students, or in the worst case, a good way for them to ignore my course. Purely in terms of learning outcomes, I think being able to get totally away from grades would be great. But that wasn’t the case here.

Not learning, but ranking

The main problem is that more than just the students see the grade. It acts–in fairly limited ways–as a reflection on their skill. Now, to my mind, we simply shouldn’t graduate students we can’t stand behind. I frankly would have no problem “advising out” those students who are not performing at an elite level. I think it would better serve both those who left and those who stayed. But that’s not a realistic option (at least not in that extreme a degree).

Without getting into details, since I can’t simply unilaterally make a course pass/not pass–which is at least closer to ungraded–I ended up saying you get either an A or an F. Really, I intended to give As to everyone, short of really utter non-completion. I think I can say, without naming names, that people got As who really weren’t doing graduate-level work. I was clear in my narrative summaries of their work that this was the case, but they still ended up with As. At least one of my fellow faculty members found this contrast to be wrong, and I can understand why.

Pass and Forget

Many of the courses in our online programs are taken serially, and so I don’t have to compete for attention with other courses. That wasn’t the case this summer, and I suspect that students paid more attention to the courses where they were still “fighting for a grade.”

A lot of the work on pass/not pass grading going back several decades looks at courses in similar contexts. Being the P/NP course in a world of graded courses means that for some students (generally those who are not already high-achievers) the time and effort will be put to the graded courses, as they are afforded a certain degree of prestige as well as attention from students simply because of their grades.

Don’t Picture an Elephant

My intention was to make students think less about grading, but because they needed to keep track of the number of points each badge was worth, and whether they had crossed a certain threshold, they ended up thinking more about it. For some, the idea that the grade would be either an A or an F, and nothing in-between, raised their anxiety level, even after I made clear that an “A” was granted for even minimal completion of work.

In the end, by doing something out of the ordinary, I ended up focusing students more on the grades and grading structures, not less.

The (Non) Solution

So, what’s a person who hates grades to do? I’ve always been considered a “hard grader.” Perhaps that’s why people have been so focused on grades in my classes. Clearly going the other way, and becoming the Oprah of As (You get an A! You get an A!) hasn’t worked out. Two possible alternatives:

Maybe the simple solution is to provide a grading rubric, but simply make it easy to get an A. That doesn’t seem like a good solution. It seems like it contributes to the “menace” of grade inflation. But if I don’t really care about grades, I’m not sure why I should care about their inflation. More importantly, although I don’t get the issue of people dwelling over an odd grading structure, I still have to contend with pulling attention away from courses with a stiffer climb up the grading ladder.

The other alternative is to go old-school. The course is graded on a curve. The top 20% of points-earners get an A, the next 40% get a B, the next 20% get a C, and the next 20% get a D or F. (Come on, you don’t really expect me to curve around a C, do you? I’m not that old.) Of course, this means that students are going head-to-head Paper Chase style, rather than cooperating and collaborating nicely. That sucks, but maybe it is what is needed to get folks to step up their game.

So, for next semester, I’m back to grading meany. A curve (and not a saving one) it is, at least in the course where I’m dictating the policy. That will be more familiar ground for me, and probably also for the students.

Related posts:

1. Against letter grades
2. A pass

Over the last few days, I’ve been exploring Google Plus a bit, in my spare time, along with a quarter million of my closest friends around the globe. There are already a lot of reviews… that’s not what this is. I find it to be an interesting entry point, and I’m curious to see what happens with it. As I said on Twitter, I’m cautiously optimistic about its future.

That said, I’m curious what that future will be. A lot of people worry, rightly so, about the privacy implications of Google properties all combining their connections to users and assimilating us all into a massive, Google-controlled borg. But if anything, the opposite is also a concern.

I work a lot with Google documents with teams, and I find it to be a really useful tool. What works for ad hoc collaboration, however, doesn’t work as well for continuing collaboration or managing groups. If I have a group or organization, keeping track of project teams and the like who want to use Google Docs is a nightmare. Not only does it tend to be fussy with non-Google emails, it’s a pain to create groups in Google and then invite them to a document.

The biggest missing feature in the Googleverse is effective management of groups. Google Groups certainly isn’t it. And I was hopeful that Circles might be.

And it could be if you could share circles. Right now, I can create circles to indicate who is who and who sees what in my update streams. Cool. But no one can see what my circles are. As I noted in an early post on G+, that’s a good thing. I wanted to know this before I created, say, a circle called “people who annoy me.” All of us have people like this in our lives, and for one reason or another are forced to interact with them, and we probably could group them this way in our head. Nonetheless, we wouldn’t want this–or many other groupings–to be public.

I am thinking back to other social networks that attempted to create classes of connections that require a whole range of negotiations. It’s hard enough for me to divide people between “friends” and “associates”–if on top of this I had to then justify that decision to people who are in those two groups, I assume I would have very few of either.

So, as a default, I’m fine with the title and grouping of circles remaining private by default. (Although it’s not as clear that this is true of the latter case, since you can find out the list of people who have access to a particular post, and perhaps surmise some group boundaries that way.) But I want the ability to make my circles public, and perhaps to have people in the circle be able to add new people to the circle.

What sort of a circle would that be? I now have a circle called “QUICM students” that includes past and present students in the grad program where I teach. I took a first stab at dropping people into that group, but I know that there are others I’ve missed. I need something that is much more akin to traditional group management. Others in the group need to be able to add members from the outside, and perhaps with some groups, they need to be able to add themselves.

I don’t see that this is a big push away from Circles, and I really hope that G+ develops in that direction. Without some way of creating and managing groups, it will lose out on a function that is missing not just from their social networking effort, but from a lot of Google applications. One group I am working with has decided to use ManyMoon as a way of managing this function, and others turn to other approaches. But a robust group management feature would be a great addition to G+.

If that group/circle identity management solution could also be cleanly integrated with Docs, with Reader, even with Groups (which might become superfluous in this case), I think we would have a platform for distributed collaboration that would be helpful to businesses, to educators, and to anyone else who wants to get things done in a group.

Related posts:

1. Google Talk
2. Google Video
3. Contest: why does google hate me?

Related posts:

1. Live Giant Squid
2. Remaindered Links

It’s no secret: one of the reasons I wanted to have kids is for the toys. It’s so much more socially acceptable (and fun) to buy Legos and light sabers for your kids. But the irony is that kids don’t want toys, they want the real thing. Sure, they’ll settle for the cardboard box fashioned to be a car, but if you turn your back and they have access to your keys, watch out!

My son loves the idea and the practice of vacuuming. I’m not sure why–perhaps because it is such an unusual and rare thing to see in our house. He’s not alone in that. I spoke with a janitor at the American Natural History Museum who noted that kids come from around the world to the museum and the they universally are drawn to his broom and dustpan. They think it is far more awesome than some stuffed black bear or 6,000-year-old dinosaur bones. (I got that right, right? Carbon dating is just Satan’s Tivo?)

And there is a whole market of vacuums for kids that “really suck”! And when I say really suck, I mean that they have some sort of weak sauce sucking mechanism that makes it seem like they are some kind of vacuum.

So, faced with the possibility of spending well over $100 on a toy Dyson, we did what any sane person would do, and bought our two-year-old a cordless vaccuum. He vacuums, and actually manages to help clean up the house.

Is it dangerous? It’s not like we are giving him his own blender or microwave, but yes, I’m sure somewhere on the box it says something like “not appropriate for children under 4.”

And even though he loves his toy power screwdriver, he runs around the house saying “screw everythiiiiing!” and attempting to do just that. So, it may be a year or two until we turn over a real power screwdriver. But we will, and probably not before he is all that much older.

Because in a world full of toys, the best ones are also tools.

Related posts:

1. Tachi toys
2. BlogPulse Tools
3. Online survey tools

michaelzimmer: Right now: I supposed to be working on a journal article (tenure), but instead I’m writing a blog post (impact).

halavais: ∴ impact ≠ tenure RT @michaelzimmer: I supposed to be working on a journal article (tenure), but instead I’m writing a blog post (impact).

kfitz: @halavais @michaelzimmer Okay, that’s creepy. I literally just proofed the paragraph of the book on the relationship btw tenure and impact.

coffee001: .@halavais @michaelzimmer In some surreal version of the future, people will get tenure for blogging.

halavais: Let me fix that for you: “In some surreal version of the future, people will get tenure.”

coffee001: You have a point. Depends on the country you’re in, though.

snurb_dot_info: @coffee001 @halavais Someone needs to create an ‘I have tenure and I blog’ bumper sticker.

coffee001: @snurb_dot_info @halavais …for that, they need to have tenure and a blog. Not yet a combination that’s common over here.

halavais: Yep: http://www.cafepress.com/jamtoday RT @snurb_dot_info: [] Someone needs to create an ‘I have tenure and I blog’ bumper sticker.

dancohen: Fairly sure if I buy this “I have tenure & I blog” t-shirt I’ll get beaten up, but I’m not sure by whom http://bit.ly/h8pWw8 (via @halavais)

thinkingshop: @dancohen @halavais Not good for crowds containing #adjuncts & grad students (the other 70% of the teaching population)

Related posts:

1. When astroturfing goes bad
2. Team Blog!
3. The perfect blog entry

I don’t care about grades. It’s not that I hate them, but I do hate that students seem so captivated by them. At least at the undergraduate level, and for students who were aiming for law school or med school, where the GPA seems to have a strong effect on admissions, I kind of understood it. But I completely do not get it among graduate students.

In order to reduce the focus on grades, I tried delaying letter grades until the end of the semester, providing more narrative feedback. I would provide feedback, but no scores or letters on students’ work. There is some indication that this can be effective, particularly with formative types of assessment (see Kitchen et al, 2006), but it really didn’t work at all well for me. It increased anxiety and concern over grades rather than decreasing it, resulting in students who were even more concerned with grades than with the material.

So, I’ve effectively thrown out grades. I see no pedagogically-driven reason to issue them, and a number to get rid of them. As long as students meet some minimum requirements in my courses this semester, they get an “A” in the course. I’ve replaced this with badges, which I will write more about soon.

Pass/Fail

There is a low bar of participation–what I would consider something like 20% of the expected contribution–in order to get this A. But isn’t this just pass/fail? I suppose it is, to a certain degree, although not formally so.

A lot of the work around doing away with letter grades is about 40 years old at this point. Sgan (1970), for example, found that the would-be grades of students taking a course pass-fail were lower than the students in the same course opting for a grade. However, this was at Brandeis where not all courses were pass-fail. In such a situation, it makes sense that a student might focus more of her energy on the graded courses. It’s also worth noting that this gap disappeared by senior year, when the students taking a course actually did very slightly better (there was no significant difference in what they would have gotten as a grade).

For our online program, students generally only take one course at a time, providing less of an issue of attention management. But I’ll be curious how it works out for our on-ground program, where students may be taking several courses at once.

Assessment vs. Evaluation

It’s important to note here that there are two things moving hand-in-hand. On one side is questions of thinking about students’ progress and effectively changing the course material to meet the students effectively. The other is communicating student progress (and the acceptability of that progress) to wider audiences. This is the divide, as Cizek (2005) has it, between assessment and evaluation.

Of course, one is related to the other–or at least can be. One can assign letter grades based on an assessment of a portfolio of work done in a class, course, or program, for example. But it is a necessarily abbreviated form of communicating the work accomplished or skills gained, rather than providing an instrument for improving learning.

Cizek notes that letter grades, no matter what they are grading, are “consistently inconsistent.” Any measurement should be both reliable (consistent) and valid (measure what we are interested in) and letter grades are almost never either of these things. This isn’t news–educational researchers have known this for at least a century. There are ways of making grades more explicitly reliable, though often at the risk of being less valid. The application of multiple-choice tests tends to push in the direction of reliability (at the cost of creating a whole generation of students who are highly skilled at taking multiple-choice tests and little else), while grading on participation in class may get at what we really want to evaluate, but it very difficult to do fairly and consistently.

For me, letter grades do a poor job of communicating what they are supposed to communicate. If I see that a student has gotten an A in “Introduction to Interactive Communication,” what does that really tell me? Especially, if I don’t know what the other students received as grades, or what was expected of the students. It’s an empty indicator.

On the other hand, students strive to get that A. Some say that a high GPA at the very least demonstrates an ability to be able to follow directions and plan your time reasonably well, but I’m not sure even that is the case. The student with a high GPA simply demonstrates that she is capable of achieving a high GPA–any correlation with other skill sets seems almost accidental.

It’s when that letter grade evaluation crowds out any room for actual assessment and self-knowledge that it deserves to be more than just ignored or disdained. If we want students to learn better, we need to destroy letter grades. Grade inflation may provide the seeds of letter grades’ own demise, but I plan to hurry it on as best I can.

First Seven Weeks

One of the courses I am teaching this semester runs on an accelerated seven-week schedule, and so has just concluded. Everyone who was registered in the course received an A, as promised. One person withdrew from the course, but no one else failed to meet the minimal requirements.

As a whole, the performance of the students in the course was well above that of those in previous versions. I secretly kept letter grades (not reporting them) and the grade average for the course would have been significantly higher. It’s hard to attribute this entirely to throwing out grades in favor of badges. We had a few students who would have done well no matter what, I think.

Among the highest achieving students in the course, the work was ridiculously good and they worked especially hard. At least one expressed relief that they didn’t have to perform to a specified level, and so they took advantage of this and really went all out. I’ll note that two of the other students in the course felt “intimidated” by the level of these leaders, and this seemed to be a bit inhibiting.

The average student in the course did, I think, marginally better than they have in other courses. I’ll note that–having sent out the final evaluation–a number of the students in this group emailed back asking what their grade was. The syllabus for the course put it pretty clearly:

There is no compelling evidence that letter grades enhance student learning. For that reason all students who meet minimum requirements will receive an A in the course. I expect that most of you, if not all, will go beyond the minimum requirements not to improve your grade, but because you are interested in learning more.

I suspect, therefore, that these students just didn’t read the syllabus carefully. Or perhaps they just didn’t believe it. Nonetheless, the large group of students “in the middle” of the class did better than their counterparts in other courses.

It was not all good news, of course. Several students ended up on the trailing end. In a normal course, they would have failed, or at least would have received a very low grade. I’m still undecided what the A means to them. They know they did poorly in the course (I told them), but the reflection to the world is still an A on their transcript. I suppose I see this as the only negative outcome of what I’ve done, and I don’t think it is that negative. I’ve given failing or very low grades to students in the past who have just gone on to retake the course with another instructor or finish their programs with low (but passing) grades.

I suppose in some sense this is “passing the buck,” to other faculty members. On the other hand, it could be seen as merely being as accepting as possible. I know from experience that having low-performing students in a course lowers the level of discourse and is frustrating for many in the course. But short of raising the bar for passing (and re-introducing the concerns over getting over that bar rather than exploring and learning), I’m not sure how to address this issue.

In sum, the positives far outweigh these negatives. We’ll see how this goes for the courses that are new, and not nearly as well planned.

Cizek, G.J. (2005). Pockets of resistance in the assessment revolution. Educational Measurement: Issues and Practice 19(2), 16-23.

Kitchen, E., S.H. King, D.F. Robison, R.R. Sudweeks, W.S. Bradshaw, & J.D. Bell (2006). Rethinking exams and letter grades: How much can teachers delegate to students?

Sgan, M.R. (1970). Letter grade achievement in pass-fail courses. The Journal of Higher Education 41(8), 638-644.

Related posts:

1. Against letter grades
2. Retreating on the Grades
3. A = Average ?

There is surely nothing other than the single purpose of the present moment. A man’s whole life is a succession of moment after moment. If one fully understands the present moment, there will be nothing else to do, and nothing else to pursue. Live being true to the single purpose of the moment.

Everyone lets the present moment slip by, and then looks for it as if he thought it were somewhere else. No one seems to notice this fact. But grasping this firmly, one must pile experience upon experience.

- Yamamoto Tsumetomo, Hagakure

Last week I went to the Digital Media and Learning Conference in Long Beach. Normally, I blog my presentations for conferences, and that should be easier now that I have trimmed my conference schedule a bit. (That is really hard for me to do, since I learn more at conferences than through most other forms of scholarly communication, but I need to refocus some of my attention on my own research and practice.) I was on the hook for two things at this conference, a workshop on setting a research agenda for badges, and an ignite talk on killing the traditional transcript. The first went better than expected, but needs a bit of digestion. The second went OK, but also needs a lot more thought.

The accidental talk

But I am going to write a little about the third. I had been scheduled for an Ignite talk on Friday, as I said. Ignite is a bit of a variation on pecha kucha–you have to present using 20 slides, each of which auto-advances every 15 seconds. It is a formalism that leads to some great presentations. There are informal elements as well: a culture of fast talk and high energy. It seems like a rhetorical form ideally suited to our shortening attention span.

On the day of my presentation, we had joked that we should switch decks and present each other’s work. Given the first presentation, given by Erin Knight and Philipp Schmidt, was on badges, which is part of what I was presenting, this wasn’t a completely bizarre idea. This came back to haunt me the next day when the ignite MC, danah boyd, called on me to fill in for a missing presenter. Never one to say “no” to a challenge like that, I jumped in, trying to riff off of slides I’d never seen before.

Now, as the short clip of the end of the talk shows, it wasn’t a very good talk. I ended up not staying with the slides as well as I would have liked. And the argument, formed as it was on the fly, lacked not just nuance, but cohesion. But despite the carnivalesque nature of the presentation, with me as geek, I actually found some stuff in my own meandering that I liked. As a fan of bibliomancy and similar randomized approaches, this is hardly surprising. But I thought I might flesh out some of the ideas that came up as a part of the talk.

The power of surprise

I may have already told this story. As a graduate student, I liked to really over prepare my lectures. I probably spent eight or ten hours of prep for every hour in the classroom. This meant that my research productivity went way down, and I went without much sleep. More importantly, although I think my lectures were generally fairly interesting and thought out, they were essentially performances. They demanded the attention of the students, but I suspect they lacked the excitement and interaction of the classes presented by some of the other grad students in my department.

For a time, another graduate student by the name of Dougie Bicket worked just outside my office. On more than one occasion, I would come out of my office a bit flustered, not having managed to stuff my eight hours of prep in before an afternoon class, and Dougie, would say something along the lines of “why not just have them form dummy media companies that have to deal with the current issues surrounding content ratings,” or something that sounded equally sensible, off the cuff. And these class ideas were successful–not invariably, but often enough to make them worthwhile.

The time crunch became more pronounced when I started as a professor, but I quickly learned that the courses for which I prepared the least were often best. Yes, there needed to be some structure, and the lack of knowing just what would happen was a bit nerve-wracking, but the lack of planning created a gap that both I and the students needed to fill dynamically.

Process vs planning

I tend to be very outcome oriented. I think a lot of that comes from a background in traditional systems development: figure out what it is you want to make, break it into its constituent pieces, and build it out. 

At least in places where people know what they are doing, software development rarely follows this linear path these days, but these sorts of systems-based approaches remain the dominant ideology for course design. We specify objectives, break them into their components, attach assessments, and once the entire course is planned, we “deliver” them. This is particularly true for online courses.

I also should say that I tend to recoil a bit when people say they are process-oriented. This probably comes from an aversion to consultants who come into an organization with no content-specific knowledge, and require a bunch of meetings that have very little to do with actually getting work done. I don’t have a lot of trust in a “talking cure” for most organizations. In theory, I certainly agree that process is important, but as a practical matter, a process-orientation is too often an excuse for not really knowing what you are doing.

Just do it

So, if I am not a big fan of structured, hierarchical planning, and not a plan of more organic, process oriented planning, you might reasonably ask what sort of planning I am a fan of. I won’t go so far as to say “none” because that isn’t true, and I will explain the sorts of preparations I think are worthwhile below, but generally speaking, I think a flexible response is most important. 

Judo is different from many “soft” martial arts. It does have a small number of kata–or “forms”–but there is a much greater emphasis on applying techniques and combinations of techniques in response to the current situation. A major part of the daily practice of judo is not endless repetition of abstract forms, but randori–a sort of random play that trains the judoka to be aware of her situation and respond accordingly.

This, after all, is what we want from students, and what we should model in our own behavior.

The “teachable moment” is much pursued among teachers. My aim is to fill my days with teachable moments. But to do that means opening yourself up to surprise, and risking utter failure. It means being willing to fall on your face at any moment, and knowing that you will on a regular basis. It is not a conservative stance, by any means. If you are doing it right, it should make you nervous when you walk into the classroom. What’s going to happen? 

Again, if you are doing it right, your students will also not know exactly what to expect. This can be a little unnerving for some of them, but frankly, life is about not knowing what is coming, and choosing the path that may not always be safest.

Mindfulness

How do you get where you want to go without planning it? Pay attention to where you are looking.

That may seem a bit too obvious. As each year passes, I realize more and more that everything I ever needed to know about teaching I learned in a judo dojo. Although I no longer practice judo, I was very lucky (and it was mainly a result of happenstance) to study with some very gifted teachers in the United States and Japan. One of those teachers was Rick Bradley, who made certain that we looked the right way when executing a throw. The natural tendency was to look down at the floor, where you intended your opponent to fall. Instead, it is important to look almost in the opposite direction, as this determines where your head is, which in turn guides the position of your body, which ultimately causes your opponent to end up where she belongs.

Something similar has probably been said to dancers and baseball players, but rarely educators. We too often have our eyes on the goal, and not our head in the game.

So the most important element of an unplanned course is to be as actively aware of your environment as possible. That means thinking at every moment about what might be learnable in the current situation, and be ready for that to change at every moment. This means less focus on your plans for two months, two weeks, two days or two hours, and to focus on the next two minutes. How do you fill those two minutes with surprise and discovery?

Of course, you will still plan a bit; humans are planning animals. All I am suggesting is paying more attention to what you can do in any particular moment to illuminate lightbulbs. Education should not be a war of attrition, but a series of lightening surgical strikes.

Reusable patterns

What sort of planning do I endorse? The creation of objects and tricks.

Objects are thing to learn with. This certainly applies to physical objects. I am not alone in seeing Legos as not just as excellent learning objects, but as an example of the form. Do they have a specific object? You could certainly identify some developmental goals met by playing with Legos, but it is also an activity that is very open ended, offering more than a single desirable outcome.

Yes, clearly these include open educational objects, but they also include simulated and real environments. Museums and zoos, of course, but also malls and  theme parks. Not to mention urban areas, farms, and nature preserves. Although most virtual environments do not provide the richness of these physical environments, they still provide a learning context. Most classrooms represent nowhere near as involving an environment.

Power of expectations

Students react to this in a couple of ways, sometimes simultaneously. Many are excited by the opportunity to shape their own learning. Many are anxious because they cannot predict just what is expected (since the main expectation is that they find their own way). Many students have spent so much time in 19th century classrooms that they don’t know how to operate outside of those constraints. As a result, they have difficulty not just in my classes, but also in most working environments that are not likewise stultifying.

So, I try to make that transition as easy as possible, but only to the degree that it doesn’t compromise on the ideals of taking risks and finding new paths. Teaching in the moment doesn’t mean you have to toss out the syllabus, the textbook, or the classroom–though it helps! It does mean being open to disruption, and inviting tangents. And if you ask any one of my students, they will tell you I love a good tangent.

No related posts.

Yes. Here are some options:

1. ICM Salon

For those students who are on campus and are willing to attend several major talks this semester, I am willing to do a one-to-three credit “salon.” Basically, you would be expected to show up to the events, tweet them, and do a short blog entry for each at the one-credit level. At the three-credit level, you would be expected to do a significant research paper based on the ideas of one of the presenters.

2. Badges

Students in my classes will be exposed quite a bit to the idea of learning badges this semester. I’m doing at least two papers based on these ideas. If you would be interested in contributing to a literature review on badges and learning, let me know. Likewise, if you want to get your hands dirty on a django project building around this idea, would be happy to work with you on it.

3. Association of Internet Researchers web projects

There is some need for designing various pieces of web presence for the AoIR, including–potentially–its main site. If you are interested in this, I cannot guarantee that your designs or coding will be adopted, but it would be a good (credited) portfolio piece if they were.

4. A Course at Peer2Peer University

The Peer2Peer University School of Webcraft is running a number of courses this semester. It is a free, peer-led learning environment. If you would like to receive credit for your work in one of these courses, talk to me and we will make the appropriate arrangements.

Related posts:

1. Independent A
2. Independent Western America
3. Ah, spring

I suspect there will be some generational spread, with serious Apple fanatics needing the new-new, they may end up giving their current device to kids, parents, and significant others. I am not one of those who goes out to get the new thing right away; I prefer to let others beta test. I wish I had done the iPad sooner–it’s a great device–but I suspect that the “marketable features” that are required to push many to upgrade won’t be enough for me. That said, if I did upgrade, my current iPad would stay in the family.

But what about all those single Apple fanboys out there. Will we see a flood of second-hand iPads on eBay? A lot of these devices have been sold, and given that they are solid-state devices without a lot of moving parts, I can see how they could have a long useful life beyond their first owners.

Part of this thinking is a result of the sea of iPads I saw installed at the Delta terminal at JFK last week. One use of iPads seems to be obvious: micro-kiosks. With a bit of tweaking, these seem to be the obvious replacement for the public telephone, or for use anywhere you see interactive kiosks now. Museum displays, floor directories, employment forms at Target–you get the idea.

The downside is the nastiness of the screen after being touched frequently. It’s bad enough when it’s my own hands that have caused it. Ick.

And they also seem like the sort of empty control panel for any manner of interesting devices. Yes, hackers love the eees for the same reason: they are cheap and easily interfaced and programmed.

But the eee and similar devices, besides being much cheaper than the iPad, are also much more easily hacked and tweaked. Sure, you can jailbreak your iPad, but–as a couple of students asked me recently–why bother? There isn’t much you can do with it once you do. Many net top devices already have a linux variant installed, and Loading the iPad with something else is not trivial.

That said, there is a great deal that can be done with HTML, javascript, and a good back end. So, if there is a prediction in here, I would say expect to find iPads in unexpected places, and affixed to other stuff.

Related posts:

1. iPad Goes Back to the Future

Oh, the whole world. The children have the run of everything. To set up specific playgrounds would be like setting up a table-sized aquarium in the depths of the ocean. It would really be pointless.

- RA Lafferty, “Primary Education of the Camiroi”

Related posts:

1. A short run of “Much Ado”

So, down the oubliette with the beginnings of my syllabus:

Mobile and locative applications

This is a course designed to make you think about the world as a place that is Internet-enabled, and give you some of the skills you need to design for that world. What does it mean to work and play in a mobile world and how does this relate to designing and building applications based on places and interfaces that go beyond the mouse and keyboard?

This course is offered in a peer-based studio format. As part of this course, you are expected not only to learn how to do new things, and demonstrate your ability to do those things, you are also expected to create materials that teach others how to do something new.

This course organizer is Alex Halavais. He is an associate professor in the interactive communication program at Quinnipiac University. More information can be found on his blog: http://alex.halavais.net

Course Meetings

The course is scheduled to meet Tuesdays from 6:30 to 9:10. In addition, there will be open studio hours from 4:30 to 6:30 on Tuesdays, in the great room at 430 Mt. Carmel Avenue. You are expected to spend the required hours on the course, but the time spent physically in the classroom/studio is flexible.

Course Communication

There are three main ways we will communicate as a group. The instructional material, badge requirements, and any other course documents will be kept at the course wiki at XXX. We will also be using a mailing list, hosted by Google Groups. Please go to XXX to join the group.

Social Contract

When you sign on as a participant of this course, you promise to:

* Be timely in your interactions with the community. If a week goes by, and we haven’t heard from you, something has gone wrong. If you can be with us physically in the hack sessions, that is best; if you cannot, we should hear from you virtually at least weekly. When it comes to responding to questions relating to the learning objects you design, or asking for evaluation for a badge you have designed, you should be especially quick in responding. I would like you to be willing to respond to requests for evaluation within 72 hours, and I will endeavor to do the same.

* Be here to learn. I know, many of you have a degree to earn, and a job market to wrangle, and the like. However, the purpose of this course is to form a learning community. That’s what is the most important thing for me, and if it isn’t the most important thing for you, please choose another course.

* Be willing to teach. This is a community, and I expect you to contribute to the learning of your fellow participants. I applaud your sponginess, but please honor the community by being willing to help your fellow classmates, and not just take what you can from it. This is particularly true for the badges you author, but across the board, I hope that you are willing to pitch in, answer questions, and help where you are able to do so.

* Strive to acquire the minimal skill set detailed below. That is, acquire the five necessary badges and at least five of the substantial skill badges.

As the organizer of this course, I promise to endeavor to embody those principles in my own work during the semester.

Badges

You may have already noted some talk about a badge system. Basically, badges indicate your skills and abilities. If you have ever gotten a Boy Scout merit badge, or been SCUBA certified, or gotten a driving license, or won a Foursquare badge, you already have a rough idea of what a badge is.

For the purposes of this course, we will be focused on badges for particular skills. For example, one of the badges I expect you to earn is the MediaWiki Editor Basic Concepts badge. To do that, you have to demonstrate that you understand how to do some basic things with the markup syntax of MediaWiki. When you submit evidence of having accomplished these tasks, a number of endorsers will acknowledge that you have accomplished earning the badge, and you will be able to show the badge on your home page on the wiki, or anywhere else you choose to do so.

There is an early tutorial on how to earn a badge and how to create your own badges. Anyone can create a badge for anything, and some time during the semester, you will create your own badge.

There are two lists of badges that you should take special note of on the wiki: the “necessary” badges and the “substantive” badges. These are badges that we agree as a community are required for the course, in the first instance, and that represent significant and substantial skills in the area of locative or mobile media in the second instance. If I agree that a badge and associated learning materials are particularly strong, I will add them to the approved “substantive” list.

Expectations, Grading, and Credit

Everyone who enrolls in the course is expected to endeavor to complete all the necessary badges (five) and at least five of the substantial badges, regardless of the way in which they are engaged in the course.

This course is offered for three graduate academic credits at Quinnipiac University as ICM500. If you are not an interactive communications graduate student and would like to enroll for credit, please contact Phillip Simon about arranging to take the course as a non-matriculating student.

As an experiment, this course is also being offered at Peer2Peer University, an open structure for engaging in peer learning online.

I don’t believe the A-F grading system is an effective way of engaging learning or providing feedback. For this reason, anyone who completes the minimum requirements at QU will receive an A in the course.

First, you must complete the following five necessary badges:

* M&L Apps Social Contract Signatory Badge
* MediaWiki Editor Basic Concepts Badge
* Helpful Colleague Badge
* Basic Open Learning Resource Creator Badge
* Badge User and Maker Badge

In addition you are expected to earn a minimum of five substantive badges during the semester. These represent some knowledge, skill, ability, or experience involving mobile or locative media. For example, if your create a piece of hardware that can tweet to the web when something happens in the physical world, you would earn the Basic Sensors badge. In each case, the evidence required to earn the badge can be accomplished by completing materials in the unit(s) associated with the badge.

The initial set of substantive badges include:
* Mobile Web Standards
* Mobile UX
* Kiosk Planning & Design
* Geocaching and Collaborative Mapping
* Geocoded Web
* Google Maps
* 7Scenes Basics
* Basic AppInventor
* AppInventor Web Services
* Blinky Lights (using Audrino)
* Basic Sensors for the Web

It is expected that you will complete the required badges within the time period of the course. Because most badges require endorsements from your peers, continuing beyond the agreed period of the course is impossible. For this reason, incompletes will not be granted. Likewise, if you have not completed a substantial number of badges, including all of the necessary badges, by the midpoint of the semester, you will receive an email from the instructor recommending you withdraw from the course.

Schedule

As a studio course, this class does not have a schedule as such. However, you are required to complete the Social Contract and MediaWiki Editor badges before moving on to more substantial badges. You are also required to complete all five necessary badges before the midpoint of the semester (for QU students).

Beyond those constraints, you are welcome to engage the material as you like. There are many factors that may influence your choices. For example, you may need to order a book or hardware to effectively complete a badge, and that may take some time to arrive. Likewise, although Google claims a one-week turn-around on AppInventor accounts, if you plan on creating an Android-based app, you should probably apply for that account right now, with the knowledge that it will be at least a week (if not longer).

For work that requires outdoor activities, you may want to wait to later in the semester, or for a warm spell, if you are local to Connecticut.

Also, doing things together is more fun. So, I hope you will chose to work in tandem on projects. Each individual is charged with creating their own evidence for a badge, but if you prefer to collaborate while learning working through the learning units, I strongly encourage it. Particularly for doing things that are community-based, working in a group can be a significant multiplier, and it may be possible to demonstrate work on a communal object.

Texts & expenses

Each badge requires you know how to do something. Materials needed to learn how to do this will be assembled on the course wiki. They may draw from open access materials out on the web, or they be entirely original. Each person who creates a badge should assemble the materials necessary for people to learn the skills represented in the badge.

While these need not be original in all cases–curated materials are fine–at least once during the semester, you must create original, open access materials that teach us how to do something in the mobile and locative realms.

It is important, whether curating materials or creating your own materials to respect authors licenses, copyright, and intellectual rights.

You are going to need web space to host some of the projects in this course–feel welcome to use whatever space works and is appropriate to the need. Likewise, as you are creating your own materials, you will may find that the most current and accessible materials are actually available in books. Libraries can be slow in acquiring such books, so you may find yourself purchasing technical books (or camping out at your local bookstore).

Some of the badges require access to particular hardware or software. A preference will be shown for free and open software, when available, but even open hardware often has an expense associated with it. (I will have a limited amount of hardware you can play with in our physical meetings.) Enough badges will be available that it should be possible to avoid such hardware or licensing expenses, but I hope small expenses won’t dissuade you from learning experiences.

Finally, I do encourage you to collaborate with others in the class to assemble these resources. An Audrino board or electronics kit can be shared among multiple people, as long as you are careful with it, and likewise other resources can be effectively shared.

Related posts:

1. Badgepost Failures
2. What makes up a badge?
3. BlogPost Progress Report: peer assessment

As part of my courses in the coming semester, I am abandoning standard grades and instead using badge-level assessments. As part of each course, students can earn any number of badges for demonstrated abilities. These are generally badges that require you to show that you can do something. That ability must be assessed–often by peers.

Starting with the “data” end, what kind of information must a badge hold? We talked through a lot of this in Barcelona, and I’ve been thinking a lot about it since. What appears below shouldn’t be seen as the consensus of that group–though I found the discussion valuable, a number of the items below are certainly not commonly agreed upon among those, e.g., at P2PU who are talking about badges. At a basic level, a badge should be transparent (everything that went into getting the badge should be as visible as possible), and it should be imbued with the authority and reputation of those who were the evaluators.

Process

First, I should briefly describe the process. In the first courses, this process will largely be implemented “manually,” but you will see that there are many opportunities to automate some of these processes.

1. A person is nominated (or nominates themselves) by filling out all of the information on a form except for the endorsements.

2. Endorsers go to the form and indicate whether they feel that the candidate qualifies, for those that require endorsements–some may not. Note that “bots” may act as the endorsers, and check automatically whether something has occurred. In that case they behave just like human endorsers. Note also that the system that records this application should in some way verify the identity of the endorsers. We won’t be do that initially, but eventually, something (e.g., OpenID check) should provide an indication that people are who they say they are.

3. Once the endorsements are complete, a person may put this badge wherever they like on the web, with a link back to the page to show that they have earned the badge.

Nomination / Evidence Form

So, what is on that form? (With * items required.)

1. Name of the badge*

A short description of what the badge signifies: e.g., “Javascript Expert.” If it is a bootstrap badge, this should be clearly indicated in the title: e.g., “Javascript Expert [bootstrap]” (see #9 below).

2. Issuer of the badge*

Eventually, this may be something like “School of Webcraft” or “Quinnipiac University.” For this initial round, it is likely to be “ICM” for the ones I am doing.

3. Version of this badge*

Date-time last updated the badge.

A unique ID for the badge is formed with #1/#2/#3, e.g., Quinnipiac University/Ph.D. in Social Computing/2011-12-25-7:00:00.

4. Badge Image*

For the purposes of standardization, I will say 250x250px PNG representing what the badge stands for.

5. Description

A textual description of what the badge represents. The idea is that it is reasonably brief–say, less than 200 words.

6. Recipient*

Who is it that is claiming the badge.

7. Nominator*

Who is it that nominated this person for the badge?

By default, any badge can be self-nominated. If for some reason you want to exclude this possibility, it could be listed as a requirement in section 9: E.g. “Candidate is nominated by someone other than themselves” or “Candidate may only be nominated by a member of the track team.”

8. When nominated*

Nomination timestamp.

9. Requirements & Evidence

This is the meat of the form. It includes 0 or more requirements, with links to evidence that those requirements were met. Each requirement includes a record like the following:

a) Textual description of the rubric for assessment. What needs to be shown, and how is an evaluator to decide whether it meets the standard. Outside examples may be linked, including former examples of successful badge earners.

b) Textual description or link to the evidence of assessment. (If a link, we’ll probably need to find a way to archive that link for posterity. Easier with some things than with others; e.g., video.)

c) Nominator’s comments on the work and why they think it qualifies.

d) Qualifications to endorse. For example, you might require that people have the badge they are endorsing, or that they have a badge that qualifies them as “instructors” in the skill (e.g., to get the “pilot” badge, you need to be endorsed by at least one person with the “pilot inspektor” badge, or to get a a QU-PhD badge you need endorsements from three people with the QU-Faculty badge). You might also require that people have a badge that verifies their identity. So if I have the Verisignature–ReallyMe badge, maybe it qualifies me to endorse more badges. C is a list of required badges–there may be more than one.

e) Number of qualified endorsers required. This could be zero or a thousand.

f) List of
1. Endorser name
2. Date of endorsement
3. Comments on endorsement

Note that there is a necessary and automatic exception here in the case where there do not exist in the world the number of qualified endorsers listed in D. In that case, you must be endorsed by as many qualified endorsers as currently exist. It is then clearly indicated that the badge is a [BOOTSTRAP]. At some future point you might want to re-try the badge to get a non-bootstrapped version, once there are enough potential endorsers.

10. Issued Date-Time* (or PENDING)

11. Expires Date-Time

12. Recipient’s Comments & Notes

13. List of community comments

An Example Badge Template / Form

Now certain elements of the above are part of the template of a badge. So, if I nominated someone for the “Good Discussion Summarizer” I would end up with a template that included:

“Good Discussion Summarizer”
The Human Fund
1999-8-14-09:00:00

[Some Cool Badge Art that I don't have time to dummy up at the moment]

The good discussion summarizer is issued to someone who has demonstrated that she is consistently capable of summarizing a brainstorming or other discussion in an academic setting, both verbally and textually.

Recipient:

Nominator: Alex Halavais (2010-12-2-18:55:03)

Badge Requirements, Evidence, and Endorsements

1. Statements from three members of courses in which the recipient is enrolled attesting to her abilities to accurately summarize materials. Endorser must hold the “current student” badge. (No evidence beyond the endorsements required.)

Evidence: (NB: this would be left blank.)

Endorser:
Comments:

Endorser:
Comments

Endorser:
Comments:

2. Evaluation of a video of the candidate reviewing a discussion. Endorser must hold the “Good Discussion Summarizer” badge.
Evidence (Link to video or audio of summarization):
Endorser:
Comments:

3. Evaluation of a textual summary of the same discussion. Endorser must hold the “Good Discussion Summarizer” badge.
Evidence (Link or Pasted Text of a summary):
Endorser:
Comments:

Issued: PENDING
Expires: TBD

Candidate comments:

Community comments:

The nominator would fill out some of these, including, perhaps, being one of the endorsers.

Other Issues

The natural question is how would endorsers know to find the form? There are lots of possibilities here, including informal or direct invitations, and a queue of badge candidates needing assessment. But that is a solution that does not have to exist in the badge process itself (necessarily). The idea is to keep this piece as simple and light as possible.

Happy to hear any thoughts you might have. As I said, I’m going to take it for a test run in the Spring semester. I’ll likely just have people do it manually on the wiki, unless I find time over the break to code a simple form system that can handle the pieces. And I’ll point to the course and the badge description (as well as some of the early badges) as I write them up.

As a final note, this doesn’t in any way take away from the efforts of the Mozilla Badge Backpack approach. Indeed, one of the advantages to that system is that it might provide the opportunity for several dissimilar badge systems to work together. In this case, what would be passed along to the Backpack is just an image of the badge, its name, and a link back to the form that demonstrates how it was earned.

Related posts:

1. BlogPost Progress Report: peer assessment
2. Badgepost Failures

Related posts:

1. Teachable moment
2. Alex Halavais Quotes