So I had an interesting interchange with Josh Cavalier the other day.  Somebody was posting about AI, and I offered up my experience with the accuracy and usefulness of AI in my own work.  I said that when I’ve used AI to help with things, the usefulness has ranged from 20%-80% useful.  Sometimes I can glean a few things that I can use, and sometimes I can use most of the output with tweaks. 

Josh let me know he could probably get me up to 95%.  I totally believe him (he’s very smart about this stuff, and my approach to prompt engineering is fairly haphazard), but what struck me about the conversation was all the places where my expertise was necessary in any of these scenarios. 

The 80% scenario: I needed to vet the content and make the relatively small adjustments to make it closer to 100%. 

The 20% scenario: I needed to realize that I should discard most of the material, identify what was useful, and revise the question I’m asking to get a better result.

The 95% scenario: I need to use my expertise to craft a prompt that will get the most accurate result, while still recognizing the parts that need revision.

At the Devlearn Conference last November, I took part in a Guildmaster panel on AI, and I talked about how most of the AI implementation has been taking place in an interesting window where most of the people using it already know how to do their jobs. 

I think this is significant because we are already moving out of that window — we will soon have people coming into jobs with AI processes in place who do not already know how to do their jobs (assuming that they are not entry-level people being replaced by AI). It raises the question of whether or not the AI processes we are implementing right now will stand up to people who do not have the expertise I identified in the 20%-80%-95% scenarios. 

I am absolutely seeing the accuracy and usefulness of the LLMs improve.  I am not seeing them get to the point where we don’t still need to have enough expertise to recognize whether your result was 80% or just 20%.

Diane Elkins had my favorite comment of the panel.  She described how the AI results are essentially average.  If someone has been performing below average (we’ve all seen it) then the AI is an amazing improvement.  It’s not going to get above average though, or at least not without the expertise we talked about.

So, yeah, AI is very helpful AND you still have to know stuff.

Really Define the Behavior

Instead of fuzzy goals like “build a safety culture,” participants will practice writing outcomes and behaviors so clear you could capture them in a photo. This exercise sharpens your ability to spot measurable, observable actions that drive change

Mapping Your Learners Motivation

Using a case study (like freelance writers saving for retirement), you’ll highlight all the reasons people might — or might not — adopt a behavior, and then connect them to autonomy, competence, and relatedness from self-determination theory

Figure Out Where Are They in the Change Process

Not everyone is starting at the same place. In this activity, you’ll place learners on a 10-step readiness continuum — from “doesn’t know the behavior exists” to “struggling to maintain it”. Is it a knowledge problem, or is it more about motivation and support?

Design Effective Interventions

You’ll learn to dissect a behavior using the COM-B model: Capability, Opportunity, and Motivation. This reveals hidden barriers and points to practical interventions, which you can use to design effective solutions.

You can register here: https://members.ldaccelerator.com/checkout/learning-design-for-behavior-change-with-julie-dirksen

If you’d like to bring your team, we do offer team discounts. Reach out using the contact form.

Thanks!

Julie

One of the things that has bothered me since the beginning of the AI conversation is that most of the discussions of using AI or LLM outputs contains some phrase to the importance of “evaluating the output to make sure it’s correct” or something along those lines.  Pretty much any responsible writing about AI contains a reference to the importance of not accepting the AI output at face value, but reading or viewing it to make sure it’s okay.

But here’s the thing.  They say that very casually LIKE IT’S NOT THE HARD PART.

First of all, you need the expertise to judge an output, and second you need the discipline to exert the effort required to assess an output.

One of the early ChatGPT efforts I saw was somebody using the AI tool to write a short textbook on the teaching method of Direct Instruction (this is not intended as a slight on that person – it was clearly an experiment at the same time people were experimenting with ChatGPT to write Shakespearean sonnets about their favorite dog breed). 

I have no reason to believe this was an effort to really create a textbook, but it gives us an example to work with.  If you were actually using it as a tool to help create an actual textbook, what questions would you need to ask (ethical issues aside)?

The first question would probably need to be “Is this an accurate resource about Direct Instruction?”  Possibly this person had the expertise to evaluate the output, but it seems like something that you would want an expert to review before distributing widely.

Second, you’d need to have the discipline to read the whole thing.

This scientific article was making the rounds on the internet last week.  I’m utterly unqualified to comment on the accuracy of this article, but even I know if you start your introduction with the phrase “Certainly, here is a possible introduction for your topic: Lithium-metal batteries are…” then it means somebody wasn’t being careful with the copy-and-paste function.  Academic writing can be a tedious process, and AI might be able to help with that, but if the authors of the article aren’t actually reading what the AI produces, that’s a problem.

It’s part of human nature to accept defaults.  This isn’t always the case, but it is common enough that we should be very concerned about people having the discipline to stop and review AI outputs.  I’m not convinced that it’s a realistic goal – quick copy and paste is too easy a behavior to admonish people out of – but that means we need to have other safeguards in place.  Either that means guardrails built into the AI itself, which are being developed, or it means having at least a spot check or audit process in a workplace context. 

I know one of the arguments has been that error rates in AI-generated material can be lower than error rates in human-generated material (Jennifer Solberg discusses a few good examples in this podcast), and should we hold the AI to 100% standard when we don’t hold humans to that standard. First, YES WE SHOULD. I suspect that is there is more likely to be an internal logic with human errors, but that’s a complicated topic that is probably very context dependent.

Risk level should drive this level of vigilance. What is the consequence of this being wrong? Is it a wonky social media post, or is it a missed cancer diagnosis, or is it an inaccurately placed drone attack?

More thoughts to come on this, but for now, I think there are a few questions we should be asking:

• Does this person have the knowledge and expertise to judge this output?
• Is it reasonable to expect this person has the discipline to evaluate the outputs in detail?
• What is the risk if output errors are not caught?

Virtual Workshop: Cut Through the Noise – Communicate Value using Behavioral Science

One of the hardest things we do as learning designers is communicate our value to others.  We need to help learners understand the value of a learning experience, and we need to help our stakeholders understand the value of good learning design.

When I was working on the new book (Talk to the Elephant: Design Learning for Behavior Change), I realized there were a lot of guidelines in behavioral science to help communicate value, and that those principles could be applied to learning design.

February 22, 2024 11:00 AM – 2:30 PM ET (USA)

Price: $149

https://usablelearning.com/workshops/workshop-cut-through-the-noise/To learn more, check out the full description.

Virtual Workshop: Learning Design for Behavior Change

Do you have tricky behavior change challenges that you need to design for?

This is the very first virtual offering of the full version of the behavioral design workshop that I have been teaching for the last few years, and an opportunity to practice and apply most of the tools from my new book Talk to the Elephant: Design Learning for Behavior Change.

May 2nd, 9th, 16th 2024 11:00 AM – 2:30 PM ET (USA)

Price: $499

To learn more, check out the full description.

In-person Workshop: Designing and Implementing for Behavior Change

If you can make it to Orlando in April, I will be offering Designing and Implementing for Behavior Change as a one-day pre-conference workshop at the Learning Guild’s Learning & HR Tech conference.  This covers more or less the same ground as the longer virtual workshop and comes with proximity to theme parks, if that’s your jam!

May 2nd, 9th, 16th 2024 11:00 AM – 2:30 PM ET (USA)

Price: $695.00  

(Coupon Code for $100 off – SPEAKERSHARE100)

To learn more, check out the full description here.

Okay then — challenge accepted! I will let ChatGPT create an instructional design curriculum for me!

The results were…mixed.

A few disclaimers: Yes, I recognize that the technology is constantly improving and this is just a moment in time. I was using the current free version — I know the paid version is better, and that refining the prompts would help a lot. I deliberately didn’t adjust the prompts because a completely new person wouldn’t necessarily know how to optimize the prompts based on the output. I also recognize that I kind of expected mixed results and so there’s an element of confirmation bias. And I might just be a tad nitpicky about ID curriculums (okay — let’s be honest — very nitpicky).

I used their exact prompts for an instructional design curriculum. I commented up the full version here (google doc), but here’s the short version. The prompts are paraphrased a bit here, but you can see the full versions in the doc :

Prompt 1: Pareto idea / Give me the 20% that is most useful where I can do 70-80% of the stuff

I have doubts about the intent behind this question, but gave it a shot. ChatGPT gave me six topics to focus on in a hacked together a short version of a curriculum, though it’s honestly not what I would ever put together as a short form for minimum viable curriculum. I also asked later for a list of essential ID curriculum topics, and got 14 items. This short version has six of the 14 and is arguably the top items, but left out things like Understanding Learning Theories, Needs Assessment, and Ethical and Legal Questions (including issues related to copyright, accessibility, and privacy). I don’t think a responsible curriculum (however short) could completely ignore these topics. I do a lot of ID curriculum work for large companies, and the most consistent gap I hear about is needs assessment — making sure you understand the problem needing to be solved, and that you can solve it with training (or why it’s not a training problem). Some of the weaknesses of this item are probably equally about the prompt itself, rather than just ChatGPT’s response.

If was rating this prompt, I’d give it a 5 out of 10. They hit several of the big topics, but also had some glaring omissions.

Prompt 2: Create a study plan in an appropriate number of weeks

It parsed out the six items over 12 weeks (48 hours), but then only gave me a 2 1-hr activities for most of the items (~14 hours), and I think in that time, there were two actual practice activities. The rest were just reading about each topic or looking at examples. This would be a pretty weak curriculum design, and none of the math around the hours adds up.

I’d rate this one a 2 out of 10.

Prompt 3: Suggest various learning resources for the above study plan (like books, videos, podcasts, interactive exercises)

The book references were pretty good overall. It recommended my book, so that’s nice, and several other good books. The book recommendations didn’t quite line up with the topics, or would require the learner to really scrutinize the book contents to find the relevant material, and the books would not be readable in the time allotted in the study plan. The video links were all broken. I can’t be 100% sure but I’m pretty convinced they are all ChatGPT hallucinations. They are attributed to people or entities that do exist, but don’t do a lot on youtube. There are videos on most of these topics, some of them by reputable folks, but ChatGPT did not point to any of those.

They did link to a good podcast, and one or two other reasonable resources, but this prompt did expose a big issue in instructional design. While the roots are similar, instructional design is practiced very differently in higher ed vs workplace, so some of the books and resources are primarily useful in one or the other domains, not both.

Rating would be 5 out of 10 because it picked genuinely good books.

Prompt 4: Can you give me some beginner projects I could work on to help strengthen my workplace instructional design skills?

For this prompt, I specified workplace, and it’s a pretty good list. Not because a beginner could actually work on most of these projects (they would need a subject matter expert at the very least), but because it gave me a pretty good sample of the kinds of projects that somebody in workplace learning would actually work on, and would give a new person a good idea of how they might be spending their time.

I’m rating this one high, because it’s useful even if it didn’t really answer the question asked – 7 out of 10

I stopped at that point (though there were a few add-on questions in the doc). Overall, it’s kind of what I expected (did I mention my potential confirmation bias?). I saw some genuinely useful stuff, some mediocre stuff and some utter nonsense.

This is absolutely a snapshot in time, and will likely improve as the technology gets better. It did emphasize for me that ChatGPT and the like will always be a lagging indicator for content production, and it’s propensity to make stuff up is probably a feature, not a bug.

It all has a focus on curating and parsing content for self-motivated learners that ignores the reality that content is the relatively easy part of education. Feedback is the hard expensive part. The potential use of LLMs for education that is actually exciting to me is the idea that we could use them to give feedback at scale. MOOCs could put content out to millions, but they couldn’t grade a million term papers.

I’m not sure what we would need to do to feel confident about LLMs providing student feedback, but it’s an interesting avenue to explore.

Final comment: I promise I don’t hate AI/LLMs

I didn’t do this because I hate AI or because I don’t believe AI/LLMs will be transformative. I do believe they will be transformative. But I’m also seeing a lot of “You can create XYZ in seconds!!” rhetoric out in the world, and I don’t think that’s helpful either. Are these tools amazing? Yes they are. Are they fully ready for prime time in all cases? Maybe not quite yet.

Connie Malamed created a free resource for people interested in ID (she also has a very valuable paid membership community here):

https://breakingintoid.com/

She hasn’t added new episodes for a while, but Nyla Spooner did a podcast called “I’m new here” for new IDs:

https://www.nylalxd.com/blog/categories/i-m-new-here

L&D Cares is a not-for-profit effort that works to connect people with community/mentors. It was initially a pandemic effort, so I’m not sure how active it is, but worth checking out:

https://brandonwcarson.com/ld-cares/

As part of a thought experiment, I recently asked ChatGPT to create a list of potential ID project for a workplace instructional designer (as opposed to higher ed), and it was a decent list. You can see it here:

List of typical ID projects for workplace

First stop: Connie Malamed’s book Visual Design Solutions: Principles and Creative Inspiration for Learning Professionals.

Connie’s Blog is also full of great visual design resources.

If you want the deep dive into the research behind learning graphics, Ruth Clark’s book Graphics for Learning: Proven Guidelines for Planning, Designing, and Evaluating Visuals in Training Materials, 2nd Edition is a great resource.

Bianca Woods has many delightful resources on her website.

Tom Kuhlmann’s Rapid Elearning blog has years of super useful hands-on examples on how to do elearning visuals.

Let me know if I’m missing any of your favorites!

Book update: Talk to the Elephant: Design Learning for Behavior Change is now available on most sites! Ebooks are everywhere, and physical copies are shipping now from the peachpit.com site, and getting stocked on all the other book sites right now. Thanks!

Book Description:

What do you do when your learners know what to do but still aren’t doing it?

Training is created with the goal of changing learners’ behaviors, but anyone who has created learning experiences knows that there’s a big gap between knowing and doing. You can create an engaging learning experience that informs and helps people remember, but often those people go back to their regular world and continue to do things the same way they always have.

In the last few decades, the fields of psychology, behavioral economics, and other behavioral sciences have brought an enormous amount of scientific research into helping people with behavior change. Only a fraction of that research has made its way back into learning design.

Talk to the Elephant: Design Learning for Behavior Change shows you how to add critical tools to your learning design toolbox to affect behavior change. You’ll find out how to use frameworks and strategies from behavioral science to help you research and analyze challenges, feel more confident that you’re solving the right problem, and design and test solutions that can help people with difficult behavior changes.

By the end of this book, you’ll be able to

• Map the change journey of your learners and identify their path
• Assess and clearly communicate the value of the change
• Use motivation models to better understand what learners really care about
• Utilize evidence-based models like the COM-B Model to analyze behavior-change challenges
• Use a diagnostic checklist to determine whether you actually have a training problem
• Identify behavior-change techniques to address your specific challenges

Available for pre-order at the following:

Pre-order on Peachpit (publisher website)

Pre-order on Amazon

Pre-order on Barnes and Noble

Pre-order on Bookshop.org

Thanks for their input to:

Kristin Hayden Safdie

Heidi Matthews

Jane Bozarth

Topic: xAPI…huh…what is it good for?

xAPI (the specification formerly known as Tin Can) has been available for a while, but what can instructional designers actually *do* with it? If you are a designer who suspects that you should be making use of xAPI but aren’t really sure what you might do with it, check-in for some Nerdy Shop Talk with xAPI gurus, Megan Torrance and Sean Putman. We talk about what the options are for xAPI using current authoring tools, and hear about some of Sean and Megan’s favorite xAPI examples.