Counting to Five (or, A Plan for Online Tutorials and What's Wrong With It)

This post originally appeared on the Software Carpentry website.

My daughter is five and a half years old, and some time in the last couple of weeks, she made a small cognitive breakthrough. It used to be that if I asked her, "What's five plus three?", she would carefully count five fingers, then count three, then go back and count them all together to get eight. She doesn't do that any more; instead, she holds up five fingers, then counts, "Six... seven... eight!" to get the answer. It may sound like a small thing, but it's not: the idea that five is always five, even if you didn't count it this time around, is such a big one that we forget it ever had to be learned.

A lot of my time these days goes into trying to count from five in online learning. While an ignorance of prior art might be considered an asset in Silicon Valley, I have enough failures and harder-than-it-should-have-beens behind me to make me want to build on what people have done before rather than reinvent things, even if the latter is more fun. What kinds of distance learning have worked in the past? How many of those ideas still work online? What new ideas work? What doesn't work, and why not? And how do we know? I don't believe for a moment that all the answers I want are already out there, but I'd have to be pretty arrogant to believe that none of them are, or that it'd be faster for me to rediscover things than read about them.

For example: between March and July we ran two dozen online tutorials for people who'd been through Software Carpentry workshops. We used BlueJeans (and later Vidyo) to share the presenter's desktop with the learners, and two-way audio (later supplemented with IRC-style text chat) both for questions directed at the presenter, and sideways Q&A between learners. The net effect was exciting and frustrating in equal measure: exciting because we could actually do it, and frustrating because the technology didn't work for some people, and the experience was never really satisfying for anyone. Even when learners were sitting together in small groups so that they could have high-bandwidth interactions with each other during the tutorial, the result was a pale imitation of being there in person.

At the same time, though, I was helping people one-to-one, and that seemed to work a lot better. These sessions typically started with a "can you please help me?" email, then turned into a Skype call and some desktop sharing. I found was that diagnosing a novice's problem was a lot easier when I could see what they were doing (or when they could see what I was doing—we did it both ways). The reason was that almost by definition, novices don't know how to explain their actual problem: if they knew enough to say, "I think my EDITOR variable must be set incorrectly," they'd almost certainly be able to fix the problem themselves. Email back and forth with someone who's still trying to get from A to B is a lot like a game of Twenty Questions: can you run this command and send me its output, have you edited that file, and so on. A few seconds of interaction short-circuited a lot of that, which made it much more satisfying on both sides.

That experience suggests a different kind of online teaching than the MESS (Massively Enhanced Sage on the Stage) being used by Coursera, Udacity, the Khan Academy, and so on. Grossly misrepresented for rhetorical purposes, their model is:

1. Watch a recording of a sage explaining something in a single fixed way.
2. Try to do something that the sage decided would be good for you.
3. If it doesn't work, hope that the static hints embedded in the robograder trigger a moment of satori.
4. If not, try to translate your incomplete and imperfect understanding of what you're actually doing and why it isn't working into a coherent explanation.
5. Hope that someone reads it, that you happened to include enough of the right information for them to diagnose your actual problem, and that they are able to write an explanation that you can understand given your current state of knowledge.
6. Repeat.

Minus steps 1 and 2, this is the model that Stack Overflow and similar sites use as well. I know from conversation with workshop participants than only a handful (certainly less than 10%) ever post a question on SO or similar sites. I used to think the reason was that people don't want to look stupid, but I now believe that the cost of translating "this doesn't work" into a complete enough description for someone else to work with, and the number of long-latency round trips required to converge on the actual problem and its diagnosis, are contributing factors as well.

But what if we used 21st Century technology for Q&A instead of souped-up 1970s-vintage bulletin boards? What if there was a "help me now" button on Stack Overflow that was enabled whenever the person who'd asked the question was online? When clicked, it would connect the clicker and the person needing help via VoIP and desktop sharing (with some kind of "are you sure?" handshake at the outset) so that people could get real-time interactive help when they needed it. Based on what I've seen in the last five months, this would be a lot more effective than anything with the translation bottleneck and latency of a bulletin board.

This is where "counting from five" comes in. Somebody must have done this before, or done something similar enough to tell me what potholes there are on this road. Enter Scott Gray, who's been building, using, and running online educational systems for two decades, and is now the head of the O'Reilly School of Technology. We chatted today, and here's what he said:

• GW: did you see the Mozilla Towtruck demo?
• GW: just a prototype, but the idea is to share your browser session with a tutor/helper/more knowledgeable friend when/as you need a few minutes of help
• GW: we're already finding with scientists that "hey, can we share a desktop for a couple of minutes to figure this out?" is very effective, both on the learning side, and on the engagement side.
• SG: Isn't Stackoverflow a low feature version of that?
• GW: The "low feature" adjective is the killer: novices don't understand what's going wrong well enough to summarize enough relevant detail to permit diagnosis—Catch-22
• GW: There are obviously exceptions (novices do post answerable questions on SO) but I think it's a much more stringent filter than people who've passed through it realize
• SG: Yeah... seems like adding the hookup would be a lot of help: it's what we do at work but with skype. First we use the low feature texting like we are now.
• GW: Do people screenshare with skype? Has anyone done an analysis of what they actually show each other, and how much accidental/sideways knowledge transfer is taking place (i.e., how often person X notices something that person Y didn't explicitly communicate, but which turns out to be relevant?)
• SG: My employees are spread out all over the place; we use either Skype or Google hangout constantly.
• SG: For actually teaching courses I have seen it used and abused: as far back as the mid-nineties we used CU See ME.
• SG: They use Illuminate now. I watched them use it this past weekend, and I think that it can be massively improved.
• SG: I think complete share with voice should be the last of an escalating use of communication tools all available in whatever learning or tutorial tool is being used.
• SG: It's like right now: we're nearing what we can do with just text. We're communicating but we are reaching a time where we'd be more efficient actually talking, but we didn't know that 20 minutes ago.
• GW: Do you think the initial async/text exchange between helper and tutor will be productive often enough to make it worth doing, or will they be better off jumping to real-time/full info early?
• GW: I.e., should we flip the order in which we do things when one or both of us knows a lot less about what we're talking about? :-)
• SG: To your first question, absolutely.
• GW: Absolutely the text exchange will be productive, or absolutely they should leapfrog early?
• SG: The first.
• GW: Do you have any data to back that up? Looking at Stack Overflow, and talking to scientists who don't post questions, I think there's huge survivor bias in the sample, but I don't have anything more substantial than post-workshop conversations to back that up.
• SG: What goes on at GITHUB? If everyone had to do sync, no one would. No one has that kind of time. So screen sharing is a last resort not first.
• GW: Right—I don't think sync scales to the whole planet, but is it optimal for things like an online intro-to-programming class? A few thousand people, all with some declared stake in the game?
• SG: No. I have 6000 students, all novices; we don't use any sync, ever.
• GW: Would they if it was available? Would they use it with each other (the only scalable approach)?
• GW: It works for the online gaming community, big time—lots of people helping lots of other people learn to be better dwarf axe throwers and whatever in WoW :-)
• SG: Gaming and programming are completely different.
• GW: OK—a flawed analogy is a bad point to start design from :-)
• SG: Programming is a text based endevor: it's async by its very nature. So are literature and mathematics.
• GW: Ah, OK, then that's where we disagree
• GW: If I'm trying to figure out what's wrong with a novice's program, I want to run it on various inputs, or tweak a line or two and see what happens
• GW: If I'm trying to figure out why they can't install X, I want to run commands to check settings, env variables, etc., and that's a very interactive process.
• GW: The program is text, but programming is a verb.
• SG: Yep.
• GW: So maybe it's a difference between helping them with their program, and helping them with their programming.
• SG: But when we talk sync and async we're talking human to human. Both can be done async.
• SG: The problem is the ability to easily share asnychonus activities: think Google Docs but for programming.
• GW: hm...

All right: if someone with (literally) ten times as much experience as me tell me I'm off track, I ought to listen carefully and think a little more, even if it means discarding what seemed to me to be a very promising idea. Another alternative is to try to find a really cheap way to cobble together some low-fidelity experiments to get a better feeling for how this would work in practice—the equivalent of a napkin sketch in UI design. If you have insights to share, please share 'em.