What I've Learned So Far

This post originally appeared on the Software Carpentry website.

I worked on Software Carpentry full-time for a year starting in May 2010. In that time I created over 140 PowerPoint decks, about 12 hours of video, and taught the course six times (twice online and four times in person). I also learned a few things that I hope other people will find useful:

Online teaching isn't as effective as in-person teaching.
Today's tools suck.
If content is king, then community is queen, pope, emperor, and president-for-life.
If you don't know how to tell if you succeeded, you're going to fail.
If you don't take the time to explore what other people have done, you deserve to fail.

Online teaching isn't as effective as in-person teaching. "I talk but don't listen, you listen but don't speak" is a lousy teaching model, but that's all recorded video is. Ask-and-answer is much more effective, particularly when you can see the baffled looks on students' faces, but no present-day web-based technology can deliver that for more than a handful of students at a time. In my gloomier moments, I fear there's some sort of conservation law at work: what we gain in reach, we lose in effectiveness. However the experiments we did with chat sessions seemed to work well; we should have experimented with that more, and with desktop sharing as well.

(But hey, did you see what I did there? I said that one kind of teaching isn't as effective as another, but didn't tell you what I mean by "effective", how I'm assessing it, or what data I've gathered. I'll come back to this later.)

Today's tools suck. More specifically, they suck time out of instructors. It takes me half a day to put together a one-hour lecture on something I know well; doing an hour of video takes three days or more. That might be economical if measured in hours of prep per student reached (even when "reached" is scaled down as per the previous point), but what really hurts is how long it takes to do revisions. If I spot a typo in a PowerPoint deck, I can fix it in a couple of minutes. If I want to patch a video, though, it'll take at least half an hour; that's a fifteen-fold increase, which is a powerful disincentive to incremental improvement. What's worse, the result will be disappointing: a person's voice never sounds exactly the same from one day to the next, so patches to a lecture's sound track are painfully obvious without expensive professional editing. And what am I supposed to do if I want to recycle someone else's material? Re-record the whole thing, or drop a couple of minutes of Canadian tenor into someone else's Australian contralto? The only practical solution I can see is to break things down even further than I have: instead of five-minutes videos, I should do thirty-second clips so that re-recording the whole thing is an acceptable cost [1]. I should also:

Deliberately mix up the narrators so that voice changes are regular occurrences instead of occasional jarring distractions.
Provide separate audio tracks for people who want to listen along while paging through the slides, rather than watching a video. (We get a lot of hits from developing countries, where broadband can't be taken for granted.)
Use something like Popcorn.js to link the media, the transcripts, and the screen captures. (Right now, the videos are on the same page as the transcripts and screen captures, but otherwise disconnected.)
Allow in situ commenting.
Embed the source code for all our example programs in the pages as well (assistive tools for the blind don't understand things that appear only as pixels in a screen capture, and neither do search engines).
Embed the diagrams in a format I can hyperlink and dynamically reformat, e.g., as SVG than as JPEGs or PNGs.

None of these things is particularly hard except the last—the tool I want doesn't appear to exist yet. They'd all take time, though, both up front and for maintenance. Before anyone puts in that time, there's another point to consider:

If content is king, then community is queen, pope, emperor, and president-for-life. And not just for users: it's at least as important for developers. I should have stopped creating new content in December 2010 and spent the next four months getting more people involved in the project. Running the course online the way we did wasn't nearly enough; only three grad students made episodes (for pay) since work on Version 4 started in May 2010, and only one volunteer has contributed stuff pro bono. (Thanks, Ethan—you're a star.) A few other people have helped teach the course, but its bus factor is still 1. That isn't just a threat to its future: if only one person is deeply into the project, who can they bounce ideas off? Who will hold down the fort when they're busy with other things? And who will keep them going when their enthusiasm flags?

The answer to this one is something I've long resisted. Lots of open teaching hubs have sprung up in the past few years, like P2PU, Udemy, Student of Fortune, and BlueTeach. They all have some kind of community tooling (some more than others), but none seem to offer better tools for building content than I'm using already. The main argument for moving Software Carpentry there is the same argument behind open source portals like SourceForge and GitHub: better findability, the increased usability that comes from having a standard interface, and so on. The main argument against is the risk of platform capture; the real reason I haven't done it, though, is that I'm reluctant to see my pet project become just another bunch of pages on someone else's site. I'm sure I'll work through that eventually...

My final two takeaways are somewhat personal as well. If you don't know how to tell if you succeeded, you're going to fail. Software Carpentry's goal is to help scientists and engineers use computers by teaching them basic, practical skills. I think I can tell who "gets it" in person, but I have no idea how to tell remotely. How long did it take someone to solve that exercise? How many blind alleys did they go down? And do they now understand the concepts well enough to apply them to the next problem they run into? Scores on timed drill exercises might tell us some things, but without (literally) years of painstaking refinement, they can't help us diagnose misconceptions [2].

More generally, Software Carpentry's real aim isn't to show people how to do today's work faster. It's to show them how to do entirely new kinds of work, things that were out of reach of their old skills. The only way to tell if we've accomplished that would be to observe them before and after the course [3]. Surveys and other kinds of self-reporting can't tell us, not unless they've been calibrated by exactly this kind of before-and-after observation. The problem is, no-one seems to want to fund it: they'll spend millions on supercomputers, and millions more on salaries for faculty and grad students, but not a penny to figure out whether those investments are working, or how to make them pay higher dividends. Things are different in K-12 education, where there's actually too much emphasis on assessment right now (or at least, too much emphasis is misdirected), but even there, we're still struggling to figure out what web- and computer-related skills people ought to have, much less how to tell if they've got 'em.

Finally, if you don't take the time to explore what other people have done, you deserve to fail. This was my biggest mistake; it's no consolation to see others making it as well. Imagine someone came to you and said, "I'm going to build a massive online multiplayer fantasy game, and boy howdy, it's going to be great!" If you asked them how it was going to be different from World of Warcraft^©, and their response was, "What's that?", you'd probably feel as weary as I now do when I hear someone brush aside decades of pedagogical research. How many of the people who are trying to teach online have ever taken an online course themselves? How many have taken more than one? I haven't: I signed up for an online course about online teaching at a local university last summer, but dropped out when it became clear that my fellow students were there to earn professional advancement credits rather than to learn. Now that I am doing my homework, I've discovered dozens of wheels I didn't need to reinvent, and dozens of misconceptions that I shared with most other well-meaning but uninformed newcomers [4]. I'm still struggling to figure out how to apply what I'm learning to the specific problems in front of me, but that's only fair. After all, I'm asking Software Carpentry students to do that every day.

So here's the bottom line. Some people will figure things out no matter what or how you teach, so survivor bias will easily fool you into thinking that your teaching is responsible for their success. Knowing what other people have done, how effective they've been, and how we know isn't just a matter of professional courtesy. It's what you have to do if you're serious about maximizing the odds that someone actually learning from what you're doing.

[1] Lowering production values is of course another option, but some research has shown that this leads to lower student retention. If anyone knows of more recent or more relevant results, I'd welcome a pointer.

[2] As Will Rogers said, "It isn't what we don't know that gives us trouble, it's what we know that ain't so." One of the reasons peer instruction is so effective is that it's better than lecturing at correcting students' misconceptions.

[3] If they go through the material in some kind of course. If they're dipping into it on their own, as they need it, then I have no idea how to tell how much we're helping.

[4] I've also discovered that most books about education are badly written—it's as if Jossey-Bass took Strunk & White, put "don't" in front of each recommendation, then gave it to their authors. I think this is one of the reasons people like me don't do their background research: it's hard to keep plugging away when it feels like being smothered under a pile of wet sheep. Of course, I could just be looking for excuses to skip the thinking bits and get back to coding.