Tracking Utility and Impact

This post originally appeared on the Software Carpentry website.

Mark Guzdial recently posted some interesting (and for us, slightly depressing) statistics about MIT Open Courseware. Long story short, it looks like that flagship effort isn't as widely used as many of us had hoped or believed. I can't find equivalent stats for the Khan Academy (a scrappy "agile" alternative to MOC that has been getting a lot of attention in the geek community), but even if I could, those numbers probably wouldn't answer my real question: who is this reaching, and what impact is it having? As Guzdial points out, asking users to self-report is subject to large sampling bias, as is googling for links back to lectures. Our sponsors want to know who's using our stuff, and whether it's helping them: how do we answer those questions?