Independent Assessment of the Past Six Months

This post originally appeared on the Software Carpentry website.

As many of you know, Dr. Jorge Aranda has been doing an independent assessment of Software Carpentry's effectiveness over the past six months. His brief was to tell us whether we were having an impact on scientists' lives, and if so, what kind, and how we could do better. He just submitted his final report (PDF), and we would welcome your thoughts.

Execute Summary

This report summarizes a six-month effort to assess the efficacy of the Software Carpentry program. Through a mixed-methods approach, including surveys, pre- and post-workshop interviews, workshop observations, and screencast analysis, this assessment concludes that the key premises for the usefulness of Software Carpentry instruction hold true: most scientists are self-taught programmers, they have fundamental weaknesses in their software development expertise, and these weaknesses affect their ability to answer their research questions.

More importantly, this assessment concludes that Software Carpentry instruction helps scientists eliminate these weaknesses. The program increases participants' computational understanding, as measured by more than a two-fold (130%) improvement in test scores after the workshop. The program also enhances their habits and routines, and leads them to adopt tools and techniques that are considered standard practice in the software industry. As a result, participants express extremely high levels of satisfaction with their involvement in Software Carpentry (85% learned what they hoped to learn; 95% would recommend the workshop to others).

While the outcome is largely positive, there are areas for improvement. Two of note are the spread in expertise among participants and the barriers that they face to change their software practice. The first problem leads to some participants feeling that the instruction is too advanced or too basic for them. The second problem damages the impact of Software Carpentry instruction, as participants learn valuable material, but find that for other reasons they are unable to adopt the relevant tools and techniques. Both problems, and other minor issues identified, can be addressed by the Software Carpentry team in subsequent versions of their workshop.