Assessment and Impact

At The Carpentries we are “Always Learning”. We embody the growth mindset we teach using data and community feedback to measure our impact and foster continuous improvement.


Measuring the Impact of Our Workshops and Trainings

We conduct pre and post workshop surveys for each Carpentries Workshop We Run

Find Copies of our Current Surveys

The Carpentries Pre and Post-Workshop Surveys
Instructor Training Pre and Post Surveys
Long-Term Impact Survey

Additionally, we began collecting data on the long-term impact our workshops are having on Carpentries learners. The Carpentries Long-Term Impact Survey was launched in March 2017, and data is collected every six months.


Reports

We release regular reports on the work and impact of The Carpentries. These reports are published through Zenodo and can be accessed through the links below.

Instructor Training Report
Analysis of Software and Data Carpentry’s Pre- and Post-Workshop Surveys
Post-Workshop Reports
Long-Term Survey Reports

Data

The Carpentries provides a series of data feeds that you may be interested in. More information about these feeds are available in the Data Feeds GitHub repository.

Data sets (provided as csv files) and assessment reports are in a GitHub repository, along with the version of the surveys that were in use at time of analysis.


Measuring The Carpentries Reach

The following reports use data from our AMY database to describe the reach of The Carpentries. This includes number of Instructors certified, number of workshops held, number of learners taught, and number of countries that held Carpentries workshops.


When Do Workshops Work? A Response to the ‘Null Effects’ paper from Feldon et al.(2017) Author: Karen R. Word. Contributors: Kari Jordan, Erin Becker, Jason Williams, Pamela Reynolds, Amy Hodge, Maxim Belkin, Ben Marwick, and Tracy Teal.

This was a collaborative response to the paper: Feldon, David F. et al. Null effects of boot camps and short-format training for PhD students in life sciences Proc Natl Acad Sci U S A. 2017 Sep 12; 114(37): 9854–9858. doi: 10.1073/pnas.1705783114. Our data suggest that we are having a positive impact, and we expect that other short-format programs can be similarly effective. Read our full response.

Webinar with Rochelle Tractenberg: Debrief (2018)

The Assessment Network held a webinar with Rochelle Tractenberg. Dr. Tractenberg directs the Collaborative for Research on Outcomes and Metrics at Georgetown University, where she is a tenured professor in the Department of Neurology. Our starting point was the controversy about short-format training which arose last year, following the publication of a 2017 PNAS paper titled “Null effects of boot camps and short-format training for PhD students in life sciences”. The Carpentries design and deliver short-format training for working with software and data; trainees are researchers from various fields. The Carpentries’ initial response to the paper discussed many ways in which we have been successful with respect to our goals for Software Carpentry and Data Carpentry workshops. However, given that short-format training is a known challenge for generating sustainable content learning, we hoped that Dr. Tractenberg’s expertise might shed some light on areas with room for improvement. See the full webinar.

Edit this page on Github