The Carpentries value a culture of assessment, and we are consistently evaluating our workshops to understand the impact workshops have on our learners, and how we can improve our content delivery.
Dr. Kari L. Jordan, our Executive Director, leads our assessment efforts. She has established the Assessment Network, a space where those working on assessment within the open source/research computing community can collaborate and share resources.
Dr. Erin Becker, our Associate Director, serves as Principal Investigator for our research efforts related to assessing the effectiveness of our workshops. This work is carried out under supervison of the Institutional Review Board at University of California, Davis. Our faculty sponsor at UC Davis is Dr. Megan Welsh, Assistant Professor within the UC Davis School of Education.
We conduct pre and post workshop surveys for each workshop
The Carpentries Pre and Post-Workshop Surveys
Instructor Training Pre and Post-Workshop Surveys
Additionally, we began collecting data on the long-term impact our workshops are having on Carpentries learners. The Carpentries Long-Term Impact Survey was launched in March 2017, and data is collected every six months.
We release regular reports from our survey results. These reports are published through Zenodo and can be accessed through the links below.
Analysis of Software and Data Carpentry’s Pre- and Post-Workshop Surveys
The analysis in this report serves the following purposes:
- To inform the community of the impact Data Carpentry workshops have made on its learners.
- To provide context for the survey responses as they relate to Data Carpentry learners.
- To discuss what Data Carpentry is doing well, areas of improvement, and questions we should be asking.
The report finds that learners are reporting that after a workshop, their level of data management and analysis skills have increased, they have increased confidence in their ability to use these skills and that the learners have increased appreciation for these skills (i.e. scripting) to improve and promote reproducible research. 95% of learners agree or strongly agree that they would recommend the workshop to a colleague.
- Analysis of Software Carpentry’s Post-Workshop Surveys . The data used in this report is available in this repository.
Long-Term Survey Reports
The long-term survey assessed confidence, motivation, and other outcomes more than six months after respondents attended a Carpentry workshop. Provided below are a few highlights from the data.
- 77% of our respondents reported being more confident in the tools that were covered during their Carpentry workshop compared to before the workshop.
- 54% of our respondents have made their analyses more reproducible as a result of completing a Carpentry workshop.
- 65% of our respondents have gained confidence in working with data as a result of completing the workshop.
74% of our respondents have recommended our workshops to a friend or colleague.
- Analysis of The Carpentries Long-Term Surveys (April 2020) These results are also available on GitHub.
Programmatic Assessment Reports
June 2018 This report reviewed Carpentries workshop activity from 2012 through the first quarter of 2018.
This report follows activity around Carpentries’ workshops, Instructors, and Trainers from 2012 through the first quarter of 2018 (January - March). From 2012 through March 2018, 58 Trainers badged 1,480 Instructors. Carpentry Instructors taught 1,332 workshops, reaching more than 37,000 learners in 44 countries. The report also highlights our significant geographic expansion, such as the growth of an active community of Instructors and Trainers in eight African countries.
August 2018 In this report we revisited and updated some of the workshop related data from the earlier report. We also analysed applications for instructor training received since we opened up our application process in July 2016. Our goal here was to compare the applicant pool with the badged instructor pool and to examine and map application and completion rates across certain characteristics such as country, occupation, areas of expertise, and previous experience and involvement with The Carpentries. Only those applications that came through an open application process (that is, people unaffiliated with existing member organisations) were analysed. Since The Carpentries does not control instructor trainee selection at member sites, including these applications might have skewed our analysis.
October 2018 In this report, we provide an overview of global workshop activity, including the number of workshops and learners involved globally. We also look at instructor training applications to better understand the backgrounds of people applying to our instructor training program. Additionally we review the growth of our Instructor Trainer community globally.
Other Assessment-Related Publications
When Do Workshops Work? A Response to the ‘Null Effects’ paper from Feldon et al. Author: Karen R. Word. Contributors: Kari Jordan, Erin Becker, Jason Williams, Pamela Reynolds, Amy Hodge, Maxim Belkin, Ben Marwick, and Tracy Teal.
This was a collaborative response to the paper: Feldon, David F. et al. Null effects of boot camps and short-format training for PhD students in life sciences Proc Natl Acad Sci U S A. 2017 Sep 12; 114(37): 9854–9858. doi: 10.1073/pnas.1705783114. Our data suggest that we are having a positive impact, and we expect that other short-format programs can be similarly effective. Read our full response.
On February 2, the Assessment Network held a webinar with Rochelle Tractenberg. Dr. Tractenberg directs the Collaborative for Research on Outcomes and Metrics at Georgetown University, where she is a tenured professor in the Department of Neurology. Our starting point was the controversy about short-format training which arose last year, following the publication of a 2017 PNAS paper titled “Null effects of boot camps and short-format training for PhD students in life sciences”. The Carpentries design and deliver short-format training for working with software and data; trainees are researchers from various fields. The Carpentries’ initial response to the paper discussed many ways in which we have been successful with respect to our goals for Software Carpentry and Data Carpentry workshops. However, given that short-format training is a known challenge for generating sustainable content learning, we hoped that Dr. Tractenberg’s expertise might shed some light on areas with room for improvement. See the full webinar.