Our long-term assessment results are in!

This post originally appeared on the Data Carpentry website What concrete changes are people implementing in their computational research practices as a result of completing a Carpentries workshop?

Our long term survey report shows that two-day Software or Data Carpentry workshops are effective for increasing skills and confidence, and the adoption of reproducible research perspectives. We see gains in our survey measures for learners’ motivation to continue their learning, change in reproducible research behavior, and frequency of use of computational skills and tools. We find this very exciting, especially since a recent general survey of bootcamps and short-format trainings reports no measurable impact on skill development or research productivity.

Software and Data Carpentry have taught workshops to over 27,000 learners in 35 countries around the world. Post-workshop survey reports for Software Carpentry and Data Carpentry have consistently shown that people like the workshops, that they know more about importing data sets into R and Python to work with data, write functions, and initialize repositories in git, and that they think they can apply skills immediately to their work.

Our focus has always been on long term change, including:

• Improving learners’ confidence and motivation to use computational tools,
• Changing behaviors around reproducible research and effective computational work, and
• Increasing the frequency and types of computing skills used. Therefore, we launched our first long-term assessment survey in March 2017 to gather quantitative evidence about specific behaviors our learners have adopted and continue to embody six months or more after completing a Carpentries workshop.

Assessment specialists on staff and in the community developed an instrument, based on existing instruments, for collecting information regarding learners confidence and motivation to use the tools they learned, and behaviors they adopted after attending a Carpentries workshop. Rather than focusing on learners’ skills in particular tools, we focused on assessing learner confidence, motivation and adoption of good research practices, as these elements represent the primary goals of our workshops. Confidence and motivation are important factors for learners to have to continue their learning. They also promote community building, a significant focus area of the Carpentries.

The final survey instrument included items for self-reported behaviors around good data management practices, change in confidence in the tools they learned, and other ways the workshop may have impacted learners (ex. Improved research productivity). Over 530 people who took a Software or Data Carpentry workshop 6 months or more ago responded to our long-term survey. These results show that workshop respondents had a positive impression of the workshop and the majority felt their skills and perspectives have changed as a result of attending the workshop. Results also show that these two-day impactful workshops are effective for increasing skills and confidence. The impact of these workshops is apparent in respondents’ coding practices. The majority of respondents (70%) reported having improved their coding practices by using programing languages like R or Python or the command line to automate repetitive tasks, by reusing code for other purposes, or by using databases to manage large data sets. Respondents have continued their learning and incorporated use of these tools into their weekly or daily work. Additionally, sixty-nine percent of respondents have made their analyses more reproducible as a result of completing a Carpentries workshop by reusing code and making their data and analyses available on public repositories.

Not only do these two-day coding workshops increase respondent’s daily programming usage, eighty-five percent of respondents have gained confidence in working with data and open source tools as a result of completing the workshop. The long-term assessment data showed a decline in the percentage of respondents that ‘have not been using these tools’ (-11.1%), and an increase in the percentage of those who now use the tools on daily basis (14.5%).

Highlights from our long-term survey

The majority of our respondents:

• Gained confidence in the tools that were covered during their workshop (85.3%).
• Improved their coding practices (63.1%).
• Received professional recognition for their work as a result of using the tools they learned (64.7%).

Respondents also substantially increased their frequency of use of programing languages (R, Python, etc.), databases (Access, SQL, etc.), version control software and/or the Unix shell, incorporating these tools into their regular workflows. Nineteen percent of respondents transitioned from using these tools once a month or less to weekly or daily use per the figure below.

Respondents perceive the workshop had an impact on their confidence, as well as their productivity, reproducibility and coding practices. Interestingly, respondents also felt that the workshops had a positive impact on their career as a whole, and some received recognition for their work.

The figure below shows what impact survey respondents felt for several factors including career, confidence, and continuous learning. Respondents were asked to rate their level of agreement (1-Strongly disagree to 5-Strongly agree) with the statements below. The x-axis labels for the figure are in italics, and correspond to the statement following.

• Reproducible: I have made my analyses more reproducible as a result of completing the workshop.
• Recognition: I have received professional recognition for my work as a result of using the tools I learned at the workshop.
• Productivity: My research productivity has improved as a result of completing the workshop.
• Motivation: I have been motivated to seek more knowledge about the tools I learned at the workshop.
• Confidence: I have gained confidence in working with data as a result of completing the workshop.
• Coding: I have improved my coding practices as a result of completing the workshop.
• Career: I have used skills I learned at the workshop to advance my career.

The figure shows that respondents agree or strongly agree that they gained confidence in working with data (85.3%), made their analyses more reproducible (69.4%), and receiving professional recognition for their work (64.7%) all as a result of attending a Software or Data Carpentry workshop.

From this figure we also see that there are opportunities to improve. Motivation to seek more knowledge seems unchanging, likely because learners who attend our workshops are already motivated. Perhaps these learners remain enthusiastic, having high pre-workshop motivation scores, rather than a decrease in motivation post-workshop. We’d also like to see a shift in positive trend for learners using the skills they learned to advance their career, which is why we are implementing round two of the Carpentries Mentoring Program this fall.

Interested in reading more? The full report is available, and it provides more detailed information about the motivation behind this survey, respondent demographics, and growth opportunities. Take a look at the report. We will continue to conduct this assessment at 6 month intervals to capture feedback from people who took workshops 6 months or more ago. Additionally, assessment will be the focus of our October community call. Bring your thoughts to the community call October 19th!

The surveys used in this work, anonmyized data, and R scripts for generating the figures are available in our assessment repository. This report was made possible by community input from Ben Marwick, Belinda Weaver, Naupaka Zimmerman, Jason Williams, Tracy Teal, Erin Becker, Jonah Duckles, Beth Duckles, and Elizabeth Wickes. We thank you all so much for your contributions to the code in this report and development of our long-term survey!

If you have other questions about the data or results, please use the data, re-analyze the results or ask your own questions!

What strikes you? Comment below, and tweet us your thoughts at @datacarpentry @swcarpentry and @drkariljordan using the hashtag #carpentriesassessment.

Thanks to the Gordon and Betty Moore Data Driven Discovery initiative for support of Data Carpentry and these assessment efforts.