The Carpentries Programmatic Assessment
The first quarter of 2018 brought many changes our way – including a merged Carpentries under the fiscal sponsorship of Community Initiatives, new staff, and a new Executive Council. As our community develops and grows, we want to be able to understand the kinds of growth we are experiencing so we can share our successes and direct our resources to best support and sustain this growth. In the past, our assessment efforts have focused on understanding the impact of our workshops on learners. We’ve learned that our workshops are successful in improving learner confidence in the tools covered in their workshop and increasing daily programming usage. Knowing that our workshops are effective, however, is only part of the story. We also need to understand how our programs as a whole are functioning to know whether our community is growing in a balanced and sustainable way, and to be able to direct our resources effectively.
In past programmatic reports, we have explored instructor training checkout and teaching rates. We also published reports to the Moore Foundation (a funder of Data Carpentry) in 2016 and 2017 detailing our work.
Beginning this month, we will be publishing quarterly reports with data about our programs, including the growth rates of our workshops and instructor training programs, our geographic impact, and other metrics. We know we have a strong team of data scientists in our community and welcome your involvement in helping us analyse, visualise, and understand the data we have been collecting. We have a rich set of data and want to understand the types of questions that our community members would like to be able to answer from this data.
We’ve put together an initial draft analysis of this data and would now like to invite the community to get involved with shaping the analyses and the final report. Please speak up and tell us how this data can be used to answer questions that are important to your community. We greatly value involvement of our community members in conducting these analyses, as your perspectives can help us to understand what questions are worth asking. Here are some ways you can get involved in our programmatic assessment:
Help address some issues we’ve encountered in analysing this data
Explore these analyses interactively in the Python Jupyter notebook
Download the raw data and conduct your own analyses
Create new issues or pull requests about analyses that you would like to see from this data
Leave new issues with your comments on the report narrative to help shape The Carpentries’ follow-up actions
Participate in office hours next week to discuss these analyses and share your thoughts. Please sign up on the Etherpad if you are planning on attending.
We’ll be conducting these types of analyses on a quarterly basis so we’ll have ongoing opportunities for you to get involved over the year. This process of seeking contributors early in the assessment process is inspired by Kari Jordan’s process for bringing collaborators into workshop assessment reporting. Our workshop assessment has been done entirely in R. If you’re a Pythonista or have ideas for evaluating workshop data, here’s your chance to share your skills and contribute to our reports.
Don’t want to set up a local Python environment? You can still view the Jupyter notebook interactively in Binder.
If you have any questions or comments, please feel free to contact me. I’m looking forward to working on this with all of you!