The purpose of this blog post is to share our assessment efforts and highlight the critical role that you, our community members, play in this process. We know we frequently ask for feedback in many different forums and we want you to know why we ask, why your responses matter, and how the work that we do in gathering feedback comes back around to benefit you.
We have many channels where we get to hear from the community. Our current resources include:
Some of these channels are part of our routine assessment program. Others support continual improvement in specific programs and subcommunities. But perhaps most importantly, as we learned in the transition to online training, all of these can be vital resources when we really need to ‘turn on our ears’ and hear what the community has to say.
One important way we listen to the community is through routine assessment. While this assessment may be routine, that does not mean it isn’t interesting or important! Without it, we would have no way to know how well our programs are running or where we could improve. Routine assessment is made up of our various surveys and monitoring of programmatic data (e.g. number of workshops run, location of workshops, etc.).
The heart of our routine assessment is our Pre- and Post Workshop Surveys. We survey learners before and after each Carpentries workshop. Results from these surveys serve many purposes:
Two other types of assessment are our Long-Term Survey and Programmatic Assessment.
The Long-Term Survey is conducted every six months and allows us to see how learners are using the skills they learned months after the workshop they attended. The latest available report is from April 2020 and is available here.
We regularly report on global workshop activity and the growth and diversity of our Instructor community through our Programmatic Assessment report. These reports do not incorporate community feedback, but summarise community accomplishments and compliment the survey reports in illustrating our organisational impact. The latest available report is from October 2018 and can be found here.
Routine assessment may not sound all that exciting, but it paints a critical portrait of our community. Understanding the differences The Carpentries’ community and programs make in your lives reminds each of us why we give our time and energy to this work. It is also key to demonstrating our value to our supporters, including member organisations, granting agencies, and donors.
Looking for more? Many other previous reports and data can be found on our assessment page.
We feedback of all kinds continuously to verify and improve the health and performance of our programs, procedures, and subcommunities. This becomes especially important as programs change and scale. From signups to event formats to meeting schedules and networking opportunities, we want to make sure we are changing in the right ways, cultivating rewarding interactions at every opportunity, and scaling effectively.
Each of our subcommunities and programs has a Core Team liaison associated with it. While these individuals play many roles, the role of listener is one we take particularly seriously. When you speak up on Slack or bring an idea to a meeting, we are listening. We may not be able to act on every idea, but listening prepares us to seize valued opportunities as they arise and to nudge adjustments in the right direction. Most importantly, our many communications channels ensure that we hear about it when something goes wrong.
Earlier this year it became clear that moving Carpentries workshops online was our only way forward during the Coronavirus pandemic. To figure out how to do that well, we knew we would need to draw upon community expertise and experience with this format. In support of this transition, The Carpentries Core Team undertook a massive listening campaign. We needed to know what worked well, and under what circumstances. In the face of a daunting transition for instructors, we also wanted to know how you felt – what fears and obstacles needed to be overcome to support the transition.
For this project, we took advantage of every communication channel at our disposal.
And our Carpentries community delivered! We ended up with tonnes and tonnes of feedback! It was then our turn to try to make sense of it all. Through a monumental team effort and some very messy Google Docs, we distilled the information into a list of Recommendations for Teaching Carpentries Workshops Online which is now in its second version.
Ultimately, as a community-led organisation, everything we do, we do for you. However, there remain distinctions between data that help The Carpentries Core Team sustain and promote our work, and data that help you sustain and support yours.
Community members use our data to pitch and promote Carpentries membership and participation, to demonstrate the value of their hard work to their employers, to showcase their skills and abilities, and to evaluate the success of their local community programs. We want our assessment program to support all of these use cases! But perhaps the most important application of all is the feedback that directly runs to and from Carpentries Instructors. For any workshop reported to The Carpentries – whether self-organised or centrally organised – Instructors are sent links to access the survey data from their workshops.
In order for us to share these data with you, however, we need to have them in the first place! While all workshop websites and Carpentries email templates contain survey links, active support from Instructors makes a tremendous difference in the response rate for surveys. Wherever possible, we ask that Instructors set aside time during your workshops to allow learners to complete their post-assessment surveys before leaving. Hosts can also help by reminding learners about both pre- and post- assessments in emails before and after the workshop. We know that online workshops have created new pressures on time during a workshop as well as new things to think about before and after, so we do not make this request lightly! Feedback is now more vital than ever as we look to improve our practices in this new medium, and surveys are the most reliable and consistent source of data that we have.
Could pre- or post-workshop survey reports provide different data that would be more useful to you? Are there other kinds of reports that you would like to see? While privacy laws and practical considerations sometimes constrain the kinds of data we can release, we want to do everything we can to reward your efforts with ours on behalf of The Carpentries. This means… you guessed it… we have created a feedback form just for feedback on feedback! Embrace the meta and share your ideas about new assessments and/or reports with us using this form.