How We Use Feedback at The Carpentries

As you were filling out yet another survey or feedback form, you might have found yourself wondering, “What does The Carpentries do with all of this feedback anyway? Do they even use it? Do they really need to hear from me?” And the answer is yes, we definitely do!

If you have hung around The Carpentries for any period of time, you know that community is core to all we do. Everything is co-created by the community with support from The Carpentries Core Team. But to be able to co-create, we have to have strong lines of communication in all directions. As Core Team members, our job is to listen to you and use your feedback to inform the work we do. It is just as important, however, that we close that feedback loop by sharing back to you, to facilitate your work and to showcase your accomplishments. Since some of us really love concept maps here at The Carpentries, here is a little diagram to illustrate the many ways that we listen and share through our assessment work:

The purpose of this blog post is to share our assessment efforts and highlight the critical role that you, our community members, play in this process. We know we frequently ask for feedback in many different forums and we want you to know why we ask, why your responses matter, and how the work that we do in gathering feedback comes back around to benefit you.

The Ways We Listen

We have many channels where we get to hear from the community. Our current resources include:

• Surveys
  • Pre/Post Workshop Surveys for participants
  • Online Teaching Surveys for Instructors and Helpers
  • Survey to Member Organisations regarding COVID-19
• Feedback forms
  • Community Discussion forms for hosts and participants
  • Teaching Demonstration forms for hosts
  • CarpentryCon @ Home session forms for community and Core Team members
• Blog posts
• Conversations on Slack, Topicbox, and GitHub
• Sub-community meetings (e.g., Members, Trainers, Maintainers, etc.)
• Social media (e.g., Twitter)

Some of these channels are part of our routine assessment program. Others support continual improvement in specific programs and subcommunities. But perhaps most importantly, as we learned in the transition to online training, all of these can be vital resources when we really need to ‘turn on our ears’ and hear what the community has to say.

Routine Assessment

One important way we listen to the community is through routine assessment. While this assessment may be routine, that does not mean it isn’t interesting or important! Without it, we would have no way to know how well our programs are running or where we could improve. Routine assessment is made up of our various surveys and monitoring of programmatic data (e.g. number of workshops run, location of workshops, etc.).

Pre- and Post-Workshop Surveys

The heart of our routine assessment is our Pre- and Post Workshop Surveys. We survey learners before and after each Carpentries workshop. Results from these surveys serve many purposes:

• Instructors and Hosts receive workshop reports that can support program responsiveness at the local level. (More on this below!)
• We review surveys as they come in for valuable insights into what is working and what is not.
• We periodically release summary reports to share our impact with the community; the latest available report from July 2018 can be found here.

Other Routine Assessment

Two other types of assessment are our Long-Term Survey and Programmatic Assessment.

• The Long-Term Survey is conducted every six months and allows us to see how learners are using the skills they learned months after the workshop they attended. The latest available report is from April 2020 and is available here.

• We regularly report on global workshop activity and the growth and diversity of our Instructor community through our Programmatic Assessment report. These reports do not incorporate community feedback, but summarise community accomplishments and compliment the survey reports in illustrating our organisational impact. The latest available report is from October 2018 and can be found here.

Routine assessment may not sound all that exciting, but it paints a critical portrait of our community. Understanding the differences The Carpentries’ community and programs make in your lives reminds each of us why we give our time and energy to this work. It is also key to demonstrating our value to our supporters, including member organisations, granting agencies, and donors.

Looking for more? Many other previous reports and data can be found on our assessment page.

Ongoing Improvements

We feedback of all kinds continuously to verify and improve the health and performance of our programs, procedures, and subcommunities. This becomes especially important as programs change and scale. From signups to event formats to meeting schedules and networking opportunities, we want to make sure we are changing in the right ways, cultivating rewarding interactions at every opportunity, and scaling effectively.

Each of our subcommunities and programs has a Core Team liaison associated with it. While these individuals play many roles, the role of listener is one we take particularly seriously. When you speak up on Slack or bring an idea to a meeting, we are listening. We may not be able to act on every idea, but listening prepares us to seize valued opportunities as they arise and to nudge adjustments in the right direction. Most importantly, our many communications channels ensure that we hear about it when something goes wrong.

Online Training Recommendations: Our Feedback Cycle In Action

Earlier this year it became clear that moving Carpentries workshops online was our only way forward during the Coronavirus pandemic. To figure out how to do that well, we knew we would need to draw upon community expertise and experience with this format. In support of this transition, The Carpentries Core Team undertook a massive listening campaign. We needed to know what worked well, and under what circumstances. In the face of a daunting transition for instructors, we also wanted to know how you felt – what fears and obstacles needed to be overcome to support the transition.

For this project, we took advantage of every communication channel at our disposal.

• We invited instructors to share what they were learning through blog posts. Early blog posts were particularly instrumental in supporting our task force in developing our preliminary recommendations.
• We discussed instructional choices with the Instructor Trainers’ community through meetings and asynchronous conversations.
• We developed a feedback form and later a full survey for instructors who had taught online to share their experiences.
• We held themed community discussions about online teaching, and updated our host feedback forms to collect relevant feedback from standard pre/post discussion meetings.
• We took note of conversations on Slack and Twitter. Our Community Development Team wrote weekly summaries highlighting these, and we returned to read them carefully each time we worked on our recommendations.
• We monitored our pre- and post-workshop surveys from learners, confirming their experiences were still positive and searching for pain points as workshops moved online.

And our Carpentries community delivered! We ended up with tonnes and tonnes of feedback! It was then our turn to try to make sense of it all. Through a monumental team effort and some very messy Google Docs, we distilled the information into a list of Recommendations for Teaching Carpentries Workshops Online which is now in its second version.

Feedback From You and To You

Ultimately, as a community-led organisation, everything we do, we do for you. However, there remain distinctions between data that help The Carpentries Core Team sustain and promote our work, and data that help you sustain and support yours.

Community members use our data to pitch and promote Carpentries membership and participation, to demonstrate the value of their hard work to their employers, to showcase their skills and abilities, and to evaluate the success of their local community programs. We want our assessment program to support all of these use cases! But perhaps the most important application of all is the feedback that directly runs to and from Carpentries Instructors. For any workshop reported to The Carpentries – whether Self-Organised or Centrally-Organised – Instructors are sent links to access the survey data from their workshops.

• Pre-workshop survey data may be used to prepare for a workshop, allowing instructors to customise introductions or examples and anticipate questions or pacing issues.
• Post-workshop survey data lets you know what learners appreciate – you! These surveys may confirm suspicions or alleviate fears about how things went at a workshop; they can also offer new perspectives on the learner experience. It is especially helpful to review post-survey data from a prior workshop before teaching again, as a way to refresh details that may have been forgotten.

In order for us to share these data with you, however, we need to have them in the first place! While all workshop websites and Carpentries email templates contain survey links, active support from Instructors makes a tremendous difference in the response rate for surveys. Wherever possible, we ask that Instructors set aside time during your workshops to allow learners to complete their post-assessment surveys before leaving. Hosts can also help by reminding learners about both pre- and post- assessments in emails before and after the workshop. We know that online workshops have created new pressures on time during a workshop as well as new things to think about before and after, so we do not make this request lightly! Feedback is now more vital than ever as we look to improve our practices in this new medium, and surveys are the most reliable and consistent source of data that we have.

How Can We Help?

Could pre- or post-workshop survey reports provide different data that would be more useful to you? Are there other kinds of reports that you would like to see? While privacy laws and practical considerations sometimes constrain the kinds of data we can release, we want to do everything we can to reward your efforts with ours on behalf of The Carpentries. This means… you guessed it… we have created a feedback form just for feedback on feedback! Embrace the meta and share your ideas about new assessments and/or reports with us using this form.