Piloting Open Peer Review of Lessons

For the past few years, The Carpentries curriculum community has been designing and developing lessons in The Carpentries Incubator. As illustrated by the first figure in Zhian Kamvar, Lesson Infrastructure Technology Developer’s post from last year, about alpha testing the infrastructure redesign, this lesson development is now happening at a huge scale. At the time of writing, the Incubator hosts more than 100 lessons at various stages of the development cycle. We have been supporting these efforts by providing training on lesson development, highlighting the ways community members can contribute to the development of lessons in a regular series of Incubator Lesson Spotlight posts on this blog, and maintaining channels for this subcommunity of lesson developers to communicate and collaborate.

Today, we are thrilled to announce the next step towards ensuring our community members receive credit for creating high-quality, open-source lessons.

The Carpentries Lab is a space for lessons that have passed through peer review. Community-developed lessons are eligible for submission to The Carpentries Lab when they reach a state where they can be used by anyone familiar with The Carpentries teaching methods and sufficiently knowledgable on the lesson subject matter. The single lesson currently published in The Carpentries Lab, Python for Atmosphere and Ocean Scientists, which was reviewed and accepted at the Journal of Open Source Education (JOSE), is a great example of what we are aiming to collect there.

In the coming weeks, we will begin piloting a process of open peer review for lessons developed by our community, for acceptance into The Carpentries Lab. We are delighted to collaborate with JOSE on this effort, so that lesson authors will have the option to submit their lesson for publication at the journal after passing through The Carpentries Lab review process.

A sibling journal to the Journal of Open Source Software (JOSS), JOSE publishes articles that describe open source materials and software tools for educational use. It is an entirely volunteer-led initiative, and a very good fit to the aims we have for open lesson review. Our hope is that, by providing a way for authors to improve their lessons and received authorship credit for their work, we can reward the time and effort our community members have put into producing a lesson that is ready to be re-used by others.

JOSE are looking for volunteers to become reviewers. This provides a good opportunity for members of The Carpentries community to support the project and gain valuable experience in an open review process. To volunteer as a reviewer at JOSE, you can fill out this form.

The primary purpose of the peer review process will be to improve the quality of the lessons submitted. In The Carpentries Lab, feedback from editors and reviewers will focus on the design of the lesson, the value of its challenges as tools for formative assessment, the accessibility of the content, and the support provided to Instructors who wish to teach it in a workshop. The review process will be coordinated by members of The Carpentries Curriculum Team and reviewers will be members of The Carpentries community wherever possible. Reviews will take place as publicly-visible issue threads on the GitHub repository, in a system inspired by ROpenSci’s open peer review of software.

To indicate to other community members that accepted lessons have undergone this process and met the standard expected, authors will receive a badge and banner for use in their lesson repository and pages respectively, a new URL under https://carpentries-lab.org/, and be listed in The Carpentries Lab community on Zenodo. If authors opt to also submit the lesson to JOSE, they will need to write a short paper to accompany the lesson. This will be reviewed at JOSE and, if it is accepted, the lesson and paper will receive a CrossRef DOI and be published in the journal. Authors can read more about what is required for lessons to be reviewed and accepted at JOSE in their online documentation.

In the coming weeks, we will be reaching out to authors of lessons in The Carpentries Incubator to invite submissions to The Carpentries Lab. During this intial pilot phase, we will commit only to reviewing those lessons that are invited to submit. Watch this space for our reflections on these early reviews and updates on the next phase of this new program!

If you are interested in learning more about lesson development with The Carpentries, check out the #lesson-dev channel on The Carpentries Slack, or stop by one of our monthly co-working sessions.