Alpha testing a Jekyll Pages lesson

In the past several months, we have been developing a lesson, Building Websites with Jekyll and GitHub Pages, in The Carpentries Incubator. In November 2020, we taught this material for the first time in two mini-workshops hosted by UW-Madison.

The first time a lesson is taught is always an important step in its development. It is a chance to really assess how well the lesson objectives and material fit the target audience.

It is also usually the first time exercises have been tested by learners, and can expose weaknesses in the material that are difficult to spot from our perspective as a lesson author. In the life cycle of a lesson, The Carpentries refer to these initial trial runs undertaken by lesson authors as alpha pilots.

After reflecting on how valuable the experience was to the ongoing development of our lesson, we have written up these notes to share with the community.

We will also host an open themed community discussion on the related topic of Teaching a new lesson for the first time at 15:00 UTC on 1st February (check your local time here). If you would like to join that call, head over to the Community Discussions Etherpad to sign up! (Thank you to Caroline F Ajilogba of the Instructor Development Committee for helping us organise this themed discussion.)

What went well?

The best thing about teaching the lesson for the first time was how satisfying it felt! After months of working on the repository, taking the lesson from a collection of objectives and key points through to an (almost) full set of exercises and episodes, it was great to stand (virtually) in front of an audience and deliver a trial run.

In terms of the material itself, our experience in these two workshops suggested that the lesson flows well from episode to episode.

Despite the authors being distributed across four different countries and three different timezones, much of the lesson development took place in virtual sprint sessions (including one before the second trial run, where we addressed the larger issues identified from the first pilot). These regular opportunities for synchronous collaborative development helped produce a relatively cohesive result.

Furthermore, scheduling the workshops gave us a deadline to work towards, and motivated us to focus on reviewing the flow and consistency of the earlier sections of the lesson rather than spending time creating more advanced episodes.

What did we learn?

However, there were many improvements to be made and the two mini-workshops provided opportunities to gather a lot of feedback. This really is one of the main benefits of running an alpha pilot: when you want to identify gaps and inconsistencies in a lesson, there really is no substitute for teaching it to a real group of learners!

Thanks to the strategies The Carpentries’ Instructor Training teaches about formative assessment and collecting feedback, we left the pilot workshops with a list of new issues to begin working through on the lesson repository.

Improved timing estimates

The main thing we learned from the pilot workshops was that it would take longer than three hours to teach the lesson! The shorter duration of these mini-workshops meant that we were only able to cover roughly the first half of the material we had prepared, but for these episodes we were able to collect information on the time required to teach the content and complete the exercises.

Finding the “rough edges” in a lesson

By the end of the first pilot, we had realised two important things: that the lesson devoted too much time to teaching various features of Markdown syntax, and that the lesson would benefit from a summary example that we could talk through to show what can be achieved with Jekyll after learners have become comfortable with the basics. These were the two main points we addressed in the development sprints held between pilots.

Developing instructor notes

We also began to record our experience teaching the lesson, in the form of Instructor Notes for others to benefit from in the future. At the moment, that collection of notes is quite limited, but we are planning to hold further pilots and will add to the pages as we cover more of the lesson in future.

Huge thanks to Sarah Stevens for organising and hosting the two pilots, and the Helpers, Trisha Adamus, John Caskey, and Scott Prater, who ensured that anything that was less than smooth about the workshops was a result of the lesson material and nothing else!

What’s next for the lesson?

We hope to run more alpha pilots very soon. In particular, we are aware that the second half of the material is less mature than the first as it has not yet been tried out “live” in a workshop. After these additional alpha pilots have been completed, you will hear from us again when we announce the lesson as ready for beta testing by other Instructors!

Lastly, if you want to get involved with the ongoing development of the lesson, we have more sprints planned for the first few months of 2021 - post in our Slack channel, open an Issue/Pull Request on the lesson repository, or send us an email if you would like to join us. We would be delighted to hear from you!