Webinar with Rochelle Tractenberg: Debrief

This post originally appeared on the Data Carpentry website

On February 2, the Assessment Network held a well-attended webinar with Rochelle Tractenberg. Dr. Tractenberg holds two PhDs, one in Cognitive Sciences and the other in Statistics, and directs the Collaborative for Research on Outcomes and Metrics at Georgetown University, where she is a tenured professor in the Department of Neurology. It was a great privilege for our community to be able to engage in a conversation with her and to learn from her expertise.

Our starting point was the controversy about short-format training which arose last year, following the publication of a PNAS paper titled “Null effects of boot camps and short-format training for PhD students in life sciences.” The Carpentries design and deliver short-format training for working with software and data; trainees are researchers from various fields. The Carpentries’ initial response to that paper discussed many ways in which we have been successful with respect to our goals for Software and Data Carpentry workshops. However, given that short-format training is a known challenge for generating sustainable content learning, we hoped that Dr. Tractenberg’s expertise might shed some light on areas with room for improvement.

Dr. Tractenberg identified two of our strategies (i.e., “meet learners where they are” and “explicitly address motivation and self-efficacy”) as areas which could benefit from our leveraging of tools and concepts from educational psychology. So,Dr. Tractenberg introduced us to Bloom’s taxonomy (1956) and Messick’s criteria (1989). Bloom’s taxonomy comes with six levels of learning objectives, corresponding to increasing and accumulating complexity in thinking. These are:

• Remember/Reiterate
• Understand/Summarize
• Apply/Illustrate
• Analyze/Predict
• Create/Synthesize
• Evaluate/Compare/Judge

Messick’s criteria ask three questions to be used for instructional design and evaluation:

• What are the Knowledge, Skills, and Abilities (KSAs) desired for learners?
• What actions/behaviors will reveal that those KSAs are present?
• What tasks will elicit those actions/behaviors?

To “meet learners where they are,” we could identify which Bloom’s level they are at and which higher Bloom’s level we are planning on taking them to. We would then use Messick’s KSAs to design, evaluate, and educate learners about where they are in the learning process.

For example, when teaching programming with Python, we may start from the understanding of a for loop (B2) and get to applying it to solve a specific problem (B3). However, if learners don’t know what a for loop is when they enter (B1), it might be better to constrain our short-term objectives to achieving understanding (B2). Setting our goals in terms of “growing a level” means our target outcomes might not match the level at which trainees need to operate once on their own. In their work, researchers typically operate at the highest levels of complexity, evaluating (B5) and creating (B6). The fact that researchers do habitually operate at these higher levels is helpful insofar as they know what it’s like to think in these ways. To help them be successful in meeting these post-workshop (real-world) goals, we should educate our learners about what they have achieved, but also about what the next steps of advancing in the Bloom’s hierarchy might look like. This not only gives them next steps to plan on, it also fosters metacognition – the process of thinking analytically about the process of learning, which is key to sustainability.

To “explicitly address motivation and self-efficacy,” metacognition is particularly key. Articulating expectations with regard to learning growth will also be helpful here, to help learners perceive their accomplishments within the workshop setting. However, to set the stage for sustainable learning, we should aim to educate learners about the road ahead of them and offer guidance about the kind of learning they should expect to be doing after they leave the workshop. Anticipating the steps that lie between them and the ability to actually apply their new skills to their research is key to both taking those steps and appropriately evaluating success as that learning continues.

While the Carpentries haven’t explicitly taught Messick, Bloom’s taxonomy has been present in the instructor curriculum up to its most recent iteration, and remains in spirit as we guide instructors through interpretation of our learning objectives. Its recent removal is owed to the fact that instructors are not typically the ones defining those learning objectives in the first place, so this is something we may wish to consider as we go about creating on-boarding procedures for curriculum designers and maintainers. While learning objectives are present throughout our curriculum and have been crafted with Bloom’s in mind, they are not necessarily specific in targeting a goal of “growing a level.” Furthermore, most of the education that we offer with regard to future learning occurs peripherally, e.g. as instructors model and describe their own learning process. There is clearly room for these ideas to grow within our community, and we will be keeping these suggestions firmly in mind as we move ahead.

We will also be keeping in touch with Dr. Tractenberg, as she will be joining us for our next meeting of the Virtual Assessment Network on March 23rd! Click here for your timezone. All are welcome to attend – please sign up via this Etherpad. Contact Kari Jordan (email) with questions or requests to join the Assessment Network list.

If you missed the webinar or want a refresher, check out the Etherpad, the annotated slides, the video recording, or the audio-only recording.