Moving Forward with Assessment: Interviews

This post originally appeared on the Software Carpentry website.

We would like to quantify realistic impact after the dust settles from a bootcamp. In particular, we'd like to know how (or whether) Software Carpentry has improved scientists' workflow three months or more after the bootcamp, on the theory that if people are still using things then, they must have found them useful. Our most recent steps toward this goal include obtaining Internal Review Board for Human Subjects Research (IRB) approval through Michigan State University and beginning to recruit volunteers for interviewing at the three month mark.

IRB approval is important because the process helps us ensure safety, transparency, and accountability for research projects involving people. Even the best-intentioned researcher may unintentionally put people at risk; the review helps ensure that no one falls through the safety gaps.

This week we began recruiting attendees for interviews. Unfortunately, IRB requirements are very particular about how participants are contacted, so we can't dress up the initial poll for participation beyond what is written in the usual IRB format consent message included below. So far, this means only very dedicated (or very bored?) bootcamp alumni will understand what we are asking, let alone consider participating in the study; our current response rate is about 4%, and while we'd obviously like to increase this, we're not sure how. (If any readers have suggestions, please let us know.)

Here are some of the things we've learned in working with the IRB:

Say Hello

Unless you have the super power to intuit the rules for all required forms, it is helpful to pay an office visit to the people who work with those IRB forms on a daily basis. With questions and forms in hand they will help you save a lot of time. In my case I used the telephone quite a bit, but face to face help would have been even more efficient.

Microsoft Word

This one caught me off guard. Depending on the state of technological advancement and workflows of your local IRB office, you may need to find a computer with Microsoft Word in order to complete required forms in a manner most agreeable for submission. My own experience in this process showed me that LibreOffice is not yet a viable substitute in some situations.

Plan for Change

Changing any part of your study—even just the consent message—is time-consuming and requires formal validation. It's better to write in possibilities now rather than have to officially alter to achieve those later.

The official consent email we're using is listed below.

Research project title: Software Carpentry Bootcamp Impact

Principal Investigator: [redacted]

1. Explanation of the Research and What You Will Do:

You are being asked to participate in a research study about the Softawre Carpentry Botocamp you recently attended. Researchers at Michigan State University are interested in understanding the magnitude of benefit participants gained from their experience at a Software Carpentry Bootcamp, as well as specific areas the Bootcamps may be improved. Researchers would like to analyze skills and practices common to computation in science to draw conclusions about Software Carpentry Bootcamp effectiveness and improvement. These results would be included in research reports, presentations, or publications and the anonymous data may be made publicly available. Identifiable information such as name or professional position will not be recorded. You must be at least 18 years old to participate in this research study. If you are interested, then we would like to interview subjects by phone or other audio means and record your responses by transcription. The interview should take about 20 minutes. You may also be asked for a second interview 6 months later to see if and how your workflows have changed.

2. Your Rights to Participate, Say No, or Withdraw:

Participation in this research project is completely voluntary. You have the right to say no. You may change your mind at any time and withdraw. You may choose to not answer specific questions or to stop participating at any time. Whether you choose to participate or not will have no effect on your grade or evaluation at the bootcamp or any future bootcamps.

3. Costs and Compensation for Being in the Study:

There are no costs or compensation offered to participate.

4. Contact Information for Questions and Concerns:

If you have questions or would like further information about this study, then please contact Jorden Schossau at jory@msu.edu or The Michigan State University's Human Research Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail at 408 W. Circle Drive, 207 Olds Hall, MSU, East Lansing, MI 48824.

If you consent to participating in an interview about your computational use patterns and experience at a previous Software Carpentry Bootcamp then please reply to this email and we'll work out a time and method for interview that is easiest for you. By emailing a response, you voluntarily agree to participate in this study.