Revamping the Instructor Survey

This post originally appeared on the Software Carpentry website.

In September, Greg Wilson wrote a blog post on "building better teachers" that was inspired in part by an issue brought up by John Blischak on reporting time spent on lessons. In order for us capture this information, we decided to revamp our post-workshop instructor survey.

First Round of Data Collection

We began by asking all the instructors who participated in teaching this past summer to report time spent teaching various subjects (Shell, Python, Git, etc) and anything that worked or did not work during the workshop. A pull request provided the first pieces of data for instructors with regards to hours spent on each topic.

	Git	Python	Shell	SQL
N	8	5	8	1
min	1	4	1.5	3
median	2.25	4.5	3	3
mean	2.06	4.9	2.8	3
max	3.5	6	3.5	3

Other than Git, the times were fairly consistent with one another. I say "other than Git", because Damien Irving manages to teach it in one hour by having the participants type along command by command, but not have them complete any challenges. (There is a recording of him doing this for those who would like to see how this works.) Additionally, we only had one workshop that responded that covered all four topics; SQL is usually left out in place of more Python.

Revamped Survey

The individual questions for the survey are at the end of this post. However, there are a few questions for the other instructors to keep in mind while reviewing the survey:

What order should the questions be presented in? Should we ask all the questions pertaining to shell, then all the questions relating to Python, etc., (questions by lesson) or should we have a section for time, deviation from the repo material, etc and have all the topics under each section (lessons by question)?
Is there any other information you think would be useful to track? I have added questions about the Software Carpentry Windows installer and Ipython Notebook use/installation/problems— what else should be there?
Would you take this survey? Is it too long? Is the question format clumsy? Any feedback on the survey design and layout is greatly appreciated.
Are the reference examples for the Likert scale questions well spaced? The reason for the reference example is to provide some context for instructors so we can have some standardized set of responses.
How feasible is it to track individual topics in a lesson? John mentioned that time spent on an entire lesson (i.e., total time for python) is not that useful when instructors are trying to pace the lesson. For example, if there is only 30 minutes left for Python, is there enough time to teach the v5 testing material?

First Draft of Revamped Survey

The live survey can be found here The static post of the survey is replicated below:

First name (text box - required)
Last name (text box - required)
Workshop website (text box - required): this is the web url for the workshop. It provides a more standard way for us to pull the date and location.
Attendees:
- What fraction of the attendees were complete beginners (no coding/command line experience at all)
- What fraction of the attendees were novices (doing copy-and-paste coding without really understanding it)
- What fraction of the attendees were intermediate (already using the shell and writing multi-page programs)
- These questions are on a 5 point scale via radio buttons: None, 1/4, 1/2, 3/4, All
Did you use the SWC Installer? (radio button - required): Yes, No, I was not aware there was one, Other (text box)
Any comments about the SWC installer? (text box - optional)
What topics did you teach and how long did it take (in hours)? (matrix of radio buttons - required) On the Y-axis are the lessons (Shell, Git, Python, etc...) the X-axis has time in hours (1, 1.5, 2, ..., 7.5, NA)
If you taught 'other' please describe what you taught and how long it took (e.g., "SVN (3.5); SPSS (3); SAS(2.5)")
Material:
- How much of the SWC material did you use for Shell?
- How much of the SWC material did you use for Git?
- How much of the SWC material did you use for Python?
- How much of the SWC material did you use for R
- How much of the SWC material did you use for Matlab
- How much of the SWC material did you use for Mercurial?
- How much of the SWC material did you use for Make?
- there are on a 6-point likert scale 0 - 7
  1. 0 - I wrote all my material from scratch
  2. 1 - I used some SWC material as well as material I've written myself
  3. 2 - I used about 50/50 SWC material and my own material
  4. 3 - Minor tweaks to order and/or example problems
  5. 4 - I taught SWC material verbatim
  6. 5 - I taught SWC material verbatim + more topics
What other SWC material did you teach (text box - optional)
What Shell material was covered? (check boxes - select all that apply)
- files and directories
- creating things
- pipes and filters
- scripting
- finding things
- other (text box)
What Python material was covered (check boxes - select all that apply)
- variables and assignment
- functions
- loops
- conditionals
- NumPy
- Pandas
- defensive programming
- the command line
- other (text box)
What Git material was covered?
- recording history
- push and pull
- merging conflicts
- branching
- licensing and open source
- other (text box)
What SQL material was covered?
- selecting
- soring and removing duplicates
- filtering
- calculating new values
- nulls
- aggregation
- join
- updating and deleting
- using in programs
- other (text box)
Any other comments regarding the material and/or adherence to the material? (text box - optional)
Any problems with the Git installation? (text box - optional): this question is mainly targeted to Git issues on a mac
Any problems with the IPython notebook? (text box - optional): targeted to why the ipython notebook and/or kernel was restarted
Anything else? (text box - optional): for anything else that the survey missed and/or comments about the survey itself