How Teaching Knowledge Is Transferred

This post originally appeared on the Software Carpentry website.

I hesitate to say so, but I believe that we pedagogues tend to exaggerate greatly the amount of change in educational practice that results from reading what other people say should be done...

— Stephen Corey, 1951

Being back in Edinburgh thirty years on has occasioned much reflection upon lessons learned. This has occasioned a re-reading of some papers by Prof. Sally Fincher, whose research group at the University of Kent studies the teaching and learning of computer science. In particular, I have been looking at what they have discovered about how educators share teaching practices. I hope these excerpts and reflections are of interest. (Note: section titles link to papers.)

Warren's Question

This paper is a close look at a request for help that one instructor posted to a forum and what happened next. The conclusion is that top-down dissemination rarely if ever works. Instead, instructors borrow bits and pieces that they happen to stumble upon when and as they need them:

It is not a common practice among tertiary educators to observe teaching in someone else's institution. Although peer observation is becoming more common within departments (sometimes within institutions) it is essentially unheard of between institutions. And when the motivations for peer observation are examined, this is not surprising. They are, in general, linked to the quality assurance and staff development of the observed teacher. From that perspective there is simply no point in observing practice to ensure the quality of teaching in some other institution, nor any incentive to develop their staff.

Unexpectedly, whether because observers were observing outside their home institutions, or whether because they were highly sensitised to the curriculum and material being delivered, observers were especially struck by aspects of context that are normally invisible. These observations included the physical setting of the university, the material and technological objects within the classroom, the student interactions with one another. The shared disciplinary background meant that, for the observer, the observation began not when entering the classroom, but when leaving their own.

The normal mechanisms employed for transfer of ideas in teaching and learning are in the mode of disseminator push: that is that something is identified (a teaching method, a "best practice", a theory) and it is packaged and promoted to interested parties by a staff developer or a researcher, via mechanisms such as papers, books, websites and workshops. This is the trajectory that is often assumed for educational research—that "dissemination" to practitioners occurs simply by virtue of publication in a research venue. However, evidence from empirical studies suggests that transfer of practice and knowledge to practitioners is rarely occasioned by these research-to-practice, top-down methods.

Teachers change their practices, adopting (transferring) ideas and materials from direct, personal contact with other practitioners as and when they need—often in very small, partial, pieces ("piecemeal accretion") or by virtue of having experienced it in another institution and, with a change of employment, importing it to a new context ("charismatic embedding"). Thus these transfers are achieved directly from practitioner to practitioner, from one specific setting to another, mediated neither by theory, researcher, or staff developer.

Survey results...support the conjecture that knowledge transfer rarely happens top-down, from researcher to practitioner. When asked "What published material do you read with regard to your teaching?", seven participants mention reading technical publications related to disciplinary knowledge, five mention textbooks, five mention CS Ed practitioner conference proceedings... Only one... (a CS Ed researcher) ...mentioned reading the CS Education research literature; and none mentioned reading research in the learning sciences, in the behavioural or social sciences, or disciplinary education research in cognate disciplines...

The inefficiency of "pull transfer" is its most obvious drawback, but in addition, its ad hoc nature makes even its practitioners downplay it, which in turn means that people aren't given full credit for their ideas:

[C]'s acknowledgement of this cycle of adoption and adaptation comes with the slightly shamefaced 'there's nothing like plagiarism, eh?' This...symbolises one of the fundamental features that differentiate the activities of teaching and research. 'Plagiarism' is about public attribution of the source of ideas, a basic requirement of research-based activity but one that is more-or-less unknown in teaching. It suggests that it is not important—to [C], to the colleagues within his institution, or to his disciplinary peers—to acknowledge sources of teaching knowledge in the same way as in research. This would also imply that there is no incentive or reward for giving such acknowledgement, and that there are no evolved norms that require it. Part of the reason for this is that teaching practice is often ephemeral, enacted but not documented. And those parts of teaching that are documented are rarely referenced with sources as would be ordinary in the documentation of research. As a result, provenance is easily lost in teaching practice.

This loss of provenance in teaching, this rootlessness and reinvention of practice, paradoxically places more emphasis on practitioner-to-practitioner transfer, unmediated by documentary evidence.

Stories of Change: How Educators Change Their Practice

This paper's abstract is a better summary than I could possibly write:

Innovative tools and teaching practices often fail to be adopted by educators in the field, despite evidence of their effectiveness. Naïve models of educational change assume this lack of adoption arises from failure to properly disseminate promising work, but evidence suggests that dissemination via publication is simply not effective. Instead of studying the adoption or rejection of a particular intervention, this paper turns the problem around. We asked educators to describe changes they had made to their teaching practice and analyzed the resulting stories to learn more about: the kinds of changes being made, their motivations for changing their practice, and the means by which they learned of pedagogical innovations. Of the 99 change stories analyzed, only three demonstrate an active search for new practices or materials on the part of teachers, and published materials were consulted in just eight of the stories. Most of the changes occurred locally, without input from outside sources, or involved only personal interaction with other educators.

A few quotes from the paper:

...there is little evidence that enhanced efforts at dissemination are working. A report to the National Research Council in 2008 found that "NSF- and association-funded reforms at the classroom level, however well intentioned, have not led to the hoped for magnitude of change in student learning, retention in the major, and the like in spite of empirical evidence of effectiveness". Failure to adopt effective practices is not limited to computing or education. A meta-analysis of 743 reports, books, and articles in 2005 found that dissemination alone was not an effective implementation method in human services, education, health, business, or manufacturing.

...there were notable differences in response dependent on how long the contributor had been teaching, and on the time a contributor had been at their current institution. In both cases the longer the time, the more the response moved from the change affecting individual practice only towards affecting other colleagues and then programmatic change.

The most populated catalyst category [i.e., explanation of reason for change] was of educators initiating change in response to students: in response to something they did, or something they said, or to a close observation of their attitudes and achievements.

...a least populated category was 'external imposition'...

In just over half of the stories...the instructor apparently formulated the details of their pedagogical change on their own, without consulting peers or other resources... This...is perhaps not surprising given that 'change in response to students' was...the most common catalyst category.

...[some] stories described changes resulting from chance encounters with educators who shared or demonstrated details of their teaching practice. The authors of these stories had not intended to change their practice until they were exposed to a new approach.

A larger group of change stories...involved primed serendipity, where the author encountered unanticipated information but while putting themselves in situations where they could reasonably expect to learn something about teaching practices (e.g. attending a conference or workshop, or browsing a journal issue)...Over 90% of the stories described changes that were either created without drawing on outside sources, or were informed by personal interactions with other educators.

Repositories of Teaching Material and Communities of Use: Nifty Assignments and the Greenroom

The key challenges in building a lesson repository are:

Curation: "Although it is easy to start a collection, it is often more difficult to maintain it."

Software Carpentry instructors must do this once to qualify. We have maintainers, but our contribution curve definitely has a long-tail distribution.

Content: "The focus of content, aimed broadly at a discipline or narrowly at a specific area, seems to have a significant impact on the severity of the challenges in all other aspects. The broader the focus, the harder other aspects become."

We are one course with only a handful of modules, so our focus is pretty narrow by the paper's measure.

Contribution: Quoting a personal communication from the author of another survey, "In the surveys we have done, we find a contradictory set of opinions: professors want resources to use, but they are not willing to share back the modifications they make to these resources. And the effort it takes to get a resource published is often ignored, so most people just don't do anything to publish things on the web in a way that it is truly useful."

This is not as much a problem for us: we're mostly curating.

Community: "In other community driven endeavours, such as open source software communities, reward is often received through peer recognition from fellow community members. In many online teaching repositories, community recognition is low or absent."

Our community is defined as "people who use our lessons", which helps with recognition. That said, we should probably add all the contributors' names to each lesson's home page...

Catalog: "Once materials are gathered together, there has to be a way for users to locate the item that is most useful to them for their current need, or allow them to search for it easily."

We only have a small number of modules, so it's easy to find what you want—once you've found the site, of course.

Control: "All repositories grapple with issues of control: who can contribute, who admits the contribution, and how the quality of contributed material is assured."

We borrow rules and conventions from open source, which most of our contributors accept as given, and don't try to control who uses materials.

As a side note, it's interesting to contrast Software Carpentry with the Nifty Assignments repository: new material is only added to Nifty Assignments once a year, and entry is competitive (i.e., it's not enough to be good: you have to be better than other assignments put forward in that year). Neither choice seems likely to build a culture of contribution or increase the average would-be contributor's confidence in her/his ability.

Later, quoting earlier work, the authors say that "borrowing provokes invention", so that "using a software library" is the wrong model for using a lesson, because (almost) everyone adapts to a different audience, a different precursor, or a different personal style of teaching. However, no alternative metaphor given: if sharing lessons isn't like using a library, what is it like?

Later still, "There was little distaste—indeed, some enthusiasm—for the re-use of pedagogic materials." That's hardly a ringing endorsement...

Finally, the paper teases out some implications:

1. ...the users of the Nifty repository do not appear to want library-type resources. They do not feel the need for catalogues, they are content with the current methods of searching, but would like the metadata associated with each resources extended...

2. ...they are not looking for a one-stop-shop with a comprehensive set of resources and the heavyweight 'authority' of editorial constraints... Nor are they especially interested in the comprehensiveness of the content of the resources as they recognise that they will have to adapt anything to local context.

3. They would seem to value curators above librarians.

Conclusions

None yet, but there's a lot here to think about. Please add your thoughts as comments.