An Invitation to Teaching Reproducible Research: Lessons from a Symposium

ABSTRACT

This article synthesizes ideas that emerged over the course of a 10-week symposium titled “Teaching Reproducible Research: Educational Outcomes” https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium that took place in the spring of 2021. The speakers included one linguist, three political scientists, seven psychologists, and three statisticians; about half of them were based in the United States and about half in the United Kingdom. The symposium focused on a particular form of reproducibility—namely computational reproducibility—and the paper begins with an exposition of what computational reproducibility is and how it can be achieved. Drawing on talks by the speakers and comments from participants, the paper then enumerates several reasons for which learning reproducible research methods enhance the education of college and university students; the benefits have partly to do with developing computational skills that prepare students for future education and employment, but they also have to do with their intellectual development more broadly. The article also distills insights from the symposium about practical strategies instructors can adopt to integrate reproducibility into their teaching, as well as to promote the practice among colleagues and throughout departmental curricula. The conceptual framework about the meaning and purposes of teaching reproducibility, and the practical guidance about how to get started, add up to an invitation to instructors to explore the potential for introducing reproducibility in their classes and research supervision.

KEYWORDS:

• Computational reproducibility
• Curriculum
• Educational outcomes
• Intellectual development
• Reproducibility

Notes

1 Before the NASEM (2019) report, the term reproducibility was used to mean several different things, and was sometimes conflated with the term replicability. For the history of the use of reproducibility, replicability, and related terms, see Stodden et al. (2013), Liberman (2015), Freese and Peterson (2017), Plesser (2018), Patil, Peng, and Leek (2019), and Vilhuber (2020).

2 This notion of reproducibility was proposed in an early paper by King (1995). Note, however, that King used the term replicability rather than reproducibility—an example of the lack of standardization in terminology cited in footnote 1.

3 A number of professional associations have formulated standards for reproduction documentation similar to the TIER Protocol. Notable examples of comprehensive, practical guides to reproduction documentation include the Data and Code Availability Policy https://www.aeaweb.org/journals/data/data-code-policy of the American Economic Association, and the DIME Analytics Data Handbook of the World Bank’s Development Impact Evaluation group (Bjarkefur et al. 2021).

Additional information

Funding

This work was supported by the Alfred P. Sloan Foundation.