An Invitation to Teaching Reproducible Research: Lessons from a Symposium

References

• Baumer, B., Çetinkaya-Rundel, M., Bray, A., Loi, L., and Horton, N. J. (2014), “R Markdown: Integrating a Reproducible Analysis Tool into Introductory Statistics,” Technology Innovations in Statistics Education, 8. DOI: 10.5070/T581020118..
   Google Scholar
• Bjarkefur, K., Cardoso de Andrade, L., Daniels, B., and Jones, M. R. (2021), Development Research in Practice: The DIME Analytics Data Handbook. Washington, DC: World Bank.
   Google Scholar
• Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., and Olds, J. L. (2015), “Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science: Report of the Subcommittee on Replicability in Science”, Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences. Available at http://ora.emory.edu/research-compliance/oric/documents1/8%20%20Social,%20Behavioral,%20and%20Economic%20Sciences%20Perspectives%20on%20Robust%20and%20Reliable%20Science.pdf.
   Google Scholar
• Broman, K. W., and Woo, K. H. (2018), “Data Organization in Spreadsheets,” The American Statistician, 72, 2–10.
   Web of Science ®Google Scholar
• Bukach, C. M., Bukach, N., Reed, C. L., and Couperus, J. W. (2021), “Open Science as a Path to Education of New Psychophysiologists,” International Journal of Psychophysiology, 165, 76–83.
   PubMed Web of Science ®Google Scholar
• Bussberg, N. (2021), “Incorporating an Accessible Reproducibility Workflow into Entry-Level Courses [videorecording],” TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/incorporating-accessible-reproducibility-workflow-entry-level-courses/.
   Google Scholar
• Button, K. S., Chambers, C. D., Lawrence, N., and Munafo, M. R. (2020), “Grassroots Training for Reproducible Science: A Consortium-Based Approach to the Empirical Dissertation,” Psychology Learning & Teaching, 19, 77–90.
   Google Scholar
• Campbell, D. E. (2006), “What is Education’s Impact on Civic and Social Engagement?” in Measuring the Effects of Education on Health and Civic Engagement: Proceedings of the Copenhagen Symposium, eds. R. Desjardins and T. Schuller, Paris: OECD.
   Google Scholar
• Christensen, G., and Miguel, E. (2018), “Transparency, Reproducibility, and the Credibility of Economics Research,” Journal of Economic Literature, 56, 920–980.
   Web of Science ®Google Scholar
• Converse, P. E. (1972), “Change in the American Electorate,” in The Human Meaning of Social Change, eds. A. Campbell and P. E. Converse, New York: Russell Sage Foundation.
   Google Scholar
• Delbanco, A. (2012), College: What It Was, Is, and Should Be, Princeton, NJ: Princeton University Press.
   Google Scholar
• Dewald, W. G., Thursby, J. G., and Anderson, R. G. (1986), “Replication in Empirical Economics: The Journal of Money, Credit and Banking Project,” American Economic Review, 76, 587–603.
   Web of Science ®Google Scholar
• Framework for Open and Reproducible Research Training (FORRT). (2022, March 8), “Toward Social Justice in Academia: Diversity, Equity and Inclusion,” [website]. Available at https://forrt.org/dei/.
   Google Scholar
• Freese, J., and Peterson, D. (2017), “Replication in Social Science,” Annual Review of Sociology, 43, 147–165.
   Web of Science ®Google Scholar
• Hayes-Harb, R. (2021), “Reproducibility Education in an Undergraduate Capstone Course [videorecording],” TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/reproducibility-education-undergraduate-capstone-course/.
   Google Scholar
• Hoces de la Guardia, F. (2021), “How to Teach Reproducibility in the Classroom (BITSS) [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/how-teach-reproducibility-classroom-bitts/.
   Google Scholar
• Holdren, J. F. (2013), “Memorandum for the Heads of Executive Departments and Agencies: Increasing Access to the Results of Federally Funded Scientific Research,” White House Office of Science and Technology Policy.
   Google Scholar
• Hoeffler, J. (2017a), “Replication and Economics Journals Policies,” American Economic Review Papers and Proceedings, 107, 52–55.
   Web of Science ®Google Scholar
• Hoeffler, J. (2017b). “ReplicationWiki—Improving Transparency in the Social Sciences,” D-Lib Magazine, 23, 3/4 (March/April). DOI: 10.1045/march2017-hoeffler..
   Google Scholar
• Horton, N. J. (2021), “Transparent and Reproducible Analysis as a Key Component of Data Acumen [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/transparent-and-reproducible-analysis-key-component-data-acumen/.
   Google Scholar
• Ingram, M. C. (2021), “Teaching Transparency: Principles and Practical Considerations with Illustrations in Stata and R,” in Teaching Research Methods in Political Science, ed. J. L. Bernstein, pp. 114–130, Northampton: Edward Elgar Publishing.
   Google Scholar
• Janz, N. (2016), “Bringing the Gold Standard into the Classroom: Replication in University Teaching,” International Studies Perspectives, 17, 392–407.
   Web of Science ®Google Scholar
• Janz, N. (2021), “Teaching Replication [videorecording],” TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/keynote-tba/.
   Google Scholar
• Janz, N., and Freese, J. (2021), “Replicate Others as You Would Like to Be Replicated Yourself,” PS: Political Science & Politics, 54, 305–308.
   Web of Science ®Google Scholar
• King, G. (1995), “Replication, Replication,” PS: Political Science & Politics, 28, 444–452.
   Web of Science ®Google Scholar
• Liberman, M. (2015, October 31). “Replicability vs. Reproducibility – Or Is it the Other Way Around?” Language Log [blog]. Available at https://languagelog.ldc.upenn.edu/nll/?p=21956#:˜:text=But%20just%20because%20a%20study,arrive%20at%20the%20same%20conclusions.
   Google Scholar
• Maniadis, Z., and Tufano, F. (2017), “The Research Reproducibility Crisis and Economics of Science,” The Economic Journal, 127, F200–F208.
   Web of Science ®Google Scholar
• McAleer, P. (2021), “Creating a Curriculum Centered on Reproducible Research for the Psychologists of the Future [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/creating-curriculum-centered-reproducible-research-psychologists-future/.
   Google Scholar
• McNamara, A. (2021), “Consistency Is Key: A Case Study in R Syntaxes [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/tba/.
   Google Scholar
• National Academies of Sciences, Engineering, and Medicine (NASEM). (2019), Reproducibility and Replicability in Science [online]. Washington, DC: The National Academies Press. DOI: 10.17226/25303.
   Google Scholar
• National Institutes of Health (NIH). (2020, October 29). “Final NIH Policy for Data Management and Sharing,” Notice Number: NOT-OD-21-013.
   Google Scholar
• Nüst, D., Eddelbuettel, D., Bennett, D., Cannoodt, R., Clark, D., Daróczi, G., Edmondson, M., Fay, C., Hughes, E., Kjeldgaard, L., Lopp, S., Marwick, B., Nolis, H., Nolis, J., Ooi, H., Ram, K., Ross, N., Shepherd, L., Sólymos, P., Swetnam, T. L., Turaga, N., Petegem, C. V., Williams, J., Willis, C., and Xiao, N. (2020), “The Rockerverse: Packages and Applications for Containerisation with R,” The R Journal, 12, 437–461.
   Google Scholar
• Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., and Evans, B. D. (2020), “Ten Simple Rules for Writing Dockerfiles for Reproducible Data Science,” PloS Computational Biology, 16, e1008316.
   PubMed Web of Science ®Google Scholar
• O’Hara, M. (2016), “Reproducibility as a Pedagogical Strategy [paper presentation],” AEA Conference on Teaching and Research in Economic Education (CTREE), Atlanta, GA.
   Google Scholar
• Patil, R., Peng, R., and Leek, J. (2019), “A Visual Tool for Defining Reproducibility and Replicability,” Nature Human Behaviour, 3, 650–652.
   PubMed Web of Science ®Google Scholar
• Parsons, S., and Azevedo, F. (2021), “Building a Community from Open Scholarship Pedagogy with a Framework for Open and Reproducible Research Training [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/building-community-open-scholarship-pedagogy-framework-open-and-reproducible-research-training-forrt/.
   Google Scholar
• Peng, R. D. (2011), “Reproducible Research in Computational Science,” Science (New York, NY), 334, 1226–1227.
   PubMed Web of Science ®Google Scholar
• Plesser, H. (2018), “Reproducibility vs. Replicability: A Brief History of a Confused Terminology,” Frontiers in Neuroinformatics, 11, 1–4.
   Web of Science ®Google Scholar
• Steltenpohl, C. N., Daniels, K. M., and Anderson, A. J. (2020), “Giving Community Psychology Away: A Case for Open Access Publishing,” Global Journal of Community Psychology Practice, 10, 1–14.
   Google Scholar
• Stodden, V., Bailey, D. H., Borwein, J., LeVeque, R. J., Rider, W., and Stein, W. (2013, February 16). “Setting the Default to Reproducible: Reproducibility in Computational and Experimental Mathematics.” Available at http://stodden.net/icerm_report.pdf.
   Google Scholar
• Stodden, V., Leisch, F., and Peng, R. D., eds. (2014), Implementing Reproducible Research, Boca Raton, FL: CRC Press/Taylor & Francis Group.
   Google Scholar
• Sullivan, J. (2021), “A Practical Approach to Teaching Reproducibility, and Improving Your Own Research at the Same Time [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/practical-approaches-ensuring-undergraduates-master-reproducibility/.
   Google Scholar
• Torney-Purta, J., Lehmann, R., Oswald, H., and Schulz, W. (2001), Citizenship and Education in Twenty-Eight Countries: Civic Knowledge and Engagement at Age Fourteen, Amsterdam: International Association for the Evaluation of Educational Achievement.
   Google Scholar
• Towse, J. N., Davies, R., James, R., and Ball, E. (2021), “LUSTRE: An Online Tool for Training Students in Data Management and Data Sharing [videorecording],” in TIER Spring Symposium: Instruction in Reproducible Research [online]. Available at https://www.projecttier.org/fellowships-and-workshops/2021-spring-symposium/lustre-online-tool-training-students-data-management-and-data-sharing/.
   Google Scholar
• Towse, J. N., Ellis, D. A., and Towse, A. S. (2020), “Opening Pandora’s Box: Peeking inside Psychology’s Data Sharing Practices, and Seven Recommendations for Change,” Behavior Research Methods, 53, 1455–1468.
   PubMed Web of Science ®Google Scholar
• Vilhuber, L. (2020), “Reproducibility and Replicability in Economics,” Harvard Data Science Review, 2. DOI: 10.1162/99608f92.4f6b9e67..
   Google Scholar
• Xie, Y., Allaire, J. J., and Groleman, G. (2019), R Markdown: The Definitive Guide, Boca Raton, FL: CRC Press/Taylor & Francis Group.
   Google Scholar