Teaching Reproducible Methods in Economics at Liberal Arts Colleges: A Survey

Abstract

Economics has become increasingly empirical and, alongside this shift, has come more demand for improved transparency and reproducibility in empirical economic research. In this article, we distribute a survey to almost 1500 economics faculty from the top 161 liberal arts colleges with an economics major (according to U.S. News & World Report) in the United States to determine the prevalence of teaching reproducible methods in undergraduate economics, summarize the most-common methods of instruction, and determine the intended student learning objectives. We find that of the economics faculty at liberal arts colleges who teach these reproducible methods, most do so in advanced upper-level (42%) and basic econometrics (31%) courses. Those faculty report teaching reproducibility using the following methods: transparent coding (85%), organizational skills (78%), and producing replication documentation (47%) through individual research projects (82%), homework assignments (55%), and/or workshops (33%). We conclude with some qualitative text analysis to shed light on the intended learning objectives and find that research skills (59%) and the importance of reproducibility (37%) are the most common reasons cited for teaching these methods.

KEYWORDS:

• Reproducible methods
• Reproducibility
• Teaching
• Undergraduate economics education

1 Introduction

The lack of successful replicability and reproducibility in empirical research across the sciences and social sciences has received significant attention (Dewald, Thursby, and Anderson 1986; King 1995; Hamermesh 2007; Vilhuber 2021; Vilhuber et al. 2022).¹ Approximately 90% of respondents to a recent survey conducted by Nature believe that there is a “slight” or “significant” replicability crisis (Baker 2016). In economics, researchers have found that less than half to two-thirds of papers studied are replicable (Camerer et al. 2016; Chang and Li 2017). The American Statistical Association (ASA) suggests that authors of research involving statistical analysis have an ethical responsibility to “promote sharing of data and methods” and “make documentation suitable for replicate analyses” available (ASA 2016). Thus, there have been increasing calls for more transparency by researchers (Bell and Miller 2013; Stodden et al. 2016; Coffman, Niederle, and Wilson 2017; Höffler 2017) and professional organizations including the National Academies of Sciences, Engineering, and Medicine and the National Science Foundation (Bollen et al. 2015).

Failure of research to be reproducible and/or replicable comes from a variety of sources including falsified data, data mining, search for statistical significance (“p-hacking”), validity and repeatability of experiments, incomplete documentation, and unintentional errors in data entry, cleaning, coding, or analysis. Reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials and procedures used by the original investigator. Reproducibility requires documentation that allows an independent researcher to reproduce every step of the data management and analysis process and replicate the results presented in the study (Ball and Medeiros 2012). Replicability refers to the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected (Bollen et al. 2015).²

In part, replicability in economics has become increasingly important as the discipline trends toward empirical research (Hamermesh 2013; Angrist et al. 2017). This pattern is also reflected in the undergraduate economics curriculum, with more courses in econometrics being required (6% in 1980 and 54% in 2020) and more degrees classified as “Econometrics and Quantitative Economics” (1% of economics degrees conferred in 2012 and 22% in 2019) (Siegfried and Wilkinson 1982; Marshall, Underwood, and Hyde Forthcoming; Marshall and Underwood 2020, 2022). The calls for increased transparency and trends in the discipline have led economic journals and organizations to promote policies requiring data and/or corresponding documentation for published work. (American Economic Association 2019, 2020; Vilhuber et al. 2022).

Despite the growing emphasis on replication documentation in empirical economic research, far less work has given primacy to the prevalence and assessment of reproducibility in teaching economics (Vilhuber 2020). There is little research on the degree to which transparency in research is taught in the economics classroom. In statistics and data science, where reproducibility is arguably more intrinsic, more is known, and more teaching resources are available (Çetinkaya-Rundel and Ellison 2021; Dogucu and Çetinkaya-Rundel 2022; Horton et al. 2022; Ostblom and Timbers 2022). In the social sciences, including economics, a small, but growing list of resources are available for teaching reproducibility and establishing pedagogical best practices (Orozco et al. 2020; Ball et al. 2022; Rethlefsen et al. 2022). Assessing how reproducible teaching methods are incorporated into undergraduate economics programs has been challenging, as has determining whether teaching reproducibility is beneficial to students’ understanding of data analysis and econometrics (Vilhuber 2020). Therefore, we examine the prevalence of reproducible methods in undergraduate liberal arts economics programs, as well as what reproducible methods are being taught, how instructors are teaching reproducible methods, and why reproducible methods are used.

2 Data and Methods

We survey nearly 1500 economics faculty from the top 161 liberal arts colleges (LACs) in the United States that offer an economics degree, as ranked and classified by the U.S. News and World Reports (USNWR) in June 2021.³ Our goal is to understand the prevalence of teaching reproducibility in undergraduate economics. We survey only LACs to manage the scope of the project and because economics professors at LACs may be more able to undertake new teaching methods with smaller class sizes, relative to professors at larger institutions. This is consistent with research showing that larger class sizes were found to be highly associated with specific pedagogical lecture styles and grading schema (Allgood, Walstad, and Siegfried 2015). However, large Ph.D.-granting universities are responsible for 75% of undergraduate economics degrees conferred every year (Marshall, Underwood, and Hyde Forthcoming), so this analysis is intended to be descriptive and should be viewed as a baseline for the prevalence of, and motivations for, teaching reproducible methods in economics. Participants were sent a link in the summer and fall of 2021 to an online survey asking whether they teach reproducible research methods in their courses, and if so, which methods of instruction are used and what are the intended learning objectives.⁴ The survey was distributed to 1474 potential respondents. 253 respondents agreed to participate and completed some or all the survey for a response rate of 17%. A copy of the survey instrument, copies of the distribution emails, and the data and code needed to reproduce our results are included in the supplementary material.

3 Results

Liberal arts colleges are often discussed in the context of their rankings and the rank of 50 is a common ranking threshold which, as others have suggested, may be particularly salient because it represents the last ranking on the first page in the printed version of the USNWR (Meyer, Hanson, and Hickman 2017). To that end, we organize many of the results according to this ranking threshold. We received responses from faculty at 98 of the top 161 LACs. Faculty from the most highly ranked LACs were more likely to respond to the survey. We received at least one response from 47 of the top 50 institutions and 51 responses from the remaining 111 institutions outside the top 50. Thus, 72% of respondents are from the top 50 and 44% are from the top 25 LACs. Thus, highly-ranked LACs may be over-represented in the sample.

This suggests the first of two likely limitations of the sample. First, more highly ranked institutions have larger economics faculty, so faculty from those institutions may be over-represented in the sample, as noted above. Based on our survey distribution list derived from faculty listings on institutional websites, institutions in the top 50 have around 17 economics faculty on average, while those outside the top 50 have about 9 economics faculty. Thus, as expected, we have 3.8 responses per institution in the top 50 and only 1.5 responses per institution outside the top 50. However, faculty response rates among those institutions with at least one response are identical inside and outside the top 50: 25% of faculty responded, on average. Second, faculty teaching these reproducible methods may be more likely to respond to a survey asking about it. This may lead to upwardly biased estimates of prevalence. Furthermore, if faculty at the top 50 LACs are more likely to teach these methods due to either higher incentives for quality teaching or lower teaching loads, then these limitations may be compounding and further bias our sample toward higher prevalence of teaching these methods.

We find that 51% of respondents (129 of 253 respondents) have taught some form of reproducible methods, as defined in Box 1, in at least one course over their past three years of teaching. This varies only slightly by institution rank, with 53% of respondents from the top 50 LACs teaching these methods and 45% outside the top 50. However, given the limitations of the sample noted above and a response rate of 17%, this is likely an overestimation of the prevalence of teaching reproducibility in economics at LACs. Only around 9% (129 of 1474 possible survey participants) of those that were sent the survey noted that they teach reproducible methods, so actual prevalence at LACs is likely between 9% and 51%.

Of those faculty teaching these methods, we asked in which course(s) they teach these methods, which methods they teach, and which teaching tools they use to integrate these methods into their course(s). Faculty teaching these methods, reported doing so in 167 total courses, for an average of 1.3 courses per faculty respondent; however, most (74%) teach reproducible methods in only one course.

Box 1. What are reproducible methods?

Reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator. Replicability refers to the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected (Vilhuber 2020).

Despite teaching these methods in 167 total courses, respondents only included complete course information, including course type and title, for 137 courses. Table 1 shows the most common courses and how this varies by institution rank. In the full sample, these methods are mostly taught in courses other than econometrics and statistics. Courses in the “other” category primarily include advanced electives (many with econometrics prerequisites), research methods courses (including honors theses), and senior seminar/capstone courses.⁵ Faculty at the top 50 LACs are most likely to teach these methods in these other courses, while faculty outside the top 50 are most likely to teach these methods in econometrics.

Table 1 Courses.

Course	Total	Top 50 LACs	Other LACs
	n = 137	n = 104	n = 33
other	41.6%	45.2%	30.3%
basic econometrics course	31.4%	27.9%	42.4%
introductory statistics	13.9%	16.3%	6.1%
advanced statistics or econometrics	13.1%	10.6%	21.2%

NOTE: other courses (n = 57) include titles listed in Appendix B, Table B1.

Table 2 shows which methods are most prevalent, how often faculty teach more than one method, and how this varies by institution rank. Most respondents teach coding for reproducibility (85%) and organizational skills (78%) while teaching how to produce replication documentation is also quite common (47%). This does not vary much by ranking, except for organizational skills which are more commonly taught at LACs outside the top 50 (but are very common at all institutions) and “other” methods which are more common among the top 50 (pre-analysis plans and pre-registration were the mostly commonly cited method among those few responses).⁶ The vast majority of respondents (82%) teach more than two of the listed methods in their courses and 14% report teaching four or more.

Table 2 Methods.

Method	Total	Top 50 LACs	Other LACs
	n = 136	n = 103	n = 33
Coding for reproducibility	84.6%	83.5%	87.9%
Organizational skills	77.9%	73.8%	90.9%
Producing replication documentation	47.1%	48.5%	42.4%
Using dynamic documents	18.4%	15.5%	27.3%
Accessibility and sharing	10.3%	8.7%	15.2%
Other	6.6%	8.7%	0.0%
One method only	18.4%	20.4%	12.1%
Two methods	35.3%	33.0%	42.4%
Three methods	32.4%	35.0%	24.2%
Four or more methods	14.0%	11.7%	21.2%

The methods used are consistent across course types but there are some notable differences, as shown in Table 3. While coding for reproducibility remains the most common method taught, this varies from 79% in advanced econometrics up to 94% in introductory statistics. More notably, teaching organizational skills varies from only two-thirds of basic econometrics courses to 94% of introductory statistics courses, likely reflecting the vertical nature of these skills. Students are taught these skills at lower levels of the curriculum and likely are assumed as prior knowledge at upper-levels.

Table 3 Methods by course type.

Method	Total	Basic econometrics	Advanced econometrics	Introductory statistics	Other
	n = 136	n = 43	n = 18	n = 18	n = 57
Coding for reproducibility	84.6%	81.4%	77.8%	94.4%	86.0%
Organizational skills	77.9%	81.4%	66.7%	94.4%	73.7%
Producing replication documentation	47.1%	53.5%	55.6%	44.4%	40.4%
Using dynamic documents	18.4%	11.6%	33.3%	27.8%	15.8%
Accessibility and sharing	10.3%	7.0%	16.7%	5.6%	12.3%
Other	6.6%	4.7%	0.0%	0.0%	12.3%
Coding for reproducibility and organizational skills	70.6%	69.8%	61.1%	88.9%	68.4%

Similarly, more advanced research skills, such as using dynamic documents and facilitating accessibility and sharing (GitHub, Open Science Framework, Dataverse, etc.) are more prevalent in more advanced courses. The clustering of the two most common methods, coding for reproducibility and organizational skills, is also consistent with the vertical nature of these skills. Across all courses, these two methods are taught together 70% of the time, likely due to their complementarity. However, these are taught together more often in introductory statistics (89%) compared to advanced econometrics (60%) courses.

Faculty primarily teach these methods through student projects or research papers (82%) but homework exercises (55%), workshops or labs (53%), and group work (45%) are also common methods of instruction. Table 4 shows how these instructional methods vary by course type. Student projects or research papers are most common in both basic econometrics (79%) and other (88%) courses, while workshops or labs are most common in advanced econometrics (78%) and introductory statistics (72%) courses. These differences may reflect differences in the expected student learning outcomes.

Table 4 Teaching tools by course type.

Method	Total	Basic econometrics	Advanced econometrics	Introductory statistics	Other
	n = 136	n = 43	n = 18	n = 18	n = 57
Individual projects or research papers	81.6%	79.1%	55.6%	66.7%	87.7%
Homework exercises	55.1%	62.8%	33.3%	50.0%	49.1%
Workshops or labs	52.9%	58.1%	77.8%	72.2%	43.9%
Group work and/or projects	44.9%	44.2%	66.7%	55.6%	43.9%
Exams	12.5%	18.6%	16.7%	5.6%	5.3%
Other. Please explain	3.7%	2.3%	0.0%	5.6%	5.3%

To gauge motivation for teaching these methods, faculty respondents were asked, “Why do you teach these methods in this course? In other words, what are the student learning objectives?” and “What skills do you want students to learn through these methods? Why might they be useful in the future, regardless of vocational choice?” and were able to provide an open-ended response. Of the 253 survey responses, 104 faculty answered at least one of these questions. We combined the qualitative responses to these two questions and then tagged them with up to seven possible categories via inductive coding, as shown in Table 5.⁷ The most common reason cited for teaching these methods is for students to learn research skills and methods (59%) followed by the general importance of reproducibility (37%). There are some slight differences in these motivations and desired learning outcomes based on institution rank. Faculty outside the top 50 are slightly more likely to list data, organization, and career skills (44% compared to 20%) in discussing the intended learning outcomes through these methods.

Table 5 Learning objectives.

Learning objectives	Total	Top 50 LACs	Other LACs
	n = 104	n = 77	n = 27
Research skills and/or methods	58.7%	59.7%	55.6%
Importance of reproducibility	36.5%	35.1%	40.7%
Organizational skills	34.6%	32.5%	40.7%
Career expectations, future skills,
and/or graduate school prep	31.7%	29.9%	37.0%
Coding, software, and/or syntax skills	30.8%	29.9%	33.3%
Data management and/or analysis skills	26.0%	19.5%	44.4%
Econometric and/or statistical skills	18.3%	16.9%	22.2%

The resulting learning objectives closely reflect the findings of the National Academies of Sciences, Engineering, and Medicine (2018) on instilling data acumen as the most critical task in educating future data scientists. The key concepts involved in developing data acumen include “mathematical foundations, computational foundations, statistical foundations, data management and curation, data description and visualization, data modeling and assessment, workflow and reproducibility, communication and teamwork, domain-specific consideration, ethical problem solving” (National Academies of Sciences, Engineering, and Medicine 2018). These similarities suggest that economics faculty may look to data science educators and curricula for ways to incorporate reproducible methods more widely in the economics curriculum.

4 Conclusion

Transparency and reproducibility in empirical economic research has gained more visibility and traction over the last decade. Educating current and future generations of economists is crucial for continued improvement in the reproducibility of economic research (Vilhuber 2021). To achieve this progress, educators have a responsibility to set high standards of reproducibility through the norms and incentives they provide to students. This has the potential to create a “trickle-up” effect on the discipline more broadly (Ball and Medeiros 2012; Ball et al. 2022). More and more undergraduate students are developing data and empirical skills across a variety of disciplines and thus the importance of reproducibility will continue to grow (Schwab-McCoy, Baker, and Gasper 2021; Vilhuber et al. 2022).

While some explicit training in reproducible methods in economics has emerged over the past decade, no systematic survey of economic educators had been conducted on this topic. We survey nearly 1500 economics faculty from the top 161 liberal arts colleges with an economics major in the United States and find that 51% of survey respondents teach these reproducible methods, mostly in advanced upper-level electives (42%) and basic econometrics (31%) courses.

These results suggest that while some educators value and prioritize teaching reproducible methods, there is room for improvement and continued outreach to facilitate and encourage instructors to incorporate these methods in their teaching. The similarities in the desired learning objectives noted here and those proposed as necessary to develop data acumen in future data scientists suggests that economic educators have many opportunities to collaborate and develop pedagogical best practices alongside faculty in data science and other adjacent fields. To what extent these methods are taught at other types of institutions, in graduate programs, and in other empirical disciplines needs further exploration. Finally, the question of whether teaching reproducible methods at the undergraduate level trickles up to the field more broadly remains open.

Supplementary Materials

A copy of the survey instrument, copies of the distribution emails, and the data and code needed to reproduce our findings are available here: https://figshare.com/s/ae6ec51b730d6e7564f5.

Acknowledgments

We would like to thank Richard Ball for helpful input and feedback on the survey instrument and thorough feedback on an early draft of this manuscript. We also thank our discussant and participants at the 2022 Conference on Teaching and Research in Economic Education (CTREE) and the 2022 Southern Economics Association Annual Meeting for their valuable feedback. Finally, we thank the editor, associate editor, and several reviewers whose feedback substantially improved the clarity of this manuscript.

Data Availability Statement

The authors confirm that a copy of the survey instrument, copies of the distribution emails, and the data and code needed to reproduce our findings are available here: https://figshare.com/s/ae6ec51b730d6e7564f5.

Disclosure Statement

No potential conflict of interest was reported by the authors.

Notes

1 There is also a smaller literature that defends the credibility of scientific research claiming that the prevalence of problematic research is less than currently perceived (Fanelli 2018). However, they do not deny the replicability and reproducibility issues altogether.

2 Replicability can be statistical or scientific. Statistical replication examines the same population with the same model but a different sample; while scientific replication, studying a different population and sample, is more relevant for economists (Hamermesh 2007).

3 All faculty (including visiting and adjunct faculty but excluding emeritus faculty) listed on the institutional economics department websites were included.

4 The initial email was sent on June 24, 2021, using Qualtrics. Weekly reminders were sent until the survey closed on July 17, 2021. We then resent the survey on October 26, 2021 and sent one additional reminder. The protocol 973. Teaching Reproducibility in Undergraduate Economics was verified by the Dickinson College Institutional Review Board as Exempt according to 45CFR46.101(b)(1): (1) Educational Practices on June 29, 2021.

5 In the survey, we distinguish between these course types by listing the course options in this way: “(a) an introductory statistics course students take before econometrics; (b) a basic econometrics course; (c) an advanced statistics or econometrics course, taken after basic econometrics; or (d) other. Please specify.” For those respondents who selected “other” courses, the course titles (when provided) are listed in Appendix B, Table B1. Additionally, we cannot distinguish between economic statistics courses (typically taught by economics faculty) and probability and statistics courses (typically taught by mathematics/statistics faculty). Given that our respondents are economics faculty, it is likely that these are primarily economic statistics courses, but we do not have that information in our sample.

6 These “other” methods include: Reproducing the results of well-known articles in econometrics; unit root and cointegration analysis and forecasting; preregistration of experiments and data analysis plans; Stata do-files; discussing the idea of reproducibility as a research standard; pre-analysis plans, pre-registration; discussion of replication studies and replication of a research study; version control; research plans and self-memos.

7 See Appendix A for some common phrases included in each category.

Appendix A:

Learning Objectives

The categories listed below are the result of qualitative inductive coding of the (n = 104) open-ended responses received for questions 10 and 11 of the survey from those faculty that noted teaching reproducible methods in at least one course over the past three years. The phrases listed under each category are those common and/or repeated phrases from responses coded in each category. Responses were categorized in all applicable categories, up to seven possible, however, this varies from only 1 to a maximum of 5 in the sample.

10: Why do you teach these methods in this course? In other words, what are the student learning objectives?

11: What skills do you want students to learn through these methods? Why might they be useful in the future, regardless of vocational choice?

Data management and analysis skills

Data analysis

Big Data

Data management

Cleaning data

Data wrangling

Organizational Skills

File management

Documentation methods

Organizational skills

Working directories

Relative directory/file paths

Codebook development

Coding/Software/Syntax

Coding skills

Programming skills

Basic coding

Careful coding

Commenting code

Programming skills

The Importance of Reproducibility

Best practices

Reproducible methods

Reproducible results

Replicability

Accountability

Research Skills/Methods

Research methods

Formal research paper

Empirical investigation

Scientific research

Original research

Quantitative reasoning

Empirical research

Empirical methods

Research integrity

Research skills

Literature review

Econometrics/Statistical Skills

Interpretation skills

Econometric methods

Forming testable hypothesis

Interpreting multivariate regressions

Testing for significance

Addressing endogeneity issues

Generating summary statistics

Running regressions

Creating graphs

Statistical analysis

Descriptive statistics

Data visualization

Career/Graduate School Prep

Meet current expectations

Needed skills

Appendix B:

Other Courses

Table B1 lists course titles, when provided by respondents, for courses where reproducible methods are taught, which are included in the “Other” category in Table 1 (n = 57). Some respondents provided course titles, some provided course numbers only, and some provided neither. There were also a few duplicate course titles which we have only included here once. As a result, we list 39 course titles below.

Table B1 Other courses.

Advanced Labor Economics	Introduction to R and RStudio
Advanced Topics in Labor Economics	Labor Economics
Advanced Topics in Macroeconomics
Applied Microeconomics	Macroeconomic Modeling
Computational Modeling of Economic Behavior	Macroeconomic Theory
Economics of Immigration	Money and Banking
Economics of the European Union	Research Methods
Economics of the Healthcare Industry	Research Methods in Economics
Economics Research Methods	Research Seminar
Economics Research Seminar	Seminar in Economic Development
Environment and Development	Seminar in Environmental and Resource Economics
Environmental and Resource Economics	Seminar on Law, Society and Government
Experimental Economics	Senior Capstone Seminar
Federal Reserve	Senior Honors Seminar
Health Economics	Senior Research Seminar
Honors Seminar in Economics	Senior Seminar: Women in Development
Independent Honors Project	Senior Thesis/Undergraduate Research
Intermediate Macroeconomic Analysis	Tax Policy in Global Perspective
International Finance	Theory and Measurement in Economic History
Introduction to Game Theory	Topics in Public Policy