Learning effects of the flipped classroom in a principles of microeconomics course

Abstract

The authors of this article estimate the learning effects of the flipped classroom format using data from 16 sections of principles of microeconomics over a 4-year period. The experimental design is unique in that two treatment and two control sections were taught during the fall semester in four consecutive years. Further, the instructor switched the time of day when the treatment and control sections were taught each year. Controlling for gender, ACT score, a normed high school GPA, Pell Grant award, time of day, and initial knowledge of economics, the authors find no evidence of increased learning using end-of-semester measures for students in the flipped classroom in comparison to sections with a moderate amount of active learning.

Keywords:

• Active learning
• flipped classroom
• inverted classroom
• pedagogy
• principles of microeconomics

JEL CODES:

• A20
• A22

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Acknowledgments

The authors thank the participants of a Conference on Teaching Research in Economic Education (CTREE) session in Atlanta in May/June 2016, a 2017 University of Richmond Economics seminar, James Hornsten and others at the 2018 Western Economic Association Annual Conference, and three anonymous referees for their helpful comments. They also acknowledge the assistance of the University of Richmond’s Admissions Office, Registrar’s Office, Financial Aid Office, and Office of Institutional Effectiveness. The University of Richmond Institutional Review Board approved the authors’ research protocol for requesting students’ personal data and their use.

Notes

1 Indeed, one can imagine that in preparing for a flipped classroom approach, the instructor spends significant time prior to its implementation refining lectures, preparing videos, designing assignments, and generally thinking about how to teach the course well. It may, then, simply be the additional focus on how to teach a class that leads to improved outcomes, rather than the flipped format per se.

2 Bishop and Verleger (2013) and O’Flaherty and Phillips (2015) provide broad surveys of the flipped classroom in general.

3 A flipped-blended class is one in which students watch video lectures outside of class and use class time for problem-solving in groups (flipped), but they spend less time in class and more time working online than in a traditional class (blended).

4 In addition, students were incentivized to attend class in the flipped, but not the traditional format in Lombardini, Lakkala, and Muukkonen (2018).

5 This conclusion is supported by some of the results in Calimeris and Sauer (2015), which show that when “time spent” variables are included in the estimation, most of the significant differences between flipped and traditional class formats disappear. Furthermore, Lombardini, Lakkala, and Muukkonen (2018) report, based on surveys, that students found the pretests more valuable than the lectures in the flipped class.

6 A time-of-day effect could be found, for example, if students in a section taught at noon demonstrate less learning, not because of the class format, but because they are hungry.

7 It is, of course, possible that a student in a traditional section gained access to online lectures if a roommate or teammate was in a treatment section and shared their login information. We are skeptical that such efforts were made to spend time watching online lectures that mirrored the traditional lectures a student in a control section observed.

8 The decision to make the noon and 1:30 pm sections our treatment sections in year one was made for no reason other than two sections had to be chosen, and it was easier if one section immediately followed the other. This decision was made before there was any knowledge of the registered students, who are primarily first-year students. IRB procedures do not allow the instructor any access to student academic background information such as SAT scores or high school GPAs until semester grades are submitted. The instructor also does not know which students consented until after grades were submitted.

9 This is almost certainly true of first-year students, who in general do not have access to information from prior students in the course when they register. It is possible that older students might have known about the experiment prior to registering, but they still would not have known whether the section for which they registered was a treatment or control section. Furthermore, only four percent of students were non-freshmen.

10 Students are required to take principles of microeconomics before principles of macroeconomics at the University of Richmond.

11 The final exam, which included the postcourse TUCE, was administered on three or four different days, as determined by the University Registrar. A student’s course meeting time gave a default day for the final exam, but students could take the final with another section if they desired. Each exam had the same total number of points available. We control for the day on which students took the exam. The final exam grade was curved in each year by adding the same number of points in a particular year to all students’ exam grade that year.

12 t tests reveal that the difference in means between years was not significant at the 10 percent level or lower.

13 There were an additional two students in 2014 whose scores on the TUCE did not improve, and another two students in 2014 whose score on the TUCE fell by one point; they are included in the results in table 6a but not in our baseline analysis.

14 We explicitly include these year dummies instead of using fixed effects in order to identify any potential time trend.

15 The adjustment standardizes the GPA to account for differences in the rigor of courses.

16 The first exam was administered on either the 1st or 2nd day of the exam period (on the first day in 2015 and 2017 and on the second day in 2016), while the last exam was administered on day 5 or day 4 or day 8 of the exam period (2015, 2016, and 2017, respectively).

17 The significant drop in the share who took the exam on any given day may be due to the fact that there were four exam days available in 2017 rather than three.

18 Because fractional response models admit only values between 0 and 1, we had to top-code the course grade variable (one student earned a 1.003 which we coded as a 1) and bottom-code the gap variables (for the two students in 2014 who did worse on the TUCE posttest, which we coded as 0).

19 Even when controlling for year and time of a section, the errors for students in a given section could still be correlated. Intuitively, this could happen, for example, if the instructor is energized by a particularly good dynamic in one section. Similarly, one might expect that the variance of the error term may differ for students with different levels of academic preparation. Clustered standard errors are corrected for correlation across errors within a cluster as well as for heteroscedasticity.

20 There are also two students in the 2016 sections for whom the adjusted high school GPA variable was not available, and they are also excluded.

21 Our R-squared values (which are calculated only for the linear regressions) are broadly consistent with the literature: they are around 0.3, which is quite close to the value in Balaban, Gilleskie, and Tran (2016) and Calimeris (2018), and in the middle of the range in Olitsky and Cosgrove (2016).

22 Because the gap outcome variable incorporates the TUCE precourse score, we do not include the precourse score as a control variable for regressions with gap as the dependent variable.

23 We do not include whether a student took the exam early or late in the exam period here because the final exam took place after all this work was completed.

24 We also might be concerned that the flipped classroom format might incentivize students to spend more time studying overall, so that any increase in performance is the result of spending more time studying rather than the result of a different pedagogical technique. We therefore also include the self-reported time spent studying variables as controls when estimating equation (1)$${\text{Outcome}}_{\mathit{ijt}}=\alpha +{\beta }_1{\text{Treatment}}_{\mathit{ijt}}+{\beta }_2{\text{Female}}_{\mathit{ijt}}+{\beta }_3{\text{ACT}}_{\mathit{ijt}}+{\beta }_4{\text{TUCE} \text{pretest}}_{\mathit{ijt}}+{\beta }_5{\text{Pell} \text{Grant}}_{\mathit{ijt}}+{\beta }_6{\text{High} \text{School} \text{GPA}}_{\mathit{ijt}}+{\beta }_7{\text{First} \text{Exam}}_{\mathit{ijt}}+{\beta }_8{\text{Last} \text{Exam}}_{\mathit{ijt}}+{\gamma }_1{9\text{am}}_{\mathit{ijt}}+{\gamma }_2{1\text{:}30\text{pm}}_{\mathit{ijt}}+{\gamma }_3{3\text{pm}}_{ij\mathrm{t}}+{\rho }_1{2015}_{\mathit{ijt}}+{\rho }_2{2016}_{\mathit{ijt}}+{\epsilon }_{\mathit{ijt}}$$, but find that there is no statistically significant impact with the exception of hours spent on the online chapter quizzes, which has a negative (but small) impact on the overall course grade. This may be because those students who struggled with the course took longer to complete the online chapter quizzes.

25 There are also some students who report spending more than 20 hours per week on the course. This seems improbable due to students’ other commitments (both curricular and extracurricular), casting even more doubt on these self-reported numbers. In order to account for these presumably erroneous estimates, we winsorize the top 5 percent of the time spent studying data.

26 Indeed, when running a probit estimation of the Treatment variable on our student characteristic data, only the gender had any predictive value (see table A1 in the appendix). The ACT score, TUCE pre-test score, Pell Grant amount, and high school GPA were all far from conventional levels for statistical significance. This supports our contention that there was no selection into sections and that ability and aptitude were not correlated with the pedagogical approach. While the gender variable is statistically correlated with being in a treatment section, this is likely due to women registering for 9:00 am classes more often than men, because in two of our three baseline years (2015 and 2017) the 9:00 am class was a treatment section. Across all three years, the 9:00 am section had the most women (25, compared to 17 and 20 in the other sections) and also fewer men than two of the other sections (37 compared to 41 and 43 in the noon and 3:00 pm sections).

27 We are also missing two of the control variables—adjusted high school GPA and Pell Grants—for the 2014 data.

28 We thank participants of the 2016 Conference on Teaching Research in Economic Education (CTREE) at Atlanta for making this point.

29 Indeed, Jensen, Kummer, and Godoy (2015) hypothesize that it is active learning rather than a flipped classroom per se that improves student learning.

30 Becker and Proud (2018), for example, find that the impact of a flipped tutorial differs across the two universities where they implement the method and note that this may be due to tutorial size, academic environment, or other factors.

31 Of course, the instructor’s start-up time in recording lectures and preparing active learning materials is an additional consideration.