Comparing Student Performance in a Graduate-Level Introductory Biostatistics Course Using an Online versus a Traditional in-Person Learning Environment

Abstract

Our study compared the performance of students enrolled in a graduate-level introductory biostatistics course in an online versus a traditional in-person learning environment at a school of public health in the United States. We extracted data for students enrolled in the course online and in person from 2013 to 2018. We compared average quiz and final exam scores between students in the two learning environments adjusting for demographic characteristics and prior academic performance using linear mixed models. Data were available for 1461 (83.1%) students learning online and 298 (16.9%) students learning in person. After adjusting for sex, race/ethnicity, age, quantitative GRE score, undergraduate GPA, and math refresher score, we found quiz scores for students learning online were about 2.5% lower than those for students learning in person, on average, with a 95% confidence interval ranging from 4.9% lower to 0.02% higher. Differential performance was even closer to equality for the final exam where scores for students learning online were about 0.9% higher with a 95% confidence interval ranging from a 3.9% reduction to 5.8% improvement. These estimates suggest comparable student performance can be achieved in a graduate-level introductory biostatistics course among students learning online and in person. Supplementary materials for this article are available online.

Keywords:

• Distance education
• Statistics education
• Public health

1 Introduction

Imagine a world where the traditional classroom vanished and the only option available for teaching and learning was online. That imaginary world became a reality in 2020 as a result of the Coronavirus disease (COVID-19) pandemic, introducing a time when learning via online platforms became a critical necessity for students at all stages of their education and for the financial viability of universities across the nation. Every teacher, regardless of their field of study or expertise, was teaching online, and every student, regardless of their age, sex, or race/ethnicity, was learning online. Given this seemingly overnight increase in prominence, quality assessments of the online learning environment in comparison with the traditional in-person learning environment are of paramount importance (Kim and Bonk 2006; Mathieson 2010).

Even before the pandemic, online education was on the rise. According to the Education Department’s National Center for Education Statistics report from 2018, 38.0% of graduate (and 33% of all college) students took at least one online course, where 9.1% (and 17.6%) took both online and in-person courses, and 28.9% (and 15.4%) took only online courses, which were all higher than earlier reports (Ginder et al. 2018). Online education has expanded to accommodate learning styles of specific cohorts such as the Net Generation (i.e., people born between 1982 and 1991), who were the “first to grow up with digital and cyber technologies” (Barnes et al. 2007, p. 1), were “constantly exposed to computer-based technology” (Sandars and Morrison 2007, p. 85), and were inclined to be “independent” and “autonomous” learners (Barnes et al. 2007). For such students, many have contemplated whether an online education would be comparable to traditional in-person learning (Larreamendy-Joerns and Leinhardt 2006; Mathieson 2010; Means et al. 2010; Xu and Jaggars 2014; Wladis et al. 2015).

As the availability and sophistication of online courses has grown, faculty and students alike have had to decide whether they wanted to branch out into the digital world (Atchley et al. 2013). Faculty (and students) have questioned whether they would have the technical skills needed to use online tools, the ability to successfully teach (or learn) course content asynchronously, the aptitude to effectively interact with one another online and participate in synchronous discussions, and a suitable personality that would shine on screen. Addressing these questions across science, technology, engineering, and mathematics (STEM) fields is especially critical, where many continue to advocate for the in-person learning environment (Noble 2004; Smith et al. 2008). Even advocates for online STEM courses acknowledge a need for additional support far beyond what is required in the traditional classroom, especially for women and younger students (Wladis 2015).

The literature addresses these questions by examining undergraduate-level courses in non-statistical disciplines (McFarlin 2008; Winquist and Carlson 2014), undergraduate-level statistics courses (McLaren 2004; Ward 2004; Dutton and Dutton 2005; Summers et al. 2005; Wilson 2013; Gundlach et al. 2015; Milic et al. 2016), and graduate-level courses in non-statistical disciplines (Caywood and Duckett 2003; Rovai and Jordan 2004). The few studies that have investigated graduate-level introductory biostatistics courses have been limited to small sample sizes over a limited number of terms (Evans et al. 2007; McGready and Brookmeyer 2013; Loux et al. 2016). We aim to contribute to the ongoing national discussion about the relative comparability of online and in-person learning environments by providing a detailed analysis of data from our experiences teaching graduate-level introductory biostatistics.

Our graduate-level introductory biostatistics course “Biostatistical Applications for Public Health” (BAPH) has been a Master of Public Health (MPH) core course required of all incoming MPH students at a school of public health within a private university in Washington, DC. This course was taught exclusively in person until 2013, when a new online degree program was created and included BAPH as a core course. Wanting all students to have comparable educational experiences in this fundamental core course, the primary in-person course director designed the online version of BAPH. This course director developed and taught all sections of BAPH that were examined in our study to remove instructor variability. Both the online and in-person class sizes varied from approximately 50–150 students with a wide range of mathematical and statistical backgrounds. Students self-selected the version of the course in which they wanted to enroll based on the mode of delivery, so there was no random assignment of students to the online and in-person courses.

The goal of our study was to provide an objective comparison of learning outcomes between students enrolled in online and in-person versions of the same graduate-level biostatistics course. While our study was not randomized, several important sources of bias were controlled by the unique nature of our study. First, the online and in-person course materials were designed by the same course director, which eliminated differences due to variation in the course content. Second, the same course director delivered content in both learning modes, which controlled for variation in teaching styles. Third, the courses both had the same basic structure: Students learning in person attended a weekly lecture, while students learning online were expected to watch the lectures asynchronously at their convenience (where the online recordings mirrored in-person lectures); both groups then attended equivalent synchronous breakout lab sessions led by different lab instructors. Fourth, this was a multi-year study where data were compiled over a lengthy period from the 2013 to 2018 academic years, which allowed us to take year-to-year variation of the student body into account. Student demographic characteristics and academic performance upon entry into the MPH program were also available for adjusted analyses. Therefore, our study provides a unique look at the comparison of online and in-person learning environments among students with certain common characteristics.

2 Methods

2.1 Course Description

BAPH is a three-credit graduate-level introductory course in the application of biostatistical principles to the critical analysis of studies in the health sciences literature. In this course, students are taught how to select an appropriate statistical method based on a research question, perform calculations of basic statistical procedures for estimation and inference, interpret the results of statistical analysis, and understand numbers in published literature and health news.

2.2 Online versus in-Person Learning Environments

As mentioned previously, the online and in-person versions of this course were created and taught by the same course director. All the online asynchronous material was purposefully designed and recorded to match the style of the highly regarded in-person lectures. Both online and in-person sections covered the exact same content. For all students, the methods of instruction included lectures, required and recommended readings, and class and small lab group discussions. All students were expected to engage in some independent learning each week, which included reviewing assigned material, preparing for class discussions, working on assignments, studying for exams, and completing group work and independent assignments. Nongraded practice problems were assigned from Bernard Rosner’s Fundamentals of Biostatistics textbook every week to prepare them for quizzes and exams.

Students learning in person attended lectures—delivered by the same course director—as an entire class once a week for two hours. Following lectures, they broke out into smaller groups of approximately 20 students for a one-hour lab session led by lab instructors. These lab instructors were Teaching Assistants (current students) or Instructional Assistants (graduates from the program) who had successfully completed the course. During these interactive lab sessions, students worked in small groups to complete and present solutions to nongraded lab exercises that reinforced the material presented during lecture, while the lab instructor moderated discussions.

Each week, students learning online independently navigated through two hours of asynchronous material stored on a cloud-based software-as-a-service platform provided by 2 U, an educational technology company. They received a handout to accompany the recorded video lectures, which closely followed the PowerPoint lecture slides of the students learning in person. As students listened to the recordings, they were cued to pause the video, look at their handout, and attempt to solve a problem on their own before watching the solution. Knowledge checks and flip books were dispersed throughout the asynchronous material to diversify the flow. Students learning online also met via Adobe Connect or Zoom in smaller groups of approximately 15 students once a week for a two-hour synchronous live session led by an instructor holding a master’s or PhD degree. These live sessions resembled the breakout lab sessions for the students learning in person, with additional time devoted to more thorough question-and-answer discussions of the asynchronous material. As these live session instructors assumed the same role as the in-person TAs and IAs, we will also refer to them as lab instructors hereafter. The course director, holding a PhD, monitored the progress of all students and lab instructors.

While BAPH was a three-credit course that required the same workload for all students regardless of delivery mode, the length of the terms differed for students learning online versus those learning in person. The in-person course ran for 15 weeks on a semester schedule, which included 13 weeks of new lecture material followed by a review session during week 14 and a final exam during week 15. The online course spanned 11 weeks on a quarterly schedule, which included 10 weeks of new lecture material followed by a review session and a final exam during week 11.

Both online and in person, student performance was evaluated using lab participation (based on attendance and completion of in-lab exercises), a math refresher, in-class and take-home quizzes, and a final exam. The math refresher was a free-response assignment used to assess students’ baseline mathematical preparedness. All students in overlapping terms were asked the same questions on the in-lab exercises, math refresher, quizzes, and the final exam. The materials were organized differently to accommodate the difference in term lengths, but the nature of the problem sets was generally the same otherwise. Quiz and exam questions were problem solving in the form of multiple choice, and students were instructed to show all work to receive full credit. Partial credit was awarded on quizzes for incorrect answers with partially correct approaches, but all-or-nothing grading was used for the final exam. For consistency, the lab instructors strictly followed the grading criteria set up by the course director.

Students learning in person had to complete three in-class quizzes and three take-home quizzes. In-class quizzes, which were designed to take between 30 and 45 minutes, were given during the first hour of the lecture period. Solutions to the quizzes were discussed immediately after all students had submitted their quizzes at the one-hour mark. Students learning online had to complete five quizzes (one every other week). The quizzes were designed to take between 45 and 60 min, but students were given two hours in case they experienced any technical difficulties with the online data entry system or scanning in their work (in an attempt to relieve technophobia-related anxiety). Online quizzes were slightly longer (with respect to number of questions and length of time) than the in-person quizzes because there were five instead of six in total. Quizzes focused on course content covered in the previous two weeks, but the final exam was cumulative, covering all topics.

All course materials for the in-person version were housed in Blackboard Learn, the university’s official learning management system (LMS). All materials for the online version (both asynchronous and synchronous) were housed in 2U’s LMS. Students were encouraged to post general questions directly to the Discussions section on Blackboard or the course wall on 2U’s platform, where the course director and lab instructors would post answers in a timely manner.

2.3 Study Design

This was a retrospective study. Demographic and prior academic performance data for all students who were enrolled in BAPH under the direction of the same course director between the Fall semester of 2013 and Fall semester of 2018 were extracted by the university’s Office of Institutional Research and Planning and the Office of Recruitment and Admissions for students learning in person and by 2 U analysts—a third-party vendor who administered and managed the online admissions process and was contractually required to archive admissions applications—for students learning online.

For in-person applicants, an Institutional Research Analyst and an Assistant Director of Admissions extracted the data from the Schools of Public Health Application Service (SOPHAS), which is a centralized application service that archives application data. If the SOPHAS application did not include the requested data, they searched in the university’s electronic filing cabinet (or database) for archiving student information. They de-identified all data to protect the students’ privacy and confidentiality. They entered all de-identified data in encrypted (password protected) Excel files. There was no link between study code numbers and direct identifiers. The files were emailed securely using ZixCorp and stored on the university’s enterprise file sharing service for online cloud storage and collaboration with two-step authentication to secure access. The university’s Office of Human Research approved our request for a waiver of consent and an exemption from Institutional Review Board (IRB) review.

The course director provided the Offices with student scores on the math refresher, quizzes, final exam, and lab participation to be de-identified and merged with the demographic and prior academic performance data, including sex (female, male), race/ethnicity (White, Black, Asian, Hispanic, not reported), age (years), student hours (full-time, part-time), quantitative Graduate Record Examination (GRE) scores, and undergraduate grade point average (GPA). We included a lab instructor variable indicating which (a) TA/IA instructed the students learning in person during the small breakout lab sessions (determined by the first letter of the student’s last name) or (b) instructor taught the students learning online during the live sessions.

2.4 Statistical Methods

We used two primary outcomes in our study to assess differences in student performance by learning environment (i.e., online versus in-person). The first outcome was quiz average, which was calculated as the average of five quizzes for those learning online and the average of six quizzes for students learning in person. The discrepancy in number of quizzes was only due to the difference in term length (i.e., 10 weeks online versus 14 weeks in person). The second outcome of interest was the final exam score, which was calculated on a scale from 0 to 100 points. In the descriptive analysis, we summarized these outcomes on their original scale. In the statistical models used to compare learning environments, however, Likelihood Ratio tests revealed an improved fit for both outcomes after a natural log-transformation.

The primary independent variable was learning environment. Other predictors of interest included sex (female, male), race/ethnicity (White, Black, Asian, Hispanic, not reported), age (years), student hours (full-time, part-time), academic year (categorical from 2013 to 2018), quantitative GRE score (above 50${}^{\text{th}}$ percentile, below 50${}^{\text{th}}$ percentile, not reported), undergraduate GPA (above 3.0, below 3.0, not reported), and math refresher score. We used the 50${}^{\text{th}}$ percentile and 3.0 as respective cutoffs for GRE and GPA to reflect how admissions decisionmakers at the university typically use this information. Relevant summary statistics for these predictors, both aggregated and stratified by learning environment, can be found in Section 3. We conducted inference for simple comparisons using Wilcoxon Rank-Sum tests for continuous variables and Pearson’s Chi-square test of independence for categorical variables.

We modeled the natural log-transformed quiz average and final exam score using separate linear mixed effects models. The covariance structure was implied through specification of two random effects. The first random effect was for lab instructor to control for variability that may be attributed to a given instructor within the course. The second random effect was for academic year to account for year-to-year random variability in the student body. We compared models with and without these random effects, as well as models with only one of the two, and chose to include both random effects since this model had the lowest Akaike information criterion (AIC).

In the mixed model analyses, we included several additional fixed effects to adjust for sex, race/ethnicity, age, quantitative GRE score, undergraduate GPA, and math refresher score. In the model for quiz scores, we also adjusted for math refresher score as a linear spline after scatterplot investigation and a Partial F-test revealed an improved fit with different slopes before and after the breakpoint of 80. The mathematical form of each model has been provided in Appendix ∼ A.

All statistical tests were two-sided. Significance levels were set to 0.05. Interval estimates were calculated to have a 95% confidence level. We performed all analyses using SAS/STAT® software, Version 9.4 of the SAS System for Windows.¹

3 Results

Descriptive analyses in Table 1 revealed our sample to have more female and more White students with an average age of about 27 years. Students learning online and in person showed little overall difference in sex distribution. The race distributions were similar, but students learning online had a slightly lower percentage of White and a slightly higher percentage of Black students compared to those learning in person. Students learning online tended to be older. Table 1 also summarizes the students’ academic backgrounds. In both groups, the majority were full-time, with an average undergraduate GPA of 3.28 and an average quantitative GRE score in about the 46th percentile. Comparing students learning online and in person, we found that the proportion of full-time students was lower online, the mean GPA was lower online, and the quantitative GRE tended to be lower online. Lastly, Table 1 summarizes course assessments based on the math refresher, average quiz score, and final exam score. The average math refresher, average quiz scores, and the final exam scores, were all, on average, lower for students learning online.

Table 1 Descriptive statistics for the students’ demographic characteristics, academic background, and performance on course assessments.

	Online	In-Person	Total
Variable	n = 1461 (100%)	n = 298 (100%)	n = 1759 (100%)
Demographics
Sex
Female	1143 (78.2)	239 (80.2)	1382 (78.6)
Male	318 (21.8)	59 (19.8)	377 (21.4)
Race/ethnicity
White	616 (42.2)	153 (51.3)	769 (43.7)
Black	293 (20.1)	42 (14.1)	335 (19.0)
Asian	224 (15.3)	32 (10.7)	256 (14.6)
Hispanic	84 (5.7)	16 (5.4)	100 (5.7)
Not reported	244 (16.7)	55 (18.5)	299 (17.0)
Age (years)
Mean (SD)	32.0 (8.9)	27.1 (5.8)	31.1 (8.7)
Median (Q1, Q3)	29.0 (26.0, 35.0)	26.0 (24.0, 28.0)	28.0 (25.0, 34.0)
Academic background
Student hours
Full-time	1139 (78.0)	277 (93.0)	1416 (80.5)
Part-time	322 (22.0)	21 (7.0)	343 (19.5)
Academic Year
2013–2014	83 (5.7)	74 (24.8)	157 (8.9)
2014–2015	276 (18.9)	50 (16.8)	326 (18.5)
2015–2016	376 (25.7)	52 (17.4)	428 (24.3)
2016–2017	427 (29.2)	59 (19.8)	486 (27.6)
2017–2018${}^{\mathrm{a}}$	214 (14.6)	0 (0.0)	214 (12.2)
2018–2019	85 (5.8)	63 (21.1)	148 (8.4)
Undergraduate GPA
Above 3.0	1096 (75.0)	257 (86.2)	1353 (76.9)
Below 3.0	303 (20.7)	36 (12.1)	339 (19.3)
None reported	62 (4.2)	5 (1.7)	67 (3.8)
Mean (SD)	3.3 (0.4)	3.4 (0.4)	3.3 (0.4)
Median (Q1, Q3)	3.3 (3.0, 3.5)	3.4 (3.2, 3.7)	3.3 (3.1, 3.6)
Quantitative GRE percentile
Above 50${}^{\text{th}}$ percentile	350 (24.0)	148 (49.7)	498 (28.3)
Below 50${}^{\text{th}}$ percentile	579 (39.6)	114 (38.3)	693 (39.4)
None reported	532 (36.4)	36 (12.1)	568 (32.3)
Mean (SD)	43.3 (20.8)	53.1 (19.7)	46.0 (29.8)
Median (Q1, Q3)	41.0 (28.0, 60.0)	53.0 (39.0, 68.0)	45.0 (29.0, 61.0)
Performance on course assessments
Math refresher score
Mean (SD)	90.7 (14.2)	94.0 (6.4)	91.2 (13.3)
Median (Q1, Q3)	94.0 (88.0, 98.0)	96.0 (92.0, 99.0)	95.0 (89.0, 98.0)
Average quiz score^b
Mean (SD)	87.1 (11.6)	92.1 (9.7)	87.9 (11.4)
Median (Q1, Q3)	89.4 (81.0, 95.5)	94.3 (86.3, 99.2)	90.5 (82.0, 96.3)
Final exam score
Mean (SD)	84.9 (17.3)	87.5 (15.2)	85.3 (17.0)
Median (Q1, Q3)	86.7 (75.0, 96.7)	89.0 (77.7, 98.3)	87.5 (75.0, 97.5)
Final course grade
Mean (SD)	86.9 (12.1)	92.6 (9.2)	87.9 (11.9)
Median (Q1, Q3)	88.9 (80.4, 95.7)	93.7 (87.9, 98.8)	90.0 (81.4, 96.4)

${}^{\mathrm{a}}$The course director did not teach this course in person during the 2017-2018 academic year.

${}^{\mathrm{b}}n=5$ students learning online were missing an average quiz score.

Results from our primary analyses are shown in Table 2, which summarizes the estimates from the linear mixed models comparing students learning online to those learning in person for the two outcomes of interest (i.e., log-transformed average quiz score and final exam score). Interactions between key demographic variables and learning environment were investigated, but we did not find evidence to support their inclusion. After adjusting for the aforementioned predictors, we found that quiz scores for students learning online were about 2.5% lower than quiz scores for students learning in person, on average, with a 95% confidence interval ranging from about 4.9% lower to about 0.02% higher, which just barely includes equal performance within the interval. Differential performance was near equality for the final exam where scores for students learning online were about 0.9% higher with a 95% confidence interval ranging from about a 3.9% reduction to about a 5.8% improvement suggesting no evidence of differential performance for this comparison. Given that the scores were out of 100 points, these estimates are not indicative of a large degree of differential performance. Little or any material difference in performance was also observed between sexes with an estimated comparison between females and males that was near 1 with interval estimates that did not show evidence substantially favoring either a positive or negative effect.

Table 2 Exponentiated estimates with corresponding confidence intervals from two adjusted mixed models with log-transformed average quiz score and final exam score as the outcomes.

	Average quiz score${}^{\mathrm{a}}$		Final exam score${}^{\mathrm{a}}$
	Exponentiated		Exponentiated
	estimate	Type III	estimate	Type III
Variable	(95% CI${}^{\mathrm{b}}$)	p-value	(95% CI${}^{\mathrm{b}}$)	p-value
Learning environment		0.057		0.72
Online	0.975 (0.951, 1.0002)		1.009 (0.961, 1.058)
In-person	Reference		Reference
Sex		0.25		0.47
Female	1.008 (0.993, 1.024)		1.015 (0.975, 1.056)
Male	Reference		Reference
Race/ethnicity		<0.0001		0.008
Asian	0.998 (0.980, 1.017)		1.020 (0.970, 1.072)
Black	0.953 (0.937, 0.970)		0.926 (0.884, 0.970)
Hispanic	0.974 (0.947, 1.0003)		0.984 (0.915, 1.058)
Not reported	0.980 (0.962, 0.997)		0.973 (0.928, 1.021)
White	Reference		Reference
Undergraduate GPA		<0.0001		0.098
Below 3.0	0.959 (0.944, 0.974)		0.962 (0.922, 1.004)
None reported	0.993 (0.960, 1.027)		1.044 (0.956, 1.140)
Above 3.0	Reference		Reference
Quantitative GRE		<0.0001		0.0004
Below 50${}^{\text{th}}$ percentile	0.957 (0.943, 0.974)		0.920 (0.883, 0.959)
None Reported	0.989 (0.971, 1.007)		0.958 (0.913, 1.006)
Above 50${}^{\text{th}}$ percentile	Reference		Reference
Age	0.998 (0.998, 0.999)	0.0001	0.999 (0.996, 1.001)	0.17
Math refresher score${}^{\mathrm{c}}$		<0.0001	1.006 (1.005, 1.007)	<0.0001
Math refresher > 80 slope	1.007 (1.006, 1.008)
Math refresher < 80 slope	1.002 (1.001, 1.002)

${}^{\mathrm{a}}$The dependent variables (i.e., average quiz score and final exam score) were transformed in the models using the natural log.

${}^{\mathrm{b}}$Confidence interval.

${}^{\mathrm{c}}$In the model with average quiz score as the outcome, math refresher score was included as a linear spline with different slopes before and after the breakpoint of 80.

In addition to the finding regarding our primary comparison of online versus in-person learning environment, the model revealed some other interesting results. After adjustment, those who scored below the 50${}^{\text{th}}$ percentile on the quantitative GRE had about 4.3% lower average performance on the quizzes compared to those who scored above the 50${}^{\text{th}}$ percentile on the quizzes. The 95% confidence interval for this comparison ranged from as much as a 5.7% to as little as a 2.6% reduction. For the final exam, this comparison was estimated to be about an 8.0% reduction with a 95% confidence interval ranging from as much as an 11.7% to as little as a 4.1% reduction. The estimated results for the quantitative GRE effect indicate the possibility of a substantial differential performance associated with better scores on this exam. For example, the 95% confidence interval indicates that being below the 50${}^{\text{th}}$ percentile may be associated with a reduction of roughly 3–10 points from a score of 90 on a scale of 100.

For GPA, those with an undergraduate GPA below 3.0 had about a 4.1% lower average quiz performance compared to those with a GPA above 3.0. The 95% confidence interval for this comparison ranged from as much as 5.6% to as little as a 2.6% reduction. The final exam showed a similar reduction, however its 95% confidence interval allowed for up to a 0.4% improvement associated with this comparison. The GPA estimate is not quite as strong as the GRE estimate, though the 95% confidence interval still indicates a reduction associated with having a GPA below 3.0 of roughly 2 to 5 points from a score of 90.

With regard to race/ethnicity, Black students appeared to fare worse on both assessments compared to White students. We found a reduction of 4.7% on average for the quizzes with a 95% confidence interval ranging from as much as a 6.3% to as little as a 3.0% reduction. For the final exam, the reduction was 7.4% with a 95% confidence interval ranging from as much as 11.6% to as little as 3.0%. A 7.4% reduction in performance for these students would lead to approximately a 7-point reduction in score from a grade of 90, which could have a material impact on the student’s grade.

For age, a negative, though not meaningful, association was found for both quiz and final exam performances. In both cases, we estimated about a 0.15% decline in performance per year increase in age. While the 95% confidence interval did not include 1 for quiz scores, this estimate is not large and does not imply a meaningful association with quiz performances. The 95% confidence interval did include 1 for final exam performance.

In the model for quiz scores, a scatterplot investigation suggested inclusion of linear spline for math refresher score with a knot at 80, which we included in the model based on evidence suggesting an improved fit from a Likelihood Ratio Test (F = 45.16, 1 numerator df, p < 0.0001). Similar investigation in the model for final exam did not reveal the necessity for such an effect. The math refresher score results used different slopes below versus above a score of 80. The positive slopes indicated that those with better mathematical preparation tended to perform better. The positive magnitude of the slopes was greater for math refresher scores above 80 than below. The slope for math refresher scores above 80 indicated an improvement of about 0.7% per one-point increase of the math refresher score with a 95% confidence interval ranging from as little as 0.6% to as much as 0.8%. As an example, this estimate would imply about a 5-point improvement in quiz score over a score of 85 associated with a 10-point improvement in math refresher score from 80 to 90. For the final exam, the slope coefficient indicated an improvement of about 0.6% per one-point increase of the math refresher score with a 95% confidence interval ranging from as little as 0.5% to as much as 0.7%. The results from the math refresher and quantitative GRE both showed a substantial association between prior mathematical experience and improved performance, which is consistent with our anecdotal experiences.

4 Discussion

In analyzing our data from students enrolled in a graduate-level introductory biostatistics course from 2013 to 2018, our data analysis showed little difference in performance when comparing students in an online environment to those learning the same content in a traditional in-person environment. Our results were adjusted for important predictors that allowed for a comparison among students with similar demographic and academic characteristics. In addition, our random effects control for variability among lab instructor and academic class year. An interesting advantage to our study is that since the lead professor for all classes was the same, we were able to control for content delivery directly through the study design. A disadvantage of our study is the fact that there was no randomization of students to online or in-person learning environments. Students self-selected into the online or in-person environments. Hence, causality cannot be inferred from our results due to the possibility of imbalance in other unmeasurable influential variables which may be associated with both learning environment and academic performance in an introductory biostatistics class.

Several studies reported similar findings to ours. At the undergraduate level, comparable performance levels between students learning online and in person have been reported for courses in statistics (Yablon and Katz 2001; Summers et al. 2005), business statistics (McLaren 2004), earth science (Werhner 2010), programming (Zacharis 2010), personal finance (Ary and Brune 2011), business application software (Wagner et al. 2011), psychology (Arviso 2019), history (Arviso 2019), algebra (Arviso 2019), and environmental science (Paul and Jefferson 2019). At the graduate level, comparable performance levels between students learning online and in person have been reported for courses in applied statistics for industry (Stephenson 2001), behavior management (Caywood and Duckett 2003), biostatistics (Evans et al. 2007; McGready and Brookmeyer 2013), and research methods (Holmes and Reid 2017). Further supporting our findings, Ary and Brune (2011) found pre-course GPAs and ACT scores, as opposed to the learning environment, predicted student performance.

4.1 Where Students Learning in Person Outperformed Students Learning Online

Contrary to our results, Paden (2006) found undergraduate students learning in a traditional in-person section of an introductory math course performed significantly better than those learning online; however, they only adjusted for sex in their analysis. Sue (2005) observed this difference in outcomes among introductory business statistics students—although the discrepancy was attributed to online students taking the exam in person since they were accustomed to being online. Tanyel and Griffin (2014) reported similar differences across multiple disciplines offered in their College of Arts and Sciences, College of Business and Economics, and the School of Education over 10 years.

4.2 Where Students Learning Online Outperformed Students Learning in Person

On the other hand, Atchley et al. (2013) found students enrolled in online courses spanning different disciplines (e.g., psychology, English, computer information systems, and accounting) performed significantly better academically than their counterparts enrolled in the traditional in-person versions of the courses. Their study, however, did not include any statistics courses, and they did not control for any other variables in their analysis. Graham and Lazari (2018) found that students enrolled in an online college algebra course performed significantly better on the departmental final exam than those students enrolled in the traditional in-person class, but they also failed to control for other factors. Similarly, Dutton and Dutton (2005) found undergraduate students enrolled in an online business statistics class achieved significantly higher final exam scores and final course grades than students enrolled in the traditional in-person version of the class, noting that students enrolled online were significantly older with higher GPAs. They also identified GPA and student effort (i.e., homework average) as other “positive predictors of success” (Dutton and Dutton 2005).

4.3 What’s in a Name: The “Online” Course

Some of these differences in student performance could be attributed to the actual design of the “online” course, which has been loosely defined and widely diverse (Sun and Chen 2016). In some studies, the descriptions of their “hybrid,” “blended,” and “flipped” statistics courses more closely resembled the design of our “online” course than other “online” courses (Utts et al. 2003; Ward 2004; Wilson 2013; Winquist and Carlson 2014; Gundlach et al. 2015). As we observed in our study, they found student performance comparable in such “hybrid” and traditional courses (Utts et al. 2003, Ward 2004).

4.4 Our Strengths

Few studies used data from multiple semesters in their analysis (Stephenson 2001; McLaren 2004; Wagner et al. 2011; Tanyel and Griffin 2014). Our study compared the performance of biostatistics students in the two learning environments using five years’ worth of data from 25 different offerings (5 in-person, 20 online) in overlapping terms. The same course director who taught the course in person designed and recorded the asynchronous material for the online version of the course, thus removing instructor variability and ensuring that the students would cover the same content and receive the same assessments. This is a strength over Tanyel and Griffin’s (2014) 10-year study, which (as they acknowledged) was limited by the small number of instructor-matched online and in-person courses available during the same semester.

4.5 Our Limitations

Just as Mathieson (2010) documented “methodological limitations” in her systematic review comparing the performance of undergraduate and graduate students enrolled in online and in-person statistics courses (e.g., “lack of randomization, lack of generalizability, differences in baseline characteristics between groups, and failure to adjust for confounders”), our study also has some limitations (p. 3). Fewer students enrolled in the course in person than online annually, so the sample size is smaller for the in-person subset. Actual class sizes for the lab sessions, however, were about the same. Due to incomplete student records from admissions, data were missing on undergraduate GPA (4.2% online, 1.7% in person) and quantitative GRE scores (36.4% online, 12.1% in person). Employment status, occupation, and undergraduate major were unavailable for most of the students, so these factors could not be used in our analysis. Students self-selected which version of the course they wanted to enroll in based on the mode of delivery, so there was no random assignment of students to the online and in-person courses. Just as we were not able to randomize students, all the other studies referenced reported the same limitation. Stack (2015), however, was able to achieve a “quasi-randomization of students into sections” of a criminology course “due to an administrative error in the course schedule,” and he still did not find a significant difference in performance between students enrolled in the traditional in-person and online sections of the course. We reference Stack (2015) to show that (1) the only study we found that did achieve randomization did so serendipitously and (2) they still reached the same conclusion we did regarding learning environments.

4.6 Future Considerations

When polling our BAPH students at the beginning of the semester to determine how they feel about taking the course, most students typically replied that they are “scared” or “terrified.” Thus, we continuously search for ways to reduce this fear factor and improve our course. Just as Devaney (2010) found graduate students enrolled in statistics courses online experienced less anxiety and developed better attitudes than those learning in person, we could explore ways of delivering content online and in person to create a more comfortable learning environment. Being in a comfortable learning environment can help a student build their confidence. Among first-year public health graduate students who completed an introductory biostatistics course, Loux et al. (2016) found that those students from the “flipped” class (resembling the structure of our online course) were just as confident in their ability to apply biostatistics as those from the traditional in-person class. Moreover, among undergraduate and graduate students who completed at least one online course of any discipline at a university in the Midwest, Eom (2006) discovered that satisfied students performed better in the course. In the future, we would like to examine end-of-semester evaluations to explore student satisfaction and how it relates to student performance in our course.

As suggested in the literature, such improvements in student satisfaction and performance may involve blending the best features of online and traditional in-person courses (Chen and Jones 2007; Gundlach et al. 2015; Chingos et al. 2017). Given the improved performance of students enrolled in “hybrid” or “blended” statistics courses reported in the literature, we could consider offering the traditional in-person version as a “hybrid” course (Wilson 2013; Winquist and Carlson 2014; Gundlach et al. 2015; Milic et al. 2016). This design would closely model the online version with the exception that the live synchronous sessions would be in person as opposed to online.

To do so, we would need to expand our analysis to identify critical success factors. In designing our BAPH course online and in person, care was taken to balance lectures, interactive small group labs, readings, practice exercises, formulas, figures, and interpretations in both learning environments. While we cannot identify which of these features primarily led to student success in our study, Eom and Ashill (2016) found that a student’s performance primarily depends on the course design, instructor, and dialogue. Their findings agree with Volery and Lord (2000), who identified technology and instructor as the critical success factors for an online management course. In their qualitative assessment of online courses presented in the literature, Sun and Chen (2016) also concluded that strong instructors who implement a high-quality course design while building a sense of community lead to a successful online learning environment. This is further supported by Barnes et al. (2007) who recognize that “to be human is to learn, and we learn from good teachers” (p. 6). Identifying these critical success factors in our BAPH course designed and taught by the same course director would allow us to further improve the course and create the optimal learning environment for our students.

5 Conclusion

What was learned from this study could facilitate improvement of educational programming at our university and perhaps elsewhere, which is of paramount importance as online education continues to gain prominence. Our study has shown some evidence that learning outcomes may not be affected materially when presented in an online environment as compared with the traditional in-person standard. Discovering that comparable student performance can be achieved in an online setting is especially important at a time when an unforeseeable event, such as a public health crisis, abruptly forces nearly all university courses to be delivered exclusively online regardless of teacher and student preference or experience. Further research is needed, however, before universities can expand their online offerings more extensively as other considerations of the student experience may also contribute to their willingness to enter an online curriculum. There is still much to be learned, and we have only begun to see the full picture of how educational instruction will evolve in the coming decade.

Software

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ^® indicates USA registration.

Financial Disclosure

The authors declare that they have no financial relationship relevant to this article to disclose.

Acknowledgments

We would like to thank all the students enrolled in the graduate-level introduction to biostatistics courses for contributing data for this study, as well as the lab instructors for their hard work in this course and dedication to these students. We would also like to thank the analysts from the Office of Institutional Research and Planning, the Office of Recruitment and Admissions, and 2U for their assistance with the data extraction. Finally, we would like to thank the Research Assistant who conducted a preliminary exploratory data analysis, as well as the university for providing funding through the Flexible Research Development Award to support this Research Assistant.

Supplementary Materials

The supplementary material includes the description of the mathematical form of each model used in our analysis.

Conflict of Interest

The authors declare that they have no potential conflicts of interest relevant to this article to disclose.

Data Sharing

In the IRB application, we declared that all data would be aggregated/summarized in all publications, and no individual-level results would be communicated to protect the students’ privacy and confidentiality. Since there was no contact with students during the data extraction, we did not provide students with consent documents to obtain necessary permissions to share the data.

Additional information

Funding

This study received partial funding through the university’s Flexible Research Development Award to support a Research Assistant to conduct a preliminary exploratory data analysis.

Notes

1 Copyright [copyright] 2016 SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA.