Online proctored assessment during COVID-19: Has cheating increased?

Abstract

The author of this article uses two empirical approaches to compare online to face-to-face proctored assessment. Using data from a Dutch economics program, he shows that the relationship between grades and human capital variables remains highly significant for courses with online proctored examinations. Additionally, a search for suspicious grade patterns does not indicate an increase in cheating. Overall, the findings do not provide convincing evidence that online proctored assessment is more conducive to cheating than assessment using face-to-face proctoring. In view of the increasing evidence that unproctored online assessment may suffer from high levels of cheating, this suggests that educational institutions can reduce the risk of cheating by using online proctoring.

Keywords:

• Academic dishonesty
• online assessment
• proctoring

JEL CODES:

• A22
• I21

Even before the COVID-19 outbreak, online learning in higher education grew rapidly. Seaman, Allen, and Seaman (2018) report that in 2016, almost one-third of U.S. college students took at least one online course. The pandemic has accelerated this trend by forcing institutions of higher education to switch to online delivery in the spring of 2020. In most cases, this switch included assessment, given the difficulty of maintaining the traditional way of on-campus testing under a lockdown or a social distancing regime. Across the world, schools and universities thus faced the immediate challenge of maintaining the integrity of their assessment after abruptly abandoning their familiar and trusted assessment environment. This development has put to the fore the issue of the validity of online assessment.

In the midst of the pandemic, most educational institutions have been able to maintain the continuity of their assessment by finding alternatives for student-packed exam halls. In a few cases, tests have been organized in larger proctored physical venues, such as football stadiums, to comply with social distancing rules. For small programs, oral examination via video-conferencing tools has been a viable proctored alternative. Programs also have considered nonproctored alternatives, such as take-home exams, written assignments, and unproctored online tests, possibly supplemented by oral examinations. In many cases, however, programs have tried to stay close to their familiar testing format by using webcam recording software to substitute online for in-person surveillance. The choice among these alternatives may depend on program characteristics. For example, conducting oral examinations or grading individual papers may be too costly for large programs. The choice is also determined by concerns about academic integrity and, more specifically, the potential increase in academic dishonesty when programs move to online assessment.

Academic integrity is one of the issues that determine whether online assessment is an effective alternative to traditional modes of assessment (Oosterhof, Conrad, and Ely 2008; Gikandi, Morrow, and Davis 2011). Among educators, the feeling is widespread that an online environment is more conducive to academic dishonesty (Mellar et al. 2018). In the literature, however, a consensus on whether online assessment increases cheating has yet to emerge. Two recent studies illustrate the lack of consensus. Examining self-reported cheating behavior, Harris et al. (2020) find that online students do not report academic misconduct more often than students at residential universities. They conclude that their research “confounds the common (mis)perception that cheating is more prevalent and easier to accomplish in online learning and assessment” (419). In stark contrast, Bilen and Matros (2021) report on rampant cheating in online assessment following the outbreak of COVID-19 in the spring of 2020.

Based on the weight of the current evidence, educational institutions wanting to maintain the integrity of their assessment cannot ignore the risk of cheating in unproctored online assessment. This explains the demand for online proctoring (OP) services, which are currently being offered by commercial vendors. OP makes use of a webcam to monitor the student during the examination. A big advantage over unproctored assessment is that it allows for identity authentication and prevents students from hiring the services of an expert substitute. The webcam may also capture suspicious behavior suggesting communication with others or the unauthorized consultation of resources. OP is, however, not a perfect substitute for face-to-face proctoring (F2FP). A recent report by SURF, a Dutch partnership of academic institutions on information and communication technology (ICT) issues, discusses the security concerns (SURF [Samenwerkende Universiteiten RekenFaciliteiten] 2020). For those students willing to cheat, the opportunities for cheating with OP are wider than in the setting of a proctored exam hall.

OP offers educational institutions an easily adoptable and low-cost solution to the problem of online assessment, provided that the risk of cheating is not too high compared to traditional testing. For most institutions, the relevant question is not whether OP is as good as F2FP but whether it is good enough to maintain confidence in the integrity of assessment.

Moving from F2FP to OP raises other concerns unrelated to academic integrity. Many students perceive the use of webcam recording in their home environment as an invasion of their privacy. The dependence on technology also makes OP assessment more vulnerable to connectivity issues and technical glitches, further increasing the levels of stress and anxiety during exam sessions that, for most students, are already intrinsically stressful. In addition, not all students may have the quiet home environment that is needed for optimal concentration and performance during an online examination. In addition to academic integrity, these factors may result in test outcomes that do not reflect students’ potential performance under F2FP and may thus compromise the reliability of assessment.

To date, most research on academic dishonesty in online assessment has focused on comparing unproctored online assessment with proctored assessment. For educational institutions unwilling to adopt unproctored online assessment, the more relevant policy question is whether OP is an acceptable substitute for F2FP. Therefore, we compare outcomes under OP and F2FP. To this end, we apply two methods that have been used previously to detect cheating in higher education. We first use the model of Harmon and Lambrinos (2008) (from here on denoted as HL), which relates grades to student characteristics measuring their human capital. If the link between grades and human capital variables were much weaker under OP (compared to F2FP), this would raise doubts about the reliability of OP assessment. We additionally employ an algorithm for detecting suspicious grade patterns that was originally developed by Jacob and Levitt (2003) (from here on denoted as JL) to identify cases of teacher cheating on standardized tests. The JL approach yields standardized scores, which can be compared across courses and cohorts.

While a severed link between grades and student characteristics using the HL approach and high scores using the JL approach may be suggestive of academic dishonesty, they fall short of delivering proof. Both methods offer, at best, indirect evidence of academic dishonesty and lack the rigor of an experimental design or direct observation of attempts to cheat. Unlike in a randomized controlled experiment, differences between OP and F2FP testing may result from variations in the environment in which education and testing are organized. In the context of OP, a weaker relationship between grades and student characteristics might as well be explained by increased stress and anxiety during OP exam sessions for reasons discussed above. With this important caveat in mind, we formulate our research questions as follows:

1. Has the move from F2FP to OP testing following COVID-19 weakened the relationship between grades and student characteristics?

2. Has the move from F2FP to OP testing affected the distribution of JL scores?

These formulations should convey that our objective is to look for changes in statistical patterns in moving from F2FP to OP and that we should be careful in interpreting any statistical deviations between F2FP and OP as evidence of cheating. We investigate these research questions using a large dataset of economics students from a research university in The Netherlands.

Literature review

The literature on cheating in online assessment is a small but growing subset of the literature on academic integrity in general. Holden, Kuhlmeier, and Norris (2021) provide a recent review of research in this area. Academic dishonesty cannot be ignored as a minor problem, as estimates of its prevalence range from 9 percent to 95 percent (Whitley 1998; McCabe, Treviño, and Butterfield 2001). Passow et al. (2006) and Nowell and Laufer (1997) discuss the determinants of cheating, while Kerkvliet and Sigmund (1999) investigate the effectiveness of measures to prevent cheating.

A widely debated issue is whether online assessment is more conducive to cheating than assessment using F2FP due to the absence of face-to-face contact between the student and the instructor (Kennedy et al. 2000; McNabb and Olmstead 2009; Fontaine 2012). Most studies assume that online assessment is unproctored, as the widespread adoption of OP is a recent trend. Unproctored online assessment suffers from the lack of identity authentication of students and the lack of monitoring of unauthorized cooperation and consultation of learning resources (Khare and Lam 2008; Harmon and Lambrinos 2008; Moten et al. 2013). Following the old adage that “opportunity makes the thief,” many instructors feel that academic dishonesty is more of a problem in unproctored online assessment than in traditional assessment (Hard, Conway, and Moran 2006; McNabb and Olmstead 2009; Fontaine 2012; Newton 2015; Mellar et al. 2018). The review of Hollister and Berenson (2009) concludes that maintaining the integrity of assessment is the most reported challenge in distance learning. Based on two simple models of F2FP and unproctored online examinations, Bilen and Matros (2021) conclude that higher levels of cheating should be expected in an online environment. They also note that academic dishonesty in unproctored online tests imposes a negative externality on honest students, which argues against the use of grading on the curve in online assessment. However, other studies have downplayed the risk of academic dishonesty in online assessment (Oosterhof, Conrad, and Ely 2008; Gaytan and McEwen 2007; Duers and Brown 2009). Kennedy et al. (2000) and Yates and Beaudrie (2009) argue that instructors’ concerns about cheating in online assessment result from a lack of experience in online teaching. Khare and Lam (2008) argue that academic integrity may be related to students’ maturity. As students advance in their studies, they may be more intrinsically motivated to master the field of study in which they want to pursue a career. This would imply that (post-)graduate students have a lower inclination to cheat than undergraduates.

Several studies employ student surveys to measure the prevalence of cheating in online assessment. These studies either report students’ general views about the prevalence of cheating or students’ own behavior. In the latter case, one can question whether students will always self-report their cheating truthfully. The evidence from student surveys is mixed. According to Vician, Charlesworth, and Charlesworth (2006), 40 percent of students think that online assessment is more conducive to cheating, whereas 42 percent see no difference with F2FP assessment. The latter percentage is even higher (59%) in the survey by Harmon, Lambrinos, and Buffolino (2010). In a survey of 800 students, Grijalva, Nowell, and Kerkvliet (2006) find no difference in cheating between unproctored online and F2FP assessment. Based on self-reported evidence, Stuber-McEwen, Wisely, and Hoggatt (2009) find that online assessment is less conducive to cheating. Again using self-reported behavior, Harris et al. (2020) find that students at an online university are no more likely to cheat than students in residential education. They also find that cheating decreases with student age. However, Stephens, Young, and Calabrese (2007) and Watson and Sottile (2010) find more evidence of unauthorized collaboration and the unauthorized use of notes in online assessment. In addition, a survey by Berkey and Halfond (2015) finds that 84 percent of 141 students agreed that academic integrity in online assessment is a significant issue. King, Guyette, and Piotrowski (2009) report that 73 percent of 121 students think that online assessment is more conducive to cheating than F2FP testing. Surveyed students also indicated that they were more than four times as likely to cheat in an online class. Mirroring the results for instructors, Kennedy et al. (2000) report that students’ participation in online education reduces their estimates of the prevalence of cheating in online assessment. The recent study by Janke et al. (2021) reports the results of a nationwide student survey in Germany following the shift from on-site to online examination due to COVID-19. Based on responses from more than 1,000 students, they conclude that students cheat more frequently in online than in F2FP exams. Their paper does not, however, provide information on the use of OP. Regarding the effectiveness of OP in discouraging cheating, a recent survey by Paredes, de Jesús Jasso Peña, and de La Fuente Alcazar (2021) among 106 students indicates that 58 percent of students think that OP exams will stimulate students to behave more honestly.

Given the doubts about the reliability of students’ self-reported cheating, researchers have examined academic integrity using statistical models or experimental designs. Based on a statistical analysis of grades, Hollister and Berenson (2009) and Yates and Beaudrie (2009) find no difference in academic dishonesty between unproctored online assessment and F2FP assessment. Englander, Fask, and Wang (2011) criticize the approach of Yates and Beaudrie (2009) by pointing to their failure to take student ability into account. Ladyshewsky (2015) finds no difference between unproctored online tests and F2FP tests in a statistical grade analysis. Comparing OP with unproctored online assessment, Alessio et al. (2017) find that students scored 17 points lower and used less time under OP. Bilen and Matros (2021) try to detect cheating by using access logs in Blackboard to monitor the time that students spend to correctly answer “not-so-trivial” problem-solving questions. For unproctored online tests, students spend much less time on each question than for F2FP tests, suggesting the use of online resources or private tutors. Rampant cheating in online assessment was found by Corrigan-Gibbs et al. (2015). Using innovative detection techniques based on the analysis of wording and the tracking of IP addresses, they find that 26 to 34 percent of students in a massive open online course look up answers online.

A more elaborate methodology to detect cheating is to estimate regression models in which students’ grades are explained by variables measuring their human capital or academic potential and to compare the explanatory power under different testing regimes (Anderson, Benjamin, and Fuss 1994; Brown and Liedholm 2002; Coates et al. 2004). The idea is that academic dishonesty would distort the relationship between grades and academic potential and lower the explanatory power of the regression model. Harmon and Lambrinos (2008) employ this approach to compare grades for two online courses, one of which uses unproctored assessment and the other uses proctored assessment. They find that the explanatory power is significantly lower in the case of unproctored assessment, suggesting a higher prevalence of academic dishonesty. Using the same methodology, Beck (2014) fails to find any difference between proctored and unproctored assessment and argues that variable selection and test design may account for the conflicting findings. The number of observations in these studies is, however, quite small (less than 100 students per test). Recent applications of this approach are Arnold (2016) and Dendir and Stockton Maxwell (2020). Arnold (2016) finds that the ability of student characteristics to explain the variation in unproctored test grades is slightly lower than their ability to explain proctored test grades. Adding an application of the Jacob and Levitt (2003) algorithm to detect suspicious grade patterns, he concludes that unproctored online tests are more conducive to cheating than proctored tests. Dendir and Stockton Maxwell (2020) conduct a quasi-experiment comparing unproctored online testing with OP testing in two online courses. They find that OP is associated with a decline in grades in both courses. In regression models using student characteristics, the explanatory power is also higher under OP.

Fask, Englander, and Wang (2014) use an experimental design to compare F2FP and unproctored online examinations. On a practice test, online students scored 14 percent lower than students using F2FP. On the actual summative test, however, online students did 10 percent better. The authors conclude that the greater opportunities to cheat on an online test are somewhat offset by the disadvantages of an online test environment. Karim, Kaminsky, and Behrend (2014) designed an experiment in which 295 participants from Amazon’s Mechanical Turk were randomly assigned either to an OP test or to an unproctored online test using an honor code. They show that OP is effective in reducing cheating compared to unproctored testing. Vazquez, Chiang, and Sarmiento-Barbieri (2021) conducted two field experiments to examine the effects of proctoring on grades using two different classroom settings (face-to-face and online). In a face-to-face class with 426 students, students had to take a combination of unproctored and F2FP tests; in an online class with 83 students, a combination of unproctored and OP tests was used. Scores on unproctored tests were on average 11 percent higher than those in proctored tests. Interestingly, for the purpose of this article, the use of F2FP had a larger effect on test scores than the use of OP. Finally, Wuthisatian (2020) compares grades for an economics course offered in two proctored environments: OP and F2FP. He finds that grades are significantly lower in the OP environment and attributes this to students’ unfamiliarity with the procedures and requirements of OP testing.

In contrast to much of the literature, the current study compares OP to F2FP. We first employ the HL regression model using a sample of 300 to 600 students per course. We hypothesize that greater cheating during OP exams would lead to a weaker relationship between grades and explanatory variables. This would manifest itself in less significant coefficients and a lower explanatory power for OP courses compared to F2FP courses. Next, we apply the JL algorithm to detect suspicious grade patterns and compare the distributions of JL scores across courses.

Setting, data, and method

Setting

This study was conducted at a research university in the Netherlands that offers a variety of programs in the field of economics and econometrics. We use data from the main bachelor program in economics and business over the period 2015–21. The nominal duration of this program is three years. The program offers students a general background in economics before they choose to specialize in a subfield in their final year. The first two years consist of obligatory core and support courses. We focus on first- and second-year courses, as these contain the highest number of observations per course. Table 1 describes the curriculum in the first two bachelor years in the academic year 2019–20. Note that the Dutch lockdown started in March 2020, after the module 3 examinations.

Table 1. Curriculum BSc Economics and Business, bachelor years 1 and 2.

	Module 1 (September–October)		Module 2 (November–December)		Module 3 (January–February)		Module 4 (March–April)		Module 5 (May–June)
	Course	Credits	Course	Credits	Course	Credits	Course	Credits	Course	Credits
Bachelor-1	Introduction to accounting^a,b	8	Microeconomics^b	8	Macroeconomics^b	8	Behavioral Economics	4	Organization and Strategy	8
	Mathematics1^a,b	4	ICT Skills	4	Statistics1^b	4	Philosophy of Economics	4	Skills	4
							Mathematics2	4
Bachelor-2	International Economics	8	Finance	8	Microeconomics2	8	Introduction to Econometrics	8	Intermediate Accounting	8
	Economics of Taxation	4	Statistics2	4	History of Economic Thought	4	Behavioral Economics	4	Research Project	4

Notes: The table describes the curriculum in the academic year 2019–20. During this year, the course Behavioral Economics was moved from the first to the second bachelor year; this course therefore appears twice in the table. Shaded cells = Courses not included in the empirical analysis due to variations in assessment, different assessment, or curriculum changes.

a Module-1 courses are used to measure academic potential in calculations for bachelor-1 courses.

b Bachelor-1 courses are used to measure academic potential in calculations for bachelor-2 courses.

Each bachelor year consists of courses totaling 60 credits and is organized into five eight-week modules of twelve credits each. Due to a curriculum revamp, the curriculum offered in 2019–20 is not identical to that in prior years. In the analysis below, we focus on courses that were offered during the whole sample period, that showed no major variations in assessment method, and that used proctored individual exams as their principal means of assessment.

We use two indicators of prior performance to measure the human capital of students. The first is high school GPA, which is often used in the literature as a measure of student quality. Due to incomplete high school records, the use of school GPA severely reduces the number of observations in our sample. A further disadvantage of school GPA is that it is recorded in integer values in the research database, which reduces the variability of this measure. Where possible, we therefore use a second measure of prior performance based on grades for prior bachelor courses with F2FP examinations. For our analysis of bachelor-1 courses, we use the average grade for two courses in the first module, Accounting and Mathematics 1, to measure the human capital of students. Note that we cannot do this for the academic year 2020–21, as all bachelor-1 courses switched to OP in that year. For our analysis of bachelor-2 courses, we use the average of five F2FP bachelor-1 courses to measure human capital. These courses are indicated in table 1 with superscript b. As these five courses used F2FP in 2019–20, they can be used in the analysis of bachelor-2 courses in 2020–21.

We include two courses in modules 4–5 from the bachelor-1 curriculum that switched from F2FP examination in large exam halls to OP in 2019–20: Mathematics 2 (denoted Math2) and Organization and Strategy (denoted O&S). Five courses in modules 1–3 switched to OP in 2020–21: Accounting (denoted Account), Mathematics 1 (denoted Math1), Microeconomics (denoted Micro), Macroeconomics (denoted Macro), and Statistics1 (denoted Stats1). The courses Behavioral Economics and Philosophy of Economics are new in the 2019–20 bachelor-1 curriculum and therefore are not considered. The skills courses (ICT skills and Academic Skills) use different modes of assessment.

We include two courses in modules 4–5 from the bachelor-2 curriculum that switched from F2FP to OP in 2019–20: Introduction to Econometrics (denoted IntroEctrics) and Intermediate Accounting (denoted IntermedAccount). Four courses in modules 1–3 switched to OP in 2020–21: International Economics (denoted InterEcon), Finance (denoted Finance), Applied Microeconomics (denoted AppMicro), and Statistics2 (denoted Stats2). The courses Economics of Taxation, History of Economic Thought, and Research Project were not included due to a different mode of assessment or to major variations in assessment during the sample period. Behavioral Economics was moved from bachelor-2 to bachelor-1 and is therefore not considered. In total, we include 13 courses in the analysis.

Regarding delivery, most courses use a combination of large-scale plenary lectures and small-scale tutorials. Attendance at tutorial sessions is mandatory for bachelor-1 courses. Attendance at tutorials is optional, not registered, and typically low for bachelor-2 courses. At least 70 percent of the grade is determined by a proctored individual final exam for the courses selected for the empirical analysis. These final exams either use open questions (for 8 out of 13 courses), multiple choice questions (for 4 out of 13 courses), or a mixture of open and multiple choice questions (1 course).

Regarding the remaining 30 percent of the grade, five courses use midterm tests, three courses use assignments, and two courses use a combination of midterms and assignments. For two courses, the final exam has a weight of 100 percent. For the bachelor-1 course O&S, 30 percent of the grade consists of a paper written by two students. All midterm tests have been organized with proctoring, either using F2FP before, or OP after, the lockdown. Grade components are not separately recorded in the database of the university, which precludes an analysis on the final exam grades only. Regarding the midterm tests, this is not a major problem, as these have moved from F2FP to OP in tandem with the final exams. Regarding assignments, we must keep in mind that their use may lead to differences in the explanatory power of the regression model across courses. For this reason, we will primarily look at time variation in the explanatory power of the regression model per course. Grading follows the Dutch system, in which grades are determined on a 1–10 scale. A grade of 8 or higher corresponds to an A, a grade of 7 to a B. The cutoff score for passing is 5.5. The examination rules allow for compensation, to the effect that three failed grades (between 4.5 and 5.5) can be compensated by higher grades for related courses. Resits are limited to a maximum of three out of ten courses and are scheduled during the summer break. The empirical analysis below uses grades before resits to ensure that students sat for the same exam.

On March 13, 2020, at the end of the second week of module 4, universities were ordered by the Dutch government to close their campuses and switch to online education. From that day onwards, tutorials were organized using Zoom and plenary lectures using Panopto streaming or webcasts (using various software solutions). The examinations also moved online, using third-party e-assessment solutions (either Remindo for standard exams, the Dutch software ANS for exams requiring the use of mathematical notation, or the Dutch software Sowiso for accounting exams). In line with university recommendations, the duration of the online final exam was reduced from three to two hours, and the number of questions was proportionally reduced (for a multiple choice exam, the number of questions was reduced from 40 to 25). Exam questions are created by instructors. Students take the exams simultaneously and have the same set of questions; all exams either use multiple exam versions with a different question order or present the questions in a random order (depending on the software) to reduce cheating. OP was sourced from ProctorExam, a leading commercial provider of OP services in Europe. ProctorExam uses live proctors to monitor student behavior during the examination through the webcam. Students are also required to make a webcam scan of their room before the start of the exam. Live proctors flag suspicious behavior or violations of the rules and report to the school’s examination committee. After reviewing flagged webcam recordings, the examination committee can decide to take disciplinary measures after giving the student a fair hearing.

Data

We use data from six cohorts (2015–16 through 2020–21). Grades and data on students’ background characteristics are taken from the education research database of the university. In line with privacy regulations, the data have been anonymized. The literature on study success in economics identifies several explanatory variables. Academic performance in preparatory education, either in mathematics or more generally, is a prime determinant of student retention in economics programs (Ballard and Johnson 2004; Johnson and Kuennen 2006; Lagerlöf and Seltzer 2009; Arnold and Straten 2012). Variables that measure performance at secondary school would therefore be suitable as human capital variables in our regression model. Our dataset contains high school GPA, which we use as a measure of students’ human capital. As the coverage is incomplete, we use students’ performance in prior bachelor courses as an additional measure of their human capital. Both high school GPA and prior bachelor grades may reflect factors that are important for study success, such as students’ human capital, motivation, and/or self-efficacy. In the past, male students were thought to be better at economics due to their supposedly better math aptitude (Ballard and Johnson 2004). A recent survey by Johnson, Robson, and Taengnoi (2014) concludes that the negative gender gap, in which female economics students perform less well than male students, has been closed over time. We include students’ gender in the regression model to account for a possible gender gap.

In much of the literature on study success, student age is an explanatory variable. We hypothesize a negative relationship between age and grades, as in the Dutch context a higher age at enrollment usually results from study delay at high school. Our regression model also accounts for possible tutorial group effects. Group effects may result for a variety of reasons, all based on the assumption that students in the same tutor group have higher social and academic integration. For courses that use assignments, this may result in close cooperation among group members in making unproctored assignments. Peer effects may also arise when high-performing students increase the learning experience for all students in the tutor group or vice versa. If OP assessment increases the opportunities for cheating through unauthorized collaboration, this could also increase group effects. Students may prefer to cheat with fellow students they know, possibly from their own tutor group.

Table 2 provides descriptive statistics, making a distinction between the COVID-19-impacted cohorts and the earlier cohorts. The total number of students who enrolled in the program during our sample period is 5568, of which 1849 enrolled in the two COVID-19 cohorts. The first rows of table 2 show that the student composition of the COVID-19 cohorts is very similar to that of earlier years. Age, gender, and high-school GPA (denoted GPA(School)) in 2019–20 do not differ significantly from prior years. On average, students enroll at an age of 18.6 years; 30 percent of freshmen are females. Data on high-school performance are available for 3624 students, which leads to a considerable reduction in the sample size. The means for GPA(School) indicate that the academic potential of the COVID-19 cohorts is not significantly different from that of earlier cohorts.

Table 2. Descriptive statistics.

	2015–2018		2019–2020
	Mean	N	Mean	N	t-test	p-value
Age	18.59	3719	18.59	1849	0.24	0.81
Gender	0.30	3719	0.31	1849	0.96	0.34
GPA(School)	6.94	2499	6.95	1125	0.69	0.49
	F2F-proctored		Online proctored				OP from
B1-course
IntroAccount	5.54	4239	5.75	908	3.43	0.00	2020
Math1	6.11	4020	6.12	846	0.16	0.87	2020
Micro	5.81	3730	6.10	750	4.16	0.00	2020
Macro	6.43	3368	6.54	711	1.77	0.08	2020
Stats1	6.82	3229	6.60	713	3.46	0.00	2020
Math2	6.66	2674	6.26	1248	6.74	0.00	2019
O&S	7.22	2664	7.43	1420	5.47	0.00	2019
B2-course
InterEcon	6.30	2691	6.27	541	0.42	0.68	2020
Finance	6.83	2911	6.80	515	0.29	0.77	2020
Stats2	7.10	2622	7.16	526	0.80	0.43	2020
AppMicro	5.77	2798	5.88	508	1.43	0.15	2020
IntroEctrics	6.62	2153	6.42	983	3.38	0.00	2019
IntermedAccount	6.10	2228	6.44	1155	5.75	0.00	2019

The remainder of table 2 reports the mean grades for the bachelor courses that have been selected for empirical analysis. Due to a sizable no-show for individual exams, differences in the number of observations across courses arise. In general, exam participation drops during the academic year. The main driver of this is student dropout during the academic year. To prevent the absence of students who drop out early from biasing the results for courses that take place later in the academic year, the empirical analysis below includes only students who have been active during the whole academic year, measured by their participation in module 5 exams. For bachelor-2 courses, the number of observations is smaller due to first-year dropouts. For each course in table 2, a distinction is made between the mean grades on F2FP and OP examinations. A t-test for the difference between the means is also reported. For seven out of thirteen courses, mean grades in OP courses differed significantly from mean grades in F2FP courses at a 5 percent significance level. Compared to F2FP examinations, mean grades in OP examinations were significantly higher for IntroAccount, Micro, O&S, and IntermedAccount, but lower for Stats2, Math2, and IntroEctrics. No general pattern emerges from these statistics. As test difficulty can vary across cohorts, it is not possible to draw firm conclusions from these statistics.

Method

Equation (1)(1) $$Grad{e}_{i,j,t}={\beta }_0+{\beta }_{1}PriorPer{f}_{i.t}+{\beta }_{2}Gende{r}_{i,t}+{\beta }_{3}Ag{e}_{i,t}+{{\displaystyle \sum {\beta }_{4,g}Group}}_{i,g,t}+{\epsilon }_{i,j.t},$$(1) shows the specification of our regression model: (1) $$Grad{e}_{i,j,t}={\beta }_0+{\beta }_{1}PriorPer{f}_{i.t}+{\beta }_{2}Gende{r}_{i,t}+{\beta }_{3}Ag{e}_{i,t}+{{\displaystyle \sum {\beta }_{4,g}Group}}_{i,g,t}+{\epsilon }_{i,j.t},$$(1) where Grade_i,j,t denotes the grade of student i in course j in year t and ε_i,j,t denotes a random disturbance term. PriorPerf_i,tdenotes the prior academic performance of student i in year t. It is measured in three ways. We estimate regressions using GPA(School) as the performance measure for all courses. For bachelor-1 courses, we add regressions using the average grade of the module-1 courses Math1 and Account as the prior performance measure. These are labeled as module-1 grades in tables 3, 5, and 6. For bachelor-2 courses, we add regressions using the average grade of the bachelor-1 courses Math1, Account, Micro, Macro, and Stats1 as the prior performance measure. These are labeled as bachelor-1 grades in tables 4, 5, and 6. The module-1 and bachelor-1 grades used to measure prior performance are all based on F2FP assessment. We hypothesize that high values for PriorPerf_i,t indicate the presence of more human capital and/or a stronger motivation to study economics and would result in higher grades for subsequent courses. This would imply that the coefficient β₁ is significantly positive. The dummy variable Gender_i,t takes on the value 1 for female and 0 for male students. Age_i,t measures student age. The dummy variables Group_i,g,t take on the value 1 if student i is assigned to tutorial group g and 0 otherwise. A tutorial group dummy is included when at least 5 out of 25 group members are in the sample, as lower numbers may not provide a reliable estimate of the group effect. The summation runs from g = 1 to G − 1, where G is the number of groups included in the regression.

Table 3. Summary of regression models for Bachelor-1 courses.

Prior performance:		GPA(School)						Module-1 grades
Course		2015	2016	2017	2018	2019	2020	2015	2016	2017	2018	2019
IntroAccount	β1	0.93***	0.81***	1.16***	1.06***	1.06***	0.79***
	Adj. R2	0.19	0.17	0.26	0.23	0.28	0.14
	N	431	412	430	423	377	385
	LR	92.7***	104.0***	127.6***	141.8***	47.0***	135.3***
Math1	β1	1.00***	0.60***	0.78***	0.96***	1.02***	1.02***
	Adj. R2	0.13	0.08	0.13	0.16	0.11	0.19
	N	426	413	429	418	375	378
	LR	48.8***	39.4	35.2	24.5***	54.6*	35.4**
Micro	β1	1.10***	0.57***	0.88***	0.85***	0.75***	0.98***	0.77***	0.72***	0.69***	0.59***	0.64***
	Adj. R2	0.28	0.12	0.20	0.14	0.17	0.24	0.49	0.46	0.48	0.35	0.47
	N	424	405	424	400	362	377	574	541	566	558	533
	LR	105.3***	23.8	37.9	18.3	48.0***	43.9***	47.1**	48.9**	43.6	30.1***	58.5***
Macro	β1	0.97***	0.59***	0.83***	0.75***	0.96***	0.92***	0.61***	0.51***	0.52***	0.47***	0.60***
	Adj. R2	0.24	0.18	0.21	0.15	0.20	0.21	0.33	0.40	0.37	0.30	0.33
	N	428	405	423	416	366	383	580	541	570	576	530
	LR	125.9***	50.1**	24.3	19.2	19.0	45.3***	49.4***	141.5***	79.6***	29.7	28.8
Stats1	β1	0.88***	0.64***	0.79***	0.52***	0.84***	1.01***	0.68***	0.65***	0.48***	0.41***	0.58***
	Adj. R2	0.13	0.12	0.18	0.07	0.14	0.19	0.31	0.29	0.28	0.19	0.24
	N	433	414	428	420	373	381	592	551	574	581	537
	LR	34.3	39.4	42.8*	12.9	25.9	35.4**	50.0**	34.9	48.5*	24.6	25.6
Math2	β1	0.91***	0.48***	0.81***	0.75***	0.93***	0.90***	0.71***	0.66***	0.73***	0.67***	0.71***
	Adj. R2	0.16	0.05	0.12	0.12	0.10	0.18	0.45	0.46	0.49	0.45	0.28
	N	427	417	423	416	339	378	585	553	571	577	496
	LR	43.1**	19.0	29.3	22.7***	27.1	40.8***	44.4**	57.0***	53.1**	49.4***	33.4
O&S	β1	0.67***	0.46***	0.66***	0.56***	0.49***	0.73***	0.37***	0.37***	0.41***	0.31***	0.33***
	Adj. R2	0.22	0.17	0.29	0.22	0.14	0.20	0.32	0.31	0.33	0.28	0.25
	N	436	417	434	425	370	408	595	553	578	586	531
	LR	96.3***	99.8***	121.3***	126.8**	111.0***	168.8***	76.1***	64.0***	74.3***	71.6**	48.8***

Notes: β₁ is the coefficient of the measure for prior performance. The full regression model also includes gender, age, and group effects. The LR statistic tests for the joint significance of group effects. Results for courses with online proctored exams are in italics.

Significance levels are indicated as ^*p < .10; ^**p < .05; ^***p < .01

Table 4. Summary of regression models for Bachelor-2 courses.

Prior performance:		GPA(School)						Bachelor-1 grades
Course		2015	2016	2017	2018	2019	2020	2015	2016	2017	2018	2019	2020
InterEcon	β1	0.94***	1.15***	0.92***	1.00***	0.86***	1.20***	0.92***	0.89***	0.88***	0.78***	0.86***	0.86***
	Adj. R2	0.23	0.24	0.16	0.18	0.12	0.19	0.48	0.45	0.38	0.30	0.39	0.29
	N	392	369	369	371	354	321	506	496	536	547	502	518
	LR	58.8***	41.2**	39.0	34.9	18.0	15.6	93.1***	48.1**	62.9***	44.0	105.1***	26.4
Finance	β1	0.84***	0.85***	0.68***	0.84***	0.57***	0.73***	0.87***	0.94***	0.79***	0.79***	0.73***	0.85***
	Adj. R2	0.14	0.22	0.10	0.17	0.08	0.12	0.41	0.52	0.32	0.43	0.32	0.34
	N	398	368	384	376	365	288	517	504	557	555	515	472
	LR	85.4***	122.3***	109.0***	147.6***	20.5	44.9***	125.0***	179.4***	198.9***	323.4***	45.9**	165.7**
Stats2	β1	0.58***	0.74***	0.84***	0.62***	0.69***	0.57***	0.77***	0.66***	0.84***	0.74***	0.79***	0.64***
	Adj. R2	0.16	0.17	0.17	0.18	0.16	0.09	0.38	0.33	0.36	0.40	0.38	0.26
	N	383	361	349	363	351	312	500	491	521	535	498	503
	LR	44.7**	31.1	26.6	40.4*	21.7	22.1	59.4***	36.6	34.8	67.6***	42.0**	30.6
AppMicro	β1	0.83***	0.78***	0.54***	0.74***	0.64***	0.76***	0.77***	0.78***	0.77***	0.71***	0.64***	0.71***
	Adj. R2	0.19	0.19	0.14	0.22	0.06	0.13	0.38	0.43	0.38	0.38	0.25	0.37
	N	383	359	369	362	344	293	500	486	533	532	495	483
	LR	112.3***	133.1***	150.4***	177.5***	20.4	9.9	127.7***	176.1***	289.8***	332.3**	40.0*	37.6
IntroEctrics	β1	0.86***	0.85***	0.57***	0.79***	0.63***	0.56***	0.82***	0.77***	0.78***	0.75***	0.67***	0.64***
	Adj. R2	0.19	0.19	0.10	0.18	0.10	0.08	0.45	0.44	0.36	0.38	0.27	0.27
	N	395	376	350	360	325	305	514	512	521	531	456	498
	LR	26.7	31.0	19.7	31.7	28.9*	31.3**	59.9***	47.2**	51.6**	61.9***	26.8	33.2
Intermed	β1	1.05***	1.04***	0.96***	0.88***	0.86***	0.92***	0.94***	0.85***	0.89***	0.77***	0.76***	0.78***
Account	Adj. R2	0.18	0.21	0.13	0.19	0.15	0.18	0.39	0.41	0.33	0.36	0.29	0.36
	N	379	361	354	348	326	357	492	495	515	516	454	560
	LR	94.4***	114.5***	102.3***	142.2***	26.7	155.5***	119.0***	206.1***	170.9***	251.5***	37.7*	380.1***

Notes: β₁ is the coefficient of the measure for prior performance. The full regression model also includes gender, age, and group effects. The LR statistic tests for the joint significance of group effects.

Results for courses with online proctored exams are in italics.

Significance levels are indicated as ^*p < .10; ^**p < .05; ^***p < .01

Table 5. Goldfeld-Quandt tests, OP vs. F2FP cohorts.

Prior performance:	GPA (School)	Module-1 grades		GPA (School)	Bachelor-1 grades
B1-course			B2-course
IntroAccount	0.730   (1.000)		InterEcon	1.166*** (0.033)	1.352**  (0.000)
Math1	0.959    (0.694)		Finance	1.356*** (0.000)	1.810*** (0.000)
Micro	0.983   (0.577)		Stats2	1.010   (0.446)	1.092*   (0.097)
Macro	0.904   (0.892)		AppMicro	0.758  (0.999)	0.737   (1.000)
Stats1	1.010   (0.443)		IntroEctrics	0.918  (0.896)	0.999   (0.503)
Math2	1.341***  (0.000)	1.809*** (0.000)	IntermedAccount	0.850  (0.994)	1.042   (0.214)
O&S	1.142**  (0.011)	0.943*   (0.798)

Notes: The table includes values for the F-statistic; the regression models used to calculate the F-statistics exclude group effects; p-values are shown in parentheses.

Significance levels are indicated as ^*p < .10; ^**p < .05; ^***p < .01

Table 6. Summary of regression results for academically weaker students.

		Bottom quartile of bachelor grades			GPA (School) = 6
Course		2018	2019	2020	2018	2019	2020
Prior performance measure: Module-1 grades
Micro	β1	0.50***	0.41***		0.60***	0.49***
	Adj. R2	0.06	0.10		0.26	0.40
Macro	β1	0.36***	0.32*		0.40***	0.53***
	Adj. R2	0.05	0.01		0.12	0.24
Stats1	β1	−0.14	−0.02		0.18	0.25*
	Adj. R2	−0.01	0.01		0.04	0.03
Math2	β1	0.36**	0.47*		0.55***	0.83***
	Adj. R2	0.02	0.02		0.17	0.31
O&S	β1	0.23*	0.28*		0.19*	0.41***
	Adj. R2	0.01	0.02		0.04	0.16
Prior performance measure: Bachelor-1 grades
InterEcon	β1	0.05	0.65*	0.42	0.37	1.04***	0.93***
	Adj. R2	−0.02	0.05	−0.01	0.05	0.21	0.10
Finance	β1	−0.12	0.55**	−0.33**	0.47**	0.76***	0.77**
	Adj. R2	−0.01	0.04	0.02	0.04	0.18	0.14
Stats2	β1	−0.04	0.69**	1.45***	0.78**	1.21***	0.76***
	Adj. R2	0.08	0.08	0.12	0.06	0.37	0.11
AppMicro	β1	−0.22*	0.09	0.94***	0.51**	0.51***	0.68***
	Adj. R2	0.04	0.01	0.10	0.07	0.12	0.22
IntroEctrics	β1	−0.12	0.56*	0.57*	0.61*	0.39*	0.67***
	Adj. R2	0.07	0.03	0.01	0.01	0.03	0.18
IntermedAccount	β1	−0.13	0.71**	−0.27***	0.52**	0.46**	0.20
	Adj. R2	−0.01	0.01	0.09	0.25	0.07	−0.03

Notes: β₁ is the coefficient of the measure for prior performance. The full regression model also includes gender and age. Results for courses with online proctored exams are in italics.

Significance levels are indicated as ^*p < .10; ^**p < .05; ^***p < .01

Equation (1)(1) $$Grad{e}_{i,j,t}={\beta }_0+{\beta }_{1}PriorPer{f}_{i.t}+{\beta }_{2}Gende{r}_{i,t}+{\beta }_{3}Ag{e}_{i,t}+{{\displaystyle \sum {\beta }_{4,g}Group}}_{i,g,t}+{\epsilon }_{i,j.t},$$(1) is estimated using Ordinary Least Squares (OLS). The R-squared statistic indicates the explanatory power of the regression model and has been used as an indication of cheating in the literature. The tighter the fit between human capital variables and grades, the more likely it is that these grades derive from students’ own ability instead of from cheating. We examine the significance of the differences in explanatory power between F2FP and OP cohorts using the Goldfeld-Quandt test. The Goldfeld-Quandt statistics are calculated excluding group effects, as these do not measure an individual student characteristic. We add two robustness checks to the regression analysis. First, we restrict the sample to bottom quartile students based on prior bachelor grades. Students in the lowest quartile run a higher risk of dropping out. With more “at stake,” the temptation to cheat may be higher. A related robustness check restricts the sample to students with GPA(School) equal to 6 (which is the lowest value). Second, we check for an interaction effect between gender and prior performance. If one of the sexes is more prone to cheating, this could result in a significant coefficient on the interaction term.

The presence of insignificant coefficients and a low R-squared in estimates of equation (1)(1) $$Grad{e}_{i,j,t}={\beta }_0+{\beta }_{1}PriorPer{f}_{i.t}+{\beta }_{2}Gende{r}_{i,t}+{\beta }_{3}Ag{e}_{i,t}+{{\displaystyle \sum {\beta }_{4,g}Group}}_{i,g,t}+{\epsilon }_{i,j.t},$$(1) may suggest cheating. However, as discussed in the introduction, there may be other reasons for a weakening of the link between grades and human capital variables in OP examinations. In addition, the regression output does not measure the likelihood of cheating at the level of individual students. We therefore apply the JL algorithm to generate scores that may be indicative of cheating at the student level. The original JL paper aims to identify teacher cheating on standardized tests, following the introduction of incentives for schools to improve their test performance. In this context, the JL score measures the gain in the test score of a class (compared to the previous year) and ranks this gain relative to the gains of other classes. Arnold (2016) adjusts the formula by taking the student instead of the class as the unit of observation and by defining the gain relative to prior academic performance. In this article, we use prior bachelor grades as our measure for prior performance. For bachelor-1 courses, we use the average grade of the module-1 courses Math1 and Account; for bachelor-2 courses, we use the average grade of the bachelor-1 courses Math1, Account, Micro, Macro, and Stats1. In our application, we compare the ranked gain for an OP course with the ranked gain for an F2FP course. These choices result in the following formula: (2) $${\text{Score}}_{\text{i},\text{OP},\text{F2FP}}={\left(\text{rank}\_{\text{gain}}_{\text{i},\text{OP}}\right)}^{\text{2}}+{\left(\text{1}-\text{rank}\_{\text{gain}}_{\text{i},\text{F2FP}}\right)}^{\text{2}},$$(2) where Score_i,OP,F2FP is the JL score for student i when comparing an OP course with an F2FP course. The term rank_gain_i,OP measures the percentile rank of student i’s performance gain for a course using OP examination, where the gain is measured as the grade differential with the average of prior bachelor grades and the gain’s percentile rank is determined within the student population of the course. Rank_gain_i,F2FP measures the corresponding variable for a course using F2FP examination. When student i has a large gain on an OP examination but a small gain on F2FP examination, this will result in a high value of Score_i,OP,F2FP. In case a student has the highest percentile rank for the OP examination and the lowest for the F2FP examination, Score_i,OP,F2FP will attain its maximum value of 2. In the reverse case, the value will be 0. The squares in equation (2)(2) $${\text{Score}}_{\text{i},\text{OP},\text{F2FP}}={\left(\text{rank}\_{\text{gain}}_{\text{i},\text{OP}}\right)}^{\text{2}}+{\left(\text{1}-\text{rank}\_{\text{gain}}_{\text{i},\text{F2FP}}\right)}^{\text{2}},$$(2) have the effect of giving more weight to large differences between the gains for OP and F2FP examinations, allowing the measure to pick up suspicious patterns better. Both the standardization in a range between 0 and 2 and the increased weight for outliers are advantages of the JL score over simpler measures, such as the difference between grades on OP and F2FP examinations.

Comparing unproctored and F2FP tests, Arnold (2016) finds that the distributions of JL scores for unproctored tests exhibit much fatter upper tails, signaling suspicious grade patterns. The application of this method requires the observation of students in both an OP and an F2FP environment, preferably in a time window that is not too long. We therefore calculate JL scores only for the 2019–20 academic year.

We would like to stress that our use of the JL score does not implicate individual students in dishonest behavior. While a high value for Score_i,OP,F2FP raises the suspicion of academic dishonesty in OP examination, it does not constitute proof. As discussed above, other factors related to OP may compromise the reliability of assessment. ICT issues and the lack of an optimal exam environment at home may increase students’ stress levels and reduce their ability to concentrate on the exam. These factors may weaken the link between students’ performance in an OP examination and their academic potential. As such, they offer an alternative explanation for a decrease in the explanatory power of the regression model in equation (1)(1) $$Grad{e}_{i,j,t}={\beta }_0+{\beta }_{1}PriorPer{f}_{i.t}+{\beta }_{2}Gende{r}_{i,t}+{\beta }_{3}Ag{e}_{i,t}+{{\displaystyle \sum {\beta }_{4,g}Group}}_{i,g,t}+{\epsilon }_{i,j.t},$$(1) . It is, however, more difficult to see how these factors could give rise to high JL scores. This would require students with high-ranked gains on F2FP examinations to suffer disproportionately from the stress and anxiety of OP examinations, moving students with low-ranked gains on F2FP examinations up on the OP ranking. Nevertheless, individual students may be sensitive to the difference in testing format or may have suffered a blackout during the F2FP examination, which may explain their high JL score. Therefore, we will not look at individual scores but at patterns in the distribution of the JL scores.

Results

Regression models

Tables 3 and 4 report a summary of the estimation results of equation (1)(1) $$Grad{e}_{i,j,t}={\beta }_0+{\beta }_{1}PriorPer{f}_{i.t}+{\beta }_{2}Gende{r}_{i,t}+{\beta }_{3}Ag{e}_{i,t}+{{\displaystyle \sum {\beta }_{4,g}Group}}_{i,g,t}+{\epsilon }_{i,j.t},$$(1) for, respectively, bachelor-1 and bachelor-2 courses by course and academic year. We do not lump the pre-COVID years together as the results by year give an indication of the variation in coefficients and R-squared unrelated to COVID-19. The coefficients of Gender and Age and the group effects are not reported but will be briefly summarized to conserve space. In 41 out of a total of 139 regressions, Gender is significant at a 5 percent level. In the majority of these cases, β₂ has a positive value, providing further evidence against the notion that female students are worse at economics. In 19 out of 139 regressions, the coefficient for Age has the hypothesized negative sign, which is significant at a 5 percent level. In five regressions, β₃ is significantly positive.

Regarding group effects, the LR-statistics indicate that these are jointly significant in most regressions (in 60% of all regressions when using a 5% significance level). There is no clear distinction between F2FP and OP courses. For the bachelor-1 courses in table 3, group effects are more often jointly significant in OP courses than in F2FP courses (in 73% of OP courses vs. 59% of F2FP courses). The reverse holds for bachelor-2 courses (in 31% of OP courses vs. 70% of F2FP courses). Taken together, group effects are more often jointly significant in F2FP courses than in OP courses (64% vs. 48%). The size of the group effects also has not increased following the move from F2FP to OP. The average absolute value of β_4,g (not reported in tables 3 and 4) dropped slightly from 0.368 for F2FP courses to 0.331 for OP courses. In short, the regression results do not point to a strong increase in the size and significance of group effects due to cheating among group members following the move to OP.

We now turn to β₁, our main coefficient of interest, which is reported in tables 3 and 4. Across all courses and cohorts, the coefficient for PriorPerf is positive and significant at a 1 percent level. This holds for both bachelor-1 and bachelor-2 courses. It also holds regardless of whether we measure prior academic performance using GPA(School) or using prior bachelor grades. These measures are thus a good indicator of subsequent academic performance due to their ability to measure students’ human capital, motivation to study, and/or self-efficacy. The move from F2FP to OP seems to have had no discernable effect on the size of β₁, which averaged 0.770 for F2FP courses versus 0.768 for OP courses. As discussed above, a decline in the goodness-of-fit between the human capital variables and the grades for OP courses could indicate the presence of cheating. Tables 3 and 4 show that the adjusted R-squared in general is higher for the regressions that use prior bachelor grades instead of GPA(School) as a measure of prior performance. This could be explained by the larger number of observations, the larger variation in bachelor grades compared to GPA(School), and the fact that bachelor grades are a more recent measure of academic performance. Moving from F2FP to OP courses, the average R-squared drops from 0.277 to 0.228. This drop is smaller for regressions using GPA(School) (from 0.168 to 0.158) than for regressions using bachelor grades (from 0.386 to 0.298).

The significance of the changes in explanatory power is further examined using the Goldfeld-Quandt test. Table 5 reports for each course the F-statistics of a pairwise comparison of error variances of the regression model for F2FP versus OP years. The changes in explanatory power are highly significant for the bachelor-1 courses Math2 and O&S and for the bachelor-2 courses InterEcon and Finance. From our regression models we conclude that, although human capital variables remain strongly related to course grades under OP, the explanatory power has weakened.

The case of the course Math2 cautions, however, against a premature conclusion that the lower explanatory power is due to increased cheating. As table 2 shows, the mean grade for Math2 in OP cohorts was significantly lower than in F2FP cohorts. This raises the alternative explanation that the stress from doing a difficult math exam in the new OP environment has led to underperformance and a weaker link between human capital variables and grades. In this interpretation, OP stress has the statistical effect of adding noise to the regression model, which leads to a reduction in the R-squared. To discriminate between these alternative explanations, we will check for suspicious grade patterns using the JL scores in the next subsection.

First, however, we repeat the regression approach for a subset of students with a relatively weak prior academic performance. As these students are more likely to fail, they may be more tempted to cheat. Due to the low variability of GPA(School), we do this analysis using only bachelor grades as a measure of prior academic performance. The subsample of weak students is determined in two ways: 1) by taking the bottom quartile (i.e., the lowest-scoring quartile for either the module-1 grades for bachelor-1 courses or the bachelor-1 grades for bachelor-2 courses) and 2) by selecting students with GPA(School) equal to 6. Table 6 provides a summary of the regression results for the years 2018–20. Restricting the sample to students with a weak prior performance limits the variability in student quality, and thus has a strong negative effect on the value and significance of β₁ and on the explanatory power of the regressions. The average value of β₁drops to 0.29 for regressions using the bottom quartile and to 0.57 for regressions that include students with a GPA(School) equal to 6. The average R-squared is also much lower than in tables 3 and 4 and drops to 0.04 and 0.14, respectively. However, this is the case for both F2FP and OP courses. For OP courses, the average β₁ in table 6 is higher than for F2FP courses, while the R-squared is similar. Overall, we find no evidence that the link between course grades and human capital has changed among weaker students after the move to OP.

We finally check for an interaction effect between gender and prior academic performance. Gender differences in academic integrity could come to the surface when OP facilitates cheating. For the final three years, we redo the regressions from tables 3 and 4 but now include the interaction term PriorPerf*Gender. The coefficients of the interaction term are reported in table 7. Just four out of 67 interaction terms are significant at a 5 percent level, which shows that either gender differences in cheating are small or that proctoring (either F2FP or OP) works well to prevent cheating.

Table 7. Coefficients of interaction term between gender and GPA (School).

Prior performance:	GPA (School)			Bachelor grades
Course	2018	2019	2020	2018	2019	2020
IntroAccount	0.13	−0.01	−0.10
Math1	−0.02	−0.27	−0.36
Micro	0.29	−0.68***	−0.47**	0.09	−0.08
Macro	0.12	−0.16	−0.04	−0.08	−0.06
Stats1	0.10	−0.11	−0.43	0.09	−0.11
Math2	−0.31	−0.45	−0.04	−0.02	−0.01
O&S	−0.06	−0.01	0.10	−0.04	−0.06
InterEcon	0.24	0.12	−0.14	−0.20*	−0.07	0.03
Finance	0.07	−0.06	−0.06	−0.03	0.04	0.10
Stats2	−0.06	−0.02	−0.05	−0.23**	−0.21*	0.00
AppMicro	0.14	0.40	−0.40	0.00	0.15	−0.10
IntroEctrics	−0.06	−0.14	0.16	−0.23**	−0.05	0.07
IntermedAccount	0.36	−0.05	0.13	−0.17	−0.12	−0.08

Notes: The full regression model also includes gender, age, and group effects. Results for courses with online proctored exams are in italics.

Significance levels are indicated as ^*p < .10; ^**p < .05; ^***p < .01

Ranked-gain scores

We now examine suspicious grade patterns using the JL methodology introduced above. For each student i, we calculate Score_i,OP,F2FP on the basis of equation (2)(2) $${\text{Score}}_{\text{i},\text{OP},\text{F2FP}}={\left(\text{rank}\_{\text{gain}}_{\text{i},\text{OP}}\right)}^{\text{2}}+{\left(\text{1}-\text{rank}\_{\text{gain}}_{\text{i},\text{F2FP}}\right)}^{\text{2}},$$(2) . The analysis is restricted to data for the academic year 2019–20 because in this year we can observe students’ performance in both F2FP and OP courses. For bachelor-1, we compare the ranked gain for OP courses Math2 and O&S to the F2FP courses Micro, Macro, and Stats1. In these cases, the gain is measured vis-à-vis the average grade of the module-1 courses Math1 and Account. This yields six series of JL scores. For bachelor-2, we compare the OP courses IntroEctrics and IntermedAccount to the F2FP courses AppMicro, InterEcon, and Stats2. In these cases, the gain is measured vis-à-vis the average grade of the bachelor-1 courses Math1, Account, Micro, Macro, and Stats1. This yields another six series of JL scores. Figure 1(A)–(D) plots the distributions of the scores of the OP courses vis-à-vis the F2FP courses Micro and AppMicro.¹ In each case, the top graph shows the distribution for the pre-COVID period and the bottom graph the distribution for 2019. A visual inspection of the graphs shows that the top graphs are somewhat smoother than the bottom graphs, which can be explained by the larger pre-COVID sample. Apart from that, the distributions seem quite similar. The graphs also do not suggest the presence of heavier tails in the distributions for the 2019 cohort. For Score_{i,MATH2,MICRO}. the percentage of scores above 1.5 is 1.6 percent for the pre-COVID cohorts and 0.8 percent for 2019. For Score_i,O&S,MICRO, the percentage of scores above 1.5 is 1.5 percent for the pre-COVID cohorts and 1.0 percent for 2019. For Score_{i,INTROECTRICS,APPMICRO} and Score_{i,INTERMEDACCOUNT,APPMICRO}, the pre-COVID and 2019 percentages are very close (respectively, 1.9% vs. 2.0% and 1.8% vs. 2.2%).

Figure 1. (A) JL scores of Math2 vs. Micro. (B) JL scores of O&S vs. Micro. (C) JL Scores of IntroEctrics vs. AppMicro. (D) JL Scores of IntermedAccount vs. AppMicro.

We formally test the equality of the distributions using the Mann-Whitney test and the Kolmogorov-Smirnov two-sample test. Table 8 reports the statistics. For all 12 combinations of OP and F2FP courses, the statistics are insignificant, indicating that the distributions of JL scores in 2019 do not significantly deviate from those in the pre-COVID years. Our analysis of JL scores thus does not reveal suspicious grade patterns that could be indicative of academic dishonesty.

Table 8. Distribution tests for Jacob-Levitt scores.

F2FP course:		Micro	Macro	Stats1
OP-course:
Math2	MW	−0.850	−0.154	−0.295
	p-value	0.395	0.878	0.768
	KS	0.938	1.248	0.771
	p-value	0.342	0.089*	0.593
O&S	MW	−0.888	−0.396	−0.315
	p-value	0.374	0.692	0.753
	KS	0.922	0.943	0.536
	p-value	0.363	0.337	0.936
F2FP course:		AppMicro	InterEcon	Stats2
OP-course:
IntroEctrics	MW	−0.120	−0.830	−1.060
	p-value	0.904	0.406	0.289
	KS2	0.717	0.755	0.769
	p-value	0.684	0.619	0.595
IntermedAccount	MW	−0.943	−0.583	−0.514
	p-value	0.346	0.560	0.607
	KS2	1.048	0.978	0.706
	p-value	0.222	0.295	0.701

Notes: MW = Mann-Whitney test; KS2 is Kolmogorov-Smirnov two-sample test.

Significance levels are indicated as ^*p < .10; ^**p < .05; ^***p < .01

Discussion and conclusions

We have examined the potential increase in academic dishonesty following the move from F2FP to OP examinations due to COVID-19. To this end, we have employed two statistical methods that have been used previously in the literature: the regression model of Harmon and Lambrinos (2008) and the detection algorithm of Jacob and Levitt (2003). Using a large dataset from a Dutch university, we have examined two research questions. The first is whether the move from F2FP to OP testing has weakened the relationship between grades and student characteristics, as the HL model would predict in case of increased cheating. The second is whether the move from F2FP to OP testing has changed the distribution of the JL scores in a way that is suggestive of increased cheating.

Regarding the first research question, we find that variables measuring students’ human capital remain strongly related to course grades following the move to OP testing. However, the ability of these variables to explain the variation in grades has decreased. While this could point to increased cheating, we cannot exclude an alternative explanation, according to which the stresses from the OP environment add “noise” to the regression model and lower the explanatory power of the student characteristics. Regarding the second research question, we find that the distributions of the JL scores have not significantly changed following the move to OP testing and do not indicate suspicious grade patterns.

Overall, our findings do not provide convincing evidence that OP examination is more conducive to cheating than F2FP examination. While OP and F2FP may not be perfect substitutes, we find no evidence that OP has compromised the integrity of assessment. This finding contrasts with increasing evidence that unproctored online assessment may suffer from high levels of cheating. It suggests that educational institutions that want to move their assessment online can reduce the risk of cheating by organizing OP examinations instead of unproctored online tests. While OP is more costly than unproctored online assessment, it still offers institutions an easily adoptable and relatively low-cost solution to the problem of online assessment.

We close with some caveats. First, we would like to stress that our analysis was not intended to uncover academic dishonesty of individual students and that the results offer no proof for this. Second, our findings are based on data from one university and should be generalized with caution. Third, it is conceivable that over time, students become more adept at circumventing webcam monitoring. This could result in an increase in cheating on OP tests and compromise the integrity of OP testing after all. One could imagine an “arms race” in which institutions are forced to continue to invest in technological solutions to counter students’ efforts at circumventing OP. Our suggestion for further research is therefore to repeat this type of research over time and in different institutional settings.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

1 Graphs of JL scores vis-à-vis Macro and Stats1 and InterEcon and Stats2 look similar but have been omitted out of space considerations. They are available upon request.