Revisiting student evaluation of teaching during the pandemic

ABSTRACT

The pandemic has placed unprecedented pressures upon staff and students alike. Yet performance management of academics including Student Evaluation of Teaching (SET) persists. The American Association of University Professors (AAUP) has intervened on this issue. We develop new methods enabling better treatment of pandemic-era SET. Analysis of UK National Student Survey (NSS) data suggests 85% of institutions meet reasonable performance expectations during the pandemic. Results emphasize the need for a more sensitive treatment of pandemic-era SET.

KEYWORDS:

• COVID-19
• education
• pandemic
• student evaluation of teaching
• statistics

JEL CLASSIFICATION:

• A2
• C11
• I2

I. Introduction

SETs remain a ‘ubiquitous but controversial’ part of universities (Boysen 2020). Though potentially informative about teaching problems arise when SET is used to review faculty (Sproule 2000). SETs have been termed ‘student perception data’ (Linse 2017) with students ill-equipped to judge teaching quality. SETs may contribute to grade inflation (Deem and Baird 2020; Marchant et al. 2020), display racial/gender biases and discriminate against quantitative subjects (Marchant et al. 2020). Low response rates (Bacon, Johnson, and Stewart 2016) and respondent anonymity (Raworth 2017) may encourage extreme outcomes.

The pandemic has raised concerns over low student-satisfaction levels (Sangster, Stoner, and Flood 2020). The AAUP has emphasized the need to protect faculty from SETs during the pandemic (Boysen 2020). Sources of student dissatisfaction may lie outside instructors’ control e.g. library access and IT infrastructure (Kerzic et al. 2021) and the effects of social restrictions (Park and Koo 2022). This adds to long-standing concerns about confounding factors associated with SET (Deem and Baird 2020).

The above reflects a long-standing need to analyse numerical teaching data (Sproule 2000) highlighted by the pandemic (Sangster, Stoner, and Flood 2020). Thus, we develop new methods to analyse pandemic-era SET. An application to NSS data suggests around 85% of institutions achieve reasonable performance expectations given the pandemic.

The layout of this paper is as follows. Section II quantifies the effect of the pandemic upon SETs. Section III develops a statistical model later applied to NSS data. Section IV concludes.

II. Quantifying the effect of the pandemic

The Chartered Association of Business Schools (CABS) collect NSS data. The effect of the pandemic can be measured by comparing institutions submitting to both the 2019 and 2021 exercises. Summary statistics in Table 1 show the pandemic is associated with lower student-satisfaction levels and more variable responses. A paired t-test gives evidence of a significant difference in student satisfaction levels (t = 10.058, df = 142, p = 0.000). The pandemic thus results in reduced student satisfaction once we control for different institutions. The effect can be estimated as

(1) $\frac{\mathit{Mean}\mathit{Post}\mathit{Pandemic}}{\mathit{Mean}\mathit{Pre}\mathit{Pandemic}}=\frac{0.736574}{0.8180357}=\mathrm{0.9004173.}$(1)

Table 1. Summary statistics of NSS data: Proportion of students reporting being satisfied with their course.

Statistic	Pre-pandemic	Post-pandemic
Min	0.513	0.18
Max	0.9677	1.000
Mean	0.8180357	0.7365734
Median	0.8305	0.74
Standard Deviation	0.08032912	0.1057109
Upper Quartile	0.86835	0.80
Lower Quartile	0.78735	0.69
Inter Quartile Range	0.081	0.11

Equation (1)(1) $\frac{\mathit{Mean}\mathit{Post}\mathit{Pandemic}}{\mathit{Mean}\mathit{Pre}\mathit{Pandemic}}=\frac{0.736574}{0.8180357}=\mathrm{0.9004173.}$(1) suggests the pandemic is associated with an inevitable 10% reduction in student satisfaction. Karadag (2021) obtains similar estimates.

III. Modelling student satisfaction

We model student satisfaction as follows. Suppose a respondent is satisfied with a course with probability $\mathit{\theta }$. We assume independence of different respondents.¹ Given n responses the probability that r people are satisfied is

(2) $\mathrm{Pr}(\mathit{r}\mathrm{studentssatisfied})=(\begin{array}{c}\mathit{n}\\ \mathit{r}\end{array}){\theta }^r(1-\mathit{\theta }{)}^{\mathit{n}-\mathit{r}}.$(2)

Bayesian statistics allows us to estimate the probability the satisfaction level $\mathit{\theta }$ lies above/below a certain threshold. A reasonable target in non-pandemic times might be ${\theta }_{\mathit{thresh}}=0.8$. Consistent with other commonly-used teaching metrics this is just below average pre-pandemic satisfaction levels (see Table 1). Equation (1)(1) $\frac{\mathit{Mean}\mathit{Post}\mathit{Pandemic}}{\mathit{Mean}\mathit{Pre}\mathit{Pandemic}}=\frac{0.736574}{0.8180357}=\mathrm{0.9004173.}$(1) suggests a more reasonable pandemic-era target would be ${\theta }_{\mathit{thresh}}=0.72$.

Using a standard Be($\mathit{\alpha },\mathit{\beta }$) prior distribution for $\mathit{\theta }$ (Lee 2012) means the posterior distribution for $\mathit{\theta }$ given data in (2) is

(3) $\mathit{\theta }|\mathit{X}\sim \mathit{Be}\left(\mathit{\alpha }+\mathit{r},\mathit{\beta }+\mathit{n}-\mathit{r}\right).$(3)

Using a standard Jeffrey’s prior (Jeffreys 1998) with $\mathit{\alpha }=\mathit{\beta }=1/2$ in (3) gives

(4) $\mathit{\theta }|\mathit{X}\sim \mathit{Be}\left(\mathit{r}+\frac{1}{2},\mathit{n}-\mathit{r}+1/2\right).$(4)

From (4) the probability that the process is on-target is

(5) $\mathrm{P}\mathrm{r}(\mathit{\theta }\ge {\theta }_{\mathit{thresh}})=\mathit{\hspace{0.17em}}1-F_{r+\frac{1}{2},\mathit{n}-\mathit{r}+1/2}({\theta }_{\mathit{thresh}}).$(5)

where $F_{r+\frac{1}{2},\mathit{n}-\mathit{r}+1/2}\left(\mathit{x}\right)$ denotes the $\mathit{Be}\left(\mathit{r}+\frac{1}{2},\mathit{n}\text{break}-\mathit{r}+1/2\right)$ CDF. There is thus no evidence student-satisfaction levels are unduly low unless Pr($\mathit{\theta }\ge {\theta }_{\mathit{thresh}}$)<0.05.

We analyse data for business students during the 2021 NSS. Table 2 shows once the pandemic is accounted for only 25/162 institutions clearly miss the target of ${\theta }_{\mathit{thresh}}=0.72$. This result remains robust to the specification of alternative prior distributions. Results reflect unprecedented efforts devoted to pandemic-era teaching (Sangster, Stoner, and Flood 2020). Around 85% of institutions achieve reasonable performance expectations given the pandemic.

Table 2. Student satisfaction during the pandemic: probability the process is in control.

Institution	N	r	Pr($\mathit{\theta }\ge {\theta }_{\mathit{thresh}}$)
Abertay	82	64	0.878239
Aberystwyth	71	57	0.934999
Amity Global Education	10	10	0.973044
Anglia Ruskin	910	728	1
Arden	318	251	0.997359
Arts University Bournemouth	23	15	0.205911
Aston	622	454	0.700297
Backstage Academy	20	4	7.02E–07
Bangor	126	103	0.993212
Bath Spa	126	88	0.277455
BIMM Limited	77	60	0.864133
Birkbeck College	88	64	0.532588
Birmingham City	679	502	0.865035
Blackburn College	22	16	0.475062
Bloomsbury Institute	154	136	0.999999
Bournemouth	676	412	2.63E–10
BPP University	128	87	0.145985
Bradford College	21	11	0.023599
Brunel	407	236	6.28E–10
Bury College	12	9	0.515485
Canterbury Christ Church	137	99	0.505015
Cardiff Metropolitan	244	183	0.843905
Cardiff	313	228	0.616265
City College Norwich	27	21	0.705689
City, University of London	476	347	0.658268
Coventry	1678	1309	1
CP Training Services	12	10	0.750537
Croydon College	15	14	0.96233
De Montfort	656	459	0.120153
Edge Hill	174	124	0.395893
Edinburgh Napier	358	294	0.999995
Falmouth	68	53	0.847509
Farnborough College of Technology	13	7	0.067274
Fashion Retail Academy	118	84	0.399408
Glasgow Caledonian	527	427	0.999999
Global Banking School	90	82	0.999994
Goldsmiths’ College	116	74	0.024579
Gr?p Colegau NPTC Group of Colleges	15	15	0.994784
Harper Adams	54	39	0.479187
Hartpury	47	35	0.610195
Heriot-Watt	320	266	0.999998
Istituto Marangoni	85	69	0.969327
Kingston	285	202	0.323479
Leeds Beckett	766	597	0.999901
Leeds Trinity	69	41	0.010741
Liverpool Hope	81	60	0.634139
Liverpool John Moores	608	450	0.861281
London Metropolitan	234	192	0.999788
London School of Management Education	20	20	0.998991
London School of Science and Technology	200	180	1
London South Bank	429	283	0.00293
Loughborough College	12	9	0.515485
Loughborough	379	326	1
Manchester Metropolitan	1097	768	0.069824
Middlesex	394	268	0.038744
Newman	20	17	0.879755
Norwich University of the Arts	31	25	0.833282
Nottingham Trent	1082	768	0.222401
Oxford Brookes	476	328	0.064968
Pearson College	155	104	0.083243
QAHE	739	539	0.706881
Queen Margaret University, Edinburgh	88	69	0.901943
Queen Mary University, London	373	242	0.001256
Queen’s University, Belfast	365	303	0.999999
Regent’s University	198	158	0.993224
Richmond	39	32	0.90763
Roehampton	233	163	0.232679
Royal Holloway and Bedford New College	266	181	0.073038
RTC Education	15	12	0.695889
SAE Education	11	2	0.000109
Sheffield Hallam	715	558	0.999874
Solent	238	169	0.351722
Solihull College and University Centre	11	10	0.890019
South Eastern Regional College	13	12	0.93516
St Mary’s, Twickenham	83	70	0.994648
St. Piran’s School	19	16	0.85344
Staffordshire	133	81	0.002559
Swansea	376	308	0.999995
TEC Partnership	15	11	0.478772
Teesside	86	57	0.11124
London Institute of Banking and Finance	45	28	0.067803
LSE	221	179	0.998822
Robert Gordon	301	262	1
Royal Agricultural University	86	67	0.878557
Bath	362	311	1
Birmingham	316	218	0.112306
Bolton	60	54	0.999486
Bradford	182	135	0.728415
Buckingham	59	50	0.985953
Chichester	46	36	0.804509
Cumbria	91	58	0.038731
East Anglia	355	288	0.999959
Essex	496	342	0.063838
Huddersfield	329	227	0.108617
Hull	215	163	0.88783
Kent	482	366	0.972591
Lancaster	551	479	1
University of Law	30	23	0.672687
Leeds	590	443	0.951503
Leicester	233	149	0.003433
Liverpool	633	513	1
Manchester	729	532	0.713839
Reading	489	372	0.977561
Sheffield	259	166	0.002577
Surrey	723	542	0.961636
Warwick	471	414	1
West London	422	338	0.999926
Westminster	637	452	0.271285
UCFB College of Football Business	272	171	0.000497
UCK Limited	18	16	0.933323
University Centre Peterborough	18	9	0.020221
University Centre Quayside	10	10	0.973044
University College Birmingham	175	140	0.991289
University College London	277	238	1
University College of Estate Management	10	9	0.857732
University for the Creative Arts	106	90	0.99888
Aberdeen	143	114	0.980181
Bedfordshire	210	139	0.030169
Brighton	406	244	1.11E–07
Bristol	211	156	0.720032
UCLAN	232	169	0.59628
Chester	200	156	0.970127
Derby	269	202	0.864381
Dundee	119	94	0.953228
Durham	329	273	0.999998
East London	200	154	0.939917
Edinburgh	238	171	0.462541
Exeter	478	359	0.932875
Glasgow	201	157	0.972815
Gloucestershire	164	113	0.178517
Greenwich	558	402	0.497993
Hertfordshire	634	456	0.472766
Keele	155	124	0.987136
Lincoln	428	347	0.999992
Newcastle	531	356	0.005749
Northampton	228	141	0.000411
Northumbria	551	325	2.47E–11
Nottingham	369	273	0.79369
Oxford	47	42	0.997036
Plymouth	279	187	0.031823
Portsmouth	633	462	0.701172
Salford	282	197	0.202759
South Wales	124	91	0.612254
Southampton	258	196	0.918801
St Andrews	78	69	0.999675
Stirling	206	173	0.999966
Strathclyde	424	373	1
Suffolk	30	18	0.06699
Sunderland	473	412	1
Sussex	598	454	0.983753
University of the Arts, London	280	168	6.96E–06
University of Highlands and Islands	74	59	0.924694
UWE, Bristol	815	668	1
West of Scotland	329	224	0.055564
Ulster	656	544	1
UOW Trinity Saint David	300	234	0.990039
Winchester	151	91	0.000842
Wolverhampton	172	126	0.624453
Worcester	87	60	0.245672
York	175	133	0.87336
West Suffolk College	13	7	0.067274
Wrexham Glyndwr	23	14	0.105715
York St John	78	53	0.197055

IV. Conclusions

The pandemic results in an estimated 10% reduction in student satisfaction (Karadag 2021). Much student dissatisfaction is likely unavoidable (Kerzic et al. 2021; Park and Koo 2022). The AAUP has itself intervened on SET usage during the pandemic. Using NSS data we estimate around 85% of institutions meet reasonable performance expectations. This figure is probably an under-estimate given the need to analyse SET sensitively (Deem and Baird 2020). These high-performance levels emphasize the need for a kinder evaluation of pandemic-era SET.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

¹ A reasonable starting assumption pre-pandemic this is likely further enhanced by pandemic-era social restrictions. Generalized linear mixed models can resolve correlations between survey responses (Brint and Fry 2021).