Considering medical students’ perception, concerns and needs for e-exam during COVID-19: a promising approach to improve subject specific e-exams

Figures & data

Figure 1. Study timeline and process.

Table 1. Overall scores of the 6 dimensions of the SPEAQ for pre-exam (n = 317) and post-exam (n = 34) questionnaires.

Dimension	N of items		Median (IQR)*		Test-statistic U P**	Cronbach’s alpha***
	pre-exam	evaluation	pre-exam	evaluation	-	pre-exam	evaluation
Affective factors	7	7	2.29 (1.71–3.14)	2.92 (1.96–3.64)	4,127.50.025	.864	.849
Validity	5	5	3.20 (2.60–3.60)	3.40 (2.80–4.00)	4,078.00.019	.502	.497
Practicality	6	6	2.83 (2.33–3.50)	3.79 (3.33–4.21)	2,249.50<.001	.757	.612
Reliability	3	3	2.00 (1.33–2.67)	3.00 (2.33–4.00)	2,386.50<.001	.520	.597
Security	5	5	3.00 (2.60–3.69)	3.60 (3.15–4.00)	3,243.50<.001	.692	.437
Learning and teaching	4	3****	2.50 (1.75–3.25)	3.00 (3.00–3.75)	4,530.00.126	.801	.792

* Ranging from 1.0 (negative rated) – 5.0 (positive rated) with a neutral mid-point of 3.0 (3.00 was considered neutral, <3.00 negative, and >3.00 positive).

** p-value was calculated using Mann-Whitney U test, α = .05*** Internal consistency of the six dimensions has been measured with Cronbach’s coefficient alpha

**** Because the exam was conducted digitally and in-person, the item ‘E-exams provide opportunities for knowledge testing that would not be possible with in-person examinations’ was removed from the evaluation survey.

IQR, interquartile range

Table 2. Characteristics of participants of pre-exam and evaluation questionnaires.

	Pre-exam [n (%)]	Evaluation [n (%)]
Total	317 (100)	34 (100)
Female	232 (73.2)	27 (79.4)
Male	83 (26.2)	7 (20.6)
Other	2 (0.6)	0 (0)
18–20 years old	6 (1.9)	0 (0)
21–23 years old	174 (55.9)	14 (41.2)
24–26 years old	91 (28.7)	16 (47.1)
27–30 years old	23 (7.3)	3 (8.8)
> 30 years old	23 (7.3)	1 (2.9)
Clinical semester
1^st	21 (6.6)	0 (0)
2^nd	61 (19.2)	1 (2.9)
3^rd	21 (6.6)	3 (8.8)
4^th	78 (24.6)	3 (8.8)
5^th	50 (15.8)	22 (64.7)
6^th	86 (27.1)	5 (14.7)
Exam format
E-exam*	94 (29.7)	NA
In-person paper-based**	223 (70.3)
Mock (TUM)exam*** participation
Yes	101 (31.9)	31 (91.2)
No	216 (68.1)	3 (8.8)
Information session participation	NA
Yes		12 (35.3)
No		22 (64.7)
Manual ‘e-exams with TUMexam’	NA
Yes		19 (55.9)
No		15 (44.1)
E-exam experience
0	211 (66.6)	0 (0)
1–2	88 (27.8)	3 (8.8)
3-4	12 (3.8)	8 (23.5)
>4	6 (1.9)	23 (67.6)
Exam gerontology participation****
Yes	94 (29.7)	34 (100)
No	223 (70.3)	NA

NA, not applicable * E-exam is conducted off campus without supervision and with several digital submission options. ** In-person paper-based exam is held at the Technical University Munich with in-person proctoring. *** Pre-exam questionnaire asked about participation in any mock e-exam, while evaluation questionnaire specifically asked about participation in the mock e-exam on TUMexam. **** Participation in the exam for the cross-disciplinary subject gerontology in the summer term 2021.

Table 3. Overall ratings of the 30 indicators of the SPEAQ for pre-exam (n = 317) and evaluation (n = 34) questionnaires.

Dimension/Item	Median (IQR)*		Test-statistic U	P-value****
Affective factors	Pre-exam	Evaluation	-	-
E-exams** are more stressful for me than in-person exams***.	2.00 (1.00–3.00)	2.00 (1.00–4.25)	5,017.00	.494
I prefer to write in-person exams rather than e-exams.	1.00 (1.00–3.00)	3.00 (1.00–5.00)	3,205.50	<.001
E-exams and in-person exams trigger the same level of stress in me.	2.00 (2.00–4.00)	2.50 (2.00–3.25)	5,129.00	.635
I think e-exams should play a more important role at the School of Medicine.	2.00 (1.00–4.00)	3.00 (1.00–5.00)	4,245.00	.037
In e-exams I have difficulties concentrating on the questions.	3.00 (2.00–4.00)	3.00 (2.00–4.25)	5,063.00	.554
I prefer e-exams because I am used to working on the computer.	2.00 (1.00–3.00)	2.50 (1.75–4.00)	3,638.50	.001
With e-exams, I miss the social interaction with my fellow students.	2.00 (1.00–4.00)	2.50 (1.00–3.75)	5,150.00	.658
Validity
An e-exam is a suitable exam format for the lecture series gerontology.	3.00 (3.00–4.00)	4.00 (2.00–5.00)	4,272.50	.041
E-exams test not only my knowledge but also my IT skills.	4.00 (2.00–5.00)	3.00 (1.00–3.25)	3,550.50	<.001
E-exams are a contemporary form of examination.	4.00 (3.00–4.00)	4.00 (3.00–5.00)	4,371.00	.061
A too tight time limit is set for e-exams.	3.00 (2.00–3.00)	4.00 (2.00–5.00)	3,679.50	.002
The requirement level of the exam questions is much higher with the e-exam.	2.00 (2.00–3.00)	3.00 (2.00–5.00)	3,735.00	.002
Practicality
I think it’s positive that online exams use less paper.	4.00 (4.00–5.00)	5.00 (4.00–5.00)	4,532.00	.099
I am afraid that my performance in the e-exam will be affected by a poor internet connection.	1.00 (1.00–2.00)	5.00 (4.00–5.00)	505.50	<.001
I do not have the necessary equipment to write an e-exam.	3.00 (3.00–5.00)	5.00 (4.00–5.00)	4,002.50	.009
I do not have an appropriate environment off campus to write an e-exam.	3.00 (2.00–5.00)	5.00 (4.00–5.00)	3,877.00	.005
E-exams are easier for me to access than in-person exams.	2.00 (1.00–3.00)	3.00 (2.00–5.00)	3,792.00	.003
When different subjects host e-exams on different platforms, it adds to my burden as a student.	2.00 (1.00–3.00)	2.00 (1.00–3.00)	5,078.50	.556
Reliability
Technical malfunctions of the exam platform may affect the exam procedure.	1.00 (1.00–1.00)	5.00 (3.00–5.00)	643.50	<.001
Digital MC questions are more reliable than in-person exams because transcription errors are avoided.	3.00 (1.00–4.00)	3.00 (2.00–4.00)	5,225.00	.764
In-person exams are fairer than e-exams.	2.00 (1.00–3.00)	2.50 (1.00–3.25)	5,211.50	.774
Security
E-exams are just as secure as in-person exams.	3.00 (2.00–3.50)	3.00 (2.00–4.00)	4,334.00	.054
I have confidence in the data security of the audit software.	4.00 (3.00–5.00)	5.00 (4.00–5.00)	4,067.00	.014
It is easier to cheat in e-exams than in in-person exams.	2.00 (1.00–3.00)	2.00 (1.75–3.00)	5,039.50	.519
Hacker attacks are an acute and serious problem for TUMexam.	3.00 (3.00–4.00)	4.00 (3.00–5.00)	3,515.00	<.001
Username and password provide a sufficient level of security for e-exams.	3.00 (3.00–4.00)	5.00 (3.00–5.00)	3,438.00	<.001
Learning and teaching
E-exams provide opportunities for knowledge testing that would not be possible in in-person examinations.	2.00 (1.00–3.00)	NA*****	-	-
E-exams could support my learning process.	2.00 (1.00–3.00)	3.00 (1.00–4.00)	4,512.00	.108
E-exams are just a gimmick that do not help me learn.	3.00 (2.00–4.00)	3.50 (1.75–5.00)	4,552.50	.128
E-exams go hand in hand with online learning.	3.00 (2.00–4.00)	3.00 (1.00–4.00)	5,347.00	.940

* Ranging from 1.0 (negative rated) – 5.0 (positive rated) with a neutral mid-point of 3.0 (3.00 was considered neutral, <3.00 negative, and >3.00 positive).

** E-exam is conducted off campus without supervision and with several digital submission options.

*** In-person paper-based exam is held at the Technical University Munich with in-person proctoring.

**** P-value was calculated using Mann-Whitney U test, α = .05

***** Because the exam was carried out digitally and in-person, the item ‘E-exams provide opportunities for knowledge testing that would not be possible with in-person examinations.’ was removed from the post-exam survey.

IQR, interquartile rage

Table 4. Overview of the categories and subcategories of the open-ended questions in pre-exam and evaluation surveys. Pre-exam statements (n = 297, k = 936) were considered in the design of the remote e-exam (preparation) to assuage the concerns and fears of examinees. Evaluation statements (n = 34, k = 145).

	Category Subcategory	k*	Example
Pre-exam	Technical framework	499	-
	technical concerns	221	‘[…] fear of technical failure and to fail the exam as a consequence.’
	fears regarding internet connectivity	278	‘I feel uncomfortable when the functioning of my exam depends on my […] internet connection. This adds additional stress.’
	Overall exam framework	301	-
	lack of adequate exam atmosphere	199	‘stronger “exam feeling”, i.e., somehow more stress, so that I work more concentrated like at home.’
	lack of usual exam condition	102	‘If you have any questions, you can ask the instructor right away if the exam is on-campus.’
	Examination requirements	108	-
	unknown exam format	36	‘[…] the exam will take place in new unknown format, which causes uncertainty.’
	exam equivalence	72	‘The new system will be relaunched with questions that are tougher and will fail us.’
	Privacy concerns	9	‘Privacy issues with monitored exams, […] almost an invasion of privacy.’
	Lack of social integration	19	‘It’s nice to cross paths with fellow students at least during the exams if you don’t do that during the semester already.’
Evaluation	Advantages	83	-
	fairness of the online platform	18	‘In terms of platform, I found TUMExam super and VERY much fairer than EvaExam (other exam platform used), also with the unlimited access.’ ‘The option of being able to submit PDFs periodically for security purposes is very commendable.’
	independency of internet connection	20	‘A permanent Internet connection is no longer required (see TUMexam), because the worry of losing the Internet connection was still the most stressful.’
	overall organization	19	‘The preparation could not have been better.’ ‘I also found it very good that for the gerontology exam there was an alternative offer to write the exam in paper format at the university.’ ‘I found that it was excellent, online exam for president’
	inclusion of students needs	15	‘I liked that our needs were taken into consideration when designing the exam. It has also contributed to the motivation to take the e-exam.’
	exam equivalence	11	‘Questions were fair and not harder than in prior exams. Moreover, I think it was great to have some short answer questions.’
	Disadvantages	22	-
	writing interface (Macintosh operating system)	2	‘[…] difficulties with free text tasks. These were not saved (editing on Mac)’
	limited user experience	20	‘[…] elaborate processing, normal system of EvaExam (other system used) better to use.’
	Further needs	25	-
	e-exam training	9	‘Video recording of the information session regarding TUMExam would have been nice.’
	uniform exam platform	8	‘A uniform platform, […] unnecessary stress as a result and really avoidable.’
	further e-exams	8	‘I would appreciate further e-exams.’
	Change of mind regarding e-exams	15	‘I would choose the online exam again. Because it worked out great – against other fears.’ ‘Independence from the internet connection was great and took the initial concerns away. Accordingly, I would use this format again in the future.’

* test segments that were categorized

Data availability statement

The data that support the findings of this study are available from the corresponding author, SZ, upon reasonable request.