The modified essay question: Its exit from the exit examination?

Figures & data

Table

Practice points

MEQs are often included in examinations to test higher order cognitive skills. MCQs are often regarded as testing knowledge recall only. When measured against Bloom's taxonomy, MEQs developed for a higher education assessment primarily tested lower order cognitive skills such as recall of knowledge. MCQs in the same assessment were found to be more likely to test higher order cognitive skills such as analysis and management. Many items in the MEQ and MCQ assessment suffered from IWFs and those in the MEQ were deemed more significant. Marking guides developed for the MEQ assessment were inadequate and lead to inconsistencies in marking. The MEQ was deemed to be an unsuitable assessment instrument and has been replaced by material in the script concordance format.

Table 1.  Rating scale used to judge the rigour of the MCQs according to the presence of any IWFs

Rating	Conditions required to achieve rating
1	Pass the cover test and no IWFs
2	Pass the cover test and 1–2 IWFs
3	Cover test dubious and no IWFs
4	Fail the cover test and 1–2 IWFs
5	Fail the cover test and more than 2 IWFs

Table 2.  Rating scale used to judge the rigour of the MEQs according to the presence of any IWFs

Rating	Conditions required to achieve rating
0	No identified IWFs
1	Question is ambiguous or open to misinterpretation (writer expects a response which differs from what the question asks)
2	Clarity of question: grammatically unsound/use of vague language/imprecise terms/unexplained abbreviations/miscuing
3	Question does not promote synthesis by testing a logical progression of thought. Failure of continuity between the sections of a question
4	Failure at one point in the question may lead to failure in other parts
5	Double negatives in question
6	Unnecessary information in question
7	Cueing to other parts of question
8	Question covers content, which is not necessarily accepted or is contentious
9	Question is too broad in scope

Table 3.  Modified Bloom's taxonomy

Level I	Knowledge
	–recall of information
Level II	Comprehension and application
	–understanding and being able to interpret data
Level III	Problem-solving
	–use of knowledge and understanding in new circumstances

Table 4.  Rating scale used to judge the rigour of the marking scheme for MEQs according to the presence of any IWFs

Rating	Conditions required to achieve rating
0	No identified IWFs
1	Scheme is incomplete (i.e. does not include important or correct alternative answers)
2	Scheme is highly specific/does not allow for minor variability from ‘model response’
3	Scheme is difficult to apply (i.e. awards difficult fractions of marks or is unclear where marks are allocated)
4	Scheme is poorly weighted. Inconsistent, dubious ‘relevance’/application/difficulty. Inappropriate to level of student. Answer length is not proportional to the marks allocated
5	Marks are awarded for the same answer at multiple points in the question

Table 5.  Marking results for all markers compared with official marks (maximum 12 per question)

Question number (discipline)	Original marker	Marker 1 student	Marker 2 non-clinical	Marker 3 clinical	Marker 4 clinical
1 (Surgery)	9.8 ± 0.2	9.2 ± 0.1^1	9.5 ± 0.2	8.7 ± 0.1^1	8.2 ± 0.1^1
2 (Psychiatry)	8.5 ± 0.2	8.2 ± 0.2^1	7.5 ± 0.2^1	8.1 ± 0.2^1	8.3 ± 0.2^1
3 (Obstetrics)	6.3 ± 0.1	5.8 ± 0.1	6.2 ± 0.2	5.6 ± 0.2^1	6.1 ± 0.1
4 (Surgery)	7.5 ± 0.2	7.3 ± 0.5	6.7 ± 0.1	6.7 ± 0.2	7.0 ± 0.1
5 (General Practice)	5.0 ± 0.1	5.7 ± 0.1^1	6.2 ± 0.2^1	6.4 ± 0.1^1	6.1 ± 0.1^1
6 (Psychiatry)	8.0 ± 0.2	6.9 ± 0.2^1	6.8 ± 0.1^1	6.6 ± 0.1^1	7.6 ± 0.2^1
7 (Paediatrics)	7.3 ± 0.2	7.0 ± 0.2^1	6.7 ± 0.2^1	6.6 ± 0.2^1	7.2 ±v0.2
8 (Gynaecology)	8.4 ± 0.1	8.5 ± 0.1	8.5 ± 0.1	8.9 ± 0.2^1	8.3 ± 0.1
9 (Paediatrics)	6.7 ± 0.1	6.8 ± 0.1	6.8 ± 0.1	5.4 ± 0.1^1	6.0 ± 0.1^1
10 (General Practice)	5.7 ± 0.2	5.7 ± 0.2	5.5 ± 0.2	5.5 ± 0.2^1	5.9 ± 0.2^1
11 (Surgery)	6.5 ± 0.2	4.1 ± 0.1^1	4.4 ± 0.2^1	3.4 ± 0.1^1	5.2 ± 0.2^1
12 (Medicine)	9.7 ± 0.1	9.3 ± 0.1^1	9.5 ± 0.1^1	9.5 ± 0.1^1	9.5 ± 0.1
13 (Medicine)	7.1 ± 0.1	6.3 ± 0.1^1	7.2 ± 0.1	6.4 ± 0.1^1	7.0 ± 0.1
14 (Pathology)	5.9 ± 0.2	5.8 ± 0.2	5.8 ± 0.2	5.7 ± 0.2	6.7 ± 0.2^1
15 (Medicine)	7.9 ± 0.1	7.3 ± 0.1^1	6.8 ± 0.1^1	7.7 ± 0.1	6.9 ± 0.1^1
Total	110 ± 1	104 ± 1^1	104 ± 1^1	101 ± 1^1	106 ± 1^1
number of questions significantly different from original marker	n/a	10	7	12	9

Notes: Results show the mean and standard error. 1: Significant difference with official results as indicated by Tukey post hoc test on repeated ANOVA.

Table 6.  Modified Bloom's categorisation for MCQ and MEQ questions

Question type	MCQ (total 180)	MEQ (total 65 stages)
Bloom's level 1	84 (47%)	50 (77%)
Bloom's level 2	70 (39%)	10 (15.3%)
Bloom's level 3	26 (14%)	5 (7.7%)

Table 7.  Contingency table for Fisher's exact test

	Bloom's level 1	Bloom's level 2, 3	Item flawed	Item not flawed
MCQ	84	96	65	115
MEQ	50	15	41	24
p		<0.0001		=0.0002

Table 8.  IWFs for MCQs

Question type	MCQ (total 180)
No IWFs (level 1)	115
Pass the cover test and 1–2 IWFs (level 2)	27
Cover test dubious and no IWFs (level 3)	21
Fail the cover test and 1–2 IWFs (level 4)	15
Fail the cover test and more than 2 IWFs (level 5)	2
Not phrased as a question	58

Table 9.  Item writing and marking scheme flaws for MEQs

Question type	MEQ (total 65)
No identified IWFs (level 0)	33
Question is ambiguous or open to misinterpretation (writer expects a response which differs from what the question asks). (level 1)	11
Clarity of question: grammatically unsound/use of vague language/imprecise terms/unexplained abbreviations/miscuing (level 2)	3
Question does not promote synthesis by testing a logical progression of thought. Failure of continuity between the sections of a question. (level 3)	8
Failure at one point in the question may lead to failure in other parts (level 4)	3
Double negatives in question (level 5)	0
Unnecessary information in question (level 6)	1
Cueing to other parts of question (level 7)	10
Question covers content, which is not necessarily accepted or is contentious (level 8)	1
Question is too broad in scope (level 9)	3
Marking scheme flaws
No identified flaws (level 0)	26
Scheme is incomplete (i.e. does not include important or correct alternative answers) (level 1)	28
Scheme is highly specific/does not allow for minor variability from ‘model response’ (level 2)	3
Scheme is difficult to apply (i.e. awards difficult fractions of marks unclear where marks are allocated) (level 3)	14
Scheme is poorly weighted. Inconsistent, Dubious ‘relevance’/application/difficulty. Inappropriate to level of student. Answer length is not proportional to the marks allocated (level 4)	3 (always with other flaws)
Marks are awarded for the same answer at multiple points in the question (level 5)	2 (always with other flaws)

Table 10.  The effect of marking variations on determination of student grades

	Extra candidates <50% (n)	Critical (n)	Could be critical (n)	Not critical (n)	Unaltered (n)
Student	13	6	13	41	86
Non-clinician	9	5	9	42	90
Clinician A	15	8	13	61	64
Clinician B	7	2	7	38	99

Note: n, number of students in each category.

Table 11.  Percentage of experts who strongly agreed or agreed with statements regarding the validity and fidelity of the assessments

	MEQ	MCQ	OSCE	Overall
Validity	50	60	60	70
Fidelity	40	30	60	60

Table 12.  Reliability and correlation coefficients for the three components of the Year 5 exit examination

Exam (146 candidates)	Number of questions	Duration (h)	Cronbach alpha	Pearson correlation coefficient (referenced to OSCE)	Pearson correlation coefficient (referenced to MEQ)	Pearson correlation coefficient (referenced to MCQ)
OSCE	18	2.1	0.58	1	−0.2	−0.2
MEQ	15	3.0	0.84	−0.22	1	0.77
MCQ	180	3.0	0.84	−0.22	0.77	1