Post-examination interpretation of objective test data: Monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66

Figures & data

Table

Practice points

Health profession educators need to interpret test data using psychometric methods. EFA describes how and to what extent a group of items in a test are related to a set of latent constructs or factors. CFA confirms the modelled relationship between the assessed factors. Generalisability theory extends CTT allowing assessors to isolate and estimate multiple errors that are influencing the results of a test. IRT, including Rasch modelling, produces a variety of data displays, encapsulating both student and item properties that enable test developers to monitor and improve the quality of test questions.

Figure 1. Some shapes of distributions.

Figure 2. Two different distributions from two examinations.

Table 1.  Reliability statistics, simulated output from SPSS

Cronbach's alpha	Cronbach's alpha based on standardised items	Number of items
0.725	0.724	4

Table 2.  Item-total statistics

Question	Scale mean if item deleted	Scale variance if item deleted	Corrected item-total correlation	Cronbach's alpha if item deleted
1	1.700	1.04	0.818	0.475
2	1.800	1.06	0.712	0.536
3	1.700	1.046	0.818	0.475
4	2.00	1.86	−0.051	0.950

Table 3.  Reliability statistics, simulated output from SPSS (after removing Question 4)

Cronbach's alpha	Number of items
0.950	3

Table 4.  Item-total statistics (after removing Question 4)

Question	Scale mean if item deleted	Scale variance if item deleted	Corrected item-total corrections	Cronbach's alpha if item deleted
1	1.400	0.838	0.945	0.889
2	1.300	0.869	0.802	1.00
3	1.400	0.838	0.946	0.889

Table 5.  Rotated two factors with communalities (h²)

	Question 5 included			After removing Question 5
Question	Factor 1	Factor 2	h^2	Factor 1	Factor 2	h^2
9	0.92	−0.02	0.85	0.92	0.005	0.85
2	0.92	−0.02	0.85	0.92	0.005	0.85
6	0.81	0.21	0.71	0.80	0.24	0.71
10	0.79	−0.38	0.77	0.80	−0.36	0.77
4	0.79	−0.38	0.77	0.80	−0.36	0.77
1	0.73	0.36	0.69	0.72	0.38	0.68
3	0.69	0.16	0.50	0.69	0.18	0.51
5	−0.28	0.03	0.08
7	0.01	0.96	0.92	−0.0017	0.96	0.92
8	0.01	0.96	0.85	−0.0017	0.96	0.92
Percentage of variance explained by each factor	47.23	23.60	70.83	51.80	26.20	78.00

Figure 3. The two-factor model.

Table 6.  Simulated parameters revealed by the two-factor model

Question type	Question number	Intercept	Slope: history-taking	Slope: physical examination
History-taking	1	0.97	0.21	0.50
	2	0.85	0.24	0.32
	3	0.78	0.31	0.02
	4	0.45	−0.03	0.26
Physical examination	1	0.30	0.37	0.19
	2	0.64	0.26	0.27
	3	0.08	0.29	0.22
	4	0.39	0.32	0.16

Table 7.  Goodness-of-fit indices for the two-factor model

Model	χ^2	df	p	CFI	RMSEA
Total sample	33.5	34	0.49	0.97	0.02

Figure 4. Hypothetical scoring of 10 students by three examiners on three different OSCE stations.

Figure 5. Restructured data from Figure 4.

Table 8.  Results of variance components estimates

Source of variation	Variance component (s × e) design	Percentage variance
Student	10.144	40.00
Examiner	1.578	6.20
Student × examiner	13.656^a	53.80

Note: ^aResidual variance.

Table 9.  Results of variance components estimates

Source of variation (n)	Variance component s × e × i× sp × st design	Percentage of variance
Student (10)	0.313	15.04
Examiner (3)	0.060	2.88
Item (5)	0.033	1.59
SP (3)	0.000	0.00
Station (3)	0.000	0.00
Student × item	0.096	4.62
Student × examiner	0.341	16.37
Student × item × examiner	1.231	59.16
Item × examiner × station	0.007	0.34

Notes: Residual variances for interactions between facets equal to zero have not been displayed in the table. They have no influence on the test and are redundant.

Table 10.  A simulated 7-item test of anatomy ability

Student	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Item 7	FC^a	SA^b (θ)
1	1	0	0	0	1	0	1	0.43	−0.28
2	1	1	1	1	0	1	1	0.71	0.90
3	1	1	1	0	1	0	0	0.57	0.28
4	1	0	1	0	1	0	1	0.57	0.28
5	0	0	0	0	0	0	1	0.14	−1.82
6	1	0	1	1	1	0	0	0.57	0.28
7	1	0	0	1	1	0	0	0.43	−0.28
FC^a	0.85	0.28	0.57	0.43	0.71	0.14	0.57
ID^c (b)	−1.73	0.90	−0.28	0.28	−0.89	1.80	−0.28

Note: ^aFraction correct, ^bstudent ability and ^citem difficulty.

Table 11.  Estimates of the probability of answering the correct answer for item 1

Student	b	θ	p
1	−1.73	−0.28	0.81
2	−1.73	0.90	0.93
3	−1.73	0.28	0.88
4	−1.73	0.28	0.88
5	−1.73	−1.82	0.48
6	−1.73	0.28	0.88
7	−1.73	−0.28	0.81

Figure 6. ICC for item 1 from Table 10.

Figure 7. Examples of the ICC.

Figure 8. ISM; each '#' represents four students and ‘.’ represents one student.