Psychometric evaluation of a knowledge based examination using Rasch analysis: An illustrative guide: AMEE Guide No. 72

Figures & data

Table 1.  Ways in which psychometric methods can improve medical education assessments

1. Modifying and improving individual questions and individual stations continuously: aberrant questions and stations can be detected and then restructured or discarded.  2. Improving examination blueprinting by deciding on the number of stations, cases, standardised patients (SPs) and detecting sources of measurement error in cases/SPs; psychometric methods, such as the many-facets Rasch model, enables us to consider student ability, item difficulty, the difficulty of the rating categories and the severity of examiners and their interactions simultaneously.  3. Improving the practical organisation of examination (e.g. sites, SPs, examiners, marking procedures); do SPs represent patients in a standardized and consistent way? and are all sites fair for all students?  4. Improving and developing new approaches for analysing and interpreting exam data; using IRT faculty are able to obtain further information about the item difficulty and the student ability.  5. Improving the credibility of the competence-based pass mark; using cluster analysis and Rasch analysis.  6. Optimising duration of OSCE stations; if many students fail a given station, this could be due to the fact that they did not have enough time to demonstrate their performance.  7. Improving the validity and reliability of checklists, rating and global rating scales; the construct validity and the check-lists that are interpreted by examiners can be examined by the Rasch model.  8. Improving the reliability of total OSCE scores and knowledge-based tests; improving item discrimination indices, stations and questions or increasing the number of question and stations can increase reliability.  9. Improving inter-station reliability (the global rating scale versus checklists); using Cronbach's coefficient alpha faculty can estimate the reliability of the checklist score. In addition, it is possible to compare independent alpha coefficients of global rating scales and checklists using the Hakstian-Whalen test. 10. Evaluating and improving the internal structure of multiple choice questions and OSCE stations. (This helps to identify the domains that are being measured which are important in test score interpretation.) 11. Recognising, isolating and estimating measurement errors associated with students’ scores to gain a clearer picture for estimating the true score (e.g. the effect of examination sites on students’ scores); the use of Generalisabilty (G) theory and the Rasch Model. 12. Generalisability (G) theory; monitoring and improving the quality of an OSCE through aggregate analytical methods (i.e. G studies). 13. Providing useful information about inconsistent ratings; if a category in a checklist has not been used by examiners, this category can be combined with other categories. 14. Improving cases/SPs by detecting whether they are easy or difficult. 15. Mapping item difficulty to student ability using the Rasch model. This helps faculty to compare the range of student ability and item difficulty. 16. Improving the construct validity of a test by the Rasch model; PCAR will provide diagnostic information about the construct validity of the test. 17. Identifying guessing strategies using the IRT models. 18. Investigating the variability of students’ scores on different knowledge cases or stations. 19. Improving inter-rater reliability, particularly using Rasch Modelling 21. Using the Rasch model to reveal abnormal scoring patterns and to see whether the category responses employed in OSCE rating scales are being interpreted correctly. 22. Developing item banks which can be used in national assessment databases and for CAT.

Table

Practice points

Rasch analysis is a particular method used in IRT. IRT supersedes CTT, in that it takes into consideration the interaction between student ability and item difficulty. The characteristics of a test that fits the Rasch model can be identified, so that test developers can iteratively move towards the ‘perfect’ test. The ‘perfect’ test is one on which the distribution of student ability is perfectly mirrored by the distribution of item difficulty.

Table 2.  Rasch factor analysis of the whole test, cases 1 and 2

Test	No. of questions	Rasch factor (eigenvalues)	Factor 1 (1st contrast) (eigenvalues)
The whole test	43	19.4	2.0
Case 1	24	13.3	1.5
Case 2	19	9.7	1.7

Table 3.  Item difficulty, standard error and infit and outfit statistics in case 1

			Infit		Outfit
Question	Item difficulty logits^a	SE^b	MNSQ	ZSTD	MNSQ	ZSTD
Q1	1.86	0.12	1.00	0.10	0.97	−0.40
Q2	0.71	0.13	0.96	−0.87	0.91	−1.02
Q3	0.71	0.13	1.00	0.09	0.96	−0.48
Q4	−1.52	0.23	0.97	−0.12	0.76	−0.75
Q5	2.85	0.14	1.04	0.60	1.01	0.15
Q6	−0.48	0.16	1.02	0.25	1.02	0.16
Q7	1.56	0.12	1.03	0.60	1.05	0.75
Q8	2.01	0.12	1.02	0.49	1.05	0.68
Q9	−0.28	0.15	0.98	−0.19	0.87	−0.80
Q10	0.38	0.13	1.00	0.04	0.95	−0.48
Q11	−0.78	0.17	1.04	0.38	1.63	2.59
Q12	−0.97	0.19	1.03	0.25	0.96	−0.09
Q13	−1.28	0.21	0.97	−0.14	1.00	0.08
Q14	−2.1	0.29	1.01	0.11	0.87	−0.21
Q15	−3.1	0.45	1.00	0.14	0.82	−0.15
Q16	3.09	0.15	1.01	0.10	1.03	0.30
Q17	−0.15	0.15	1.04	0.57	1.20	1.34
Q18	−0.28	0.15	1.01	0.15	1.12	0.80
Q19	−1.08	0.19	1.01	0.15	1.46	1.71
Q20	−0.63	0.17	0.99	−0.03	0.93	−0.33
Q21	−0.13	0.15	0.99	−0.14	0.98	−0.06
Q22	−1.68	0.24	0.99	0.03	0.85	−0.35
Q23	−0.28	0.15	0.96	−0.49	0.82	−1.17
Q24	1.59	0.12	0.92	−1.85	0.90	−1.60
Mean	0.00	0.17	1.00	0.00	1.00	0.00
SD	1.52	0.07	0.03	0.50	0.19	0.90

Notes: ^aItem difficulty measured in logits (negative values indicate easier questions).

^bStandard error.

MNSQ: mean square (values between 0.70 and 1.30 are within acceptable limits for the Rasch model).

ZSTD: value of t-test (values between −2 and +2 are within acceptable limits for the Rasch model).

Figures in bold indicate questions outside Rasch model (Q11 and Q19).

Table 4.  Item difficulty, standard error and infit and outfit statistics in case 2

			Infit		Outfit
Question	Measure logits^a	SE^b	MNSQ	ZSTD	MNSQ	ZSDT
Q1	1.81	0.12	1.05	1.12	1.07	1.20
Q2	2.57	0.13	1.04	0.62	1.10	1.06
Q3	1.46	0.12	0.99	−0.32	0.98	−0.27
Q4	0.21	0.13	0.93	−1.13	0.91	−0.89
Q5	−4.05	0.71	0.99	0.21	0.36	−0.99
Q6	−0.06	0.14	1.05	0.63	1.19	1.56
Q7	0.80	0.12	0.96	−1.01	0.91	−1.42
Q8	−0.47	0.16	1.04	0.42	0.94	−0.31
Q9	2.23	0.13	1.01	0.13	1.06	0.76
Q10	0.94	0.12	1.02	0.43	1.06	0.97
Q11	−0.42	0.15	0.93	−0.73	0.87	−0.85
Q12	−2.63	0.36	0.98	0.06	0.54	−1.04
Q13	−1.83	0.25	0.95	−0.16	0.78	−0.60
Q14	−2.51	0.34	0.95	−0.07	0.56	−1.06
Q15	1.06	0.12	0.98	−0.51	0.96	−0.65
Q16	0.97	0.12	1.03	0.78	1.03	0.55
Q17	−1.11	0.19	0.98	−0.08	0.82	−0.77
Q18	−0.91	0.18	1.02	0.23	1.02	0.14
Q19	1.93	0.12	1.06	1.23	1.11	1.79
Mean	0.00	0.20	1.00	0.10	0.91	0.00
SD	1.77	0.14	0.04	0.6	0.21	1.00

Notes: ^aItem difficulty measured in logits (negative values indicate easier questions).

^bStandard error.

MNSQ: mean square (values between 0.70 and 1.30 are within acceptable limits for the Rasch model).

ZSTD: value of t-test (values between −2 and +2 are within acceptable limits for the Rasch model).

Figures in bold indicate questions outside Rasch model (Q5, Q12 and Q14).

Figure 1. Test information function. Note: r = test reliability.

Figure 2. Item–student maps Each # represents four students and each ‘.’ represents one to four students. The values on the left of each scale are logits. T = 2 standard deviations from the mean, S = 1 standard deviation from the man, M = mean.

Figure 3. Item difficulty invariance in cases 1 and 2, respectively.

Figure 4. ICC for questions 15, 16, 17 from Table 3.