Setting defensible minimum-stations-passed standards in OSCE-type assessments

Figures & data

Table 1. Descriptive statistics for the key facets of the PLAB2 exam (Nov. 2016 to March 2020).

Facet	Number of unique levels (i.e. values) in data	Occurrence in data			Description
		Median (quartiles)	Mean	Total number of interactions
Candidates	18,490	18 (18, 18)	17.8^a	329,348	There are around 18,500 unique candidates in this data, each assessed in a single PLAB2 exam. Typically, candidates are assessed at 18 stations. There are 329,348 (=18,490 × 17.8) rows of data in total—one for each candidate/station interaction across 442 separate PLAB2 exams.
Examiners	862	6 (3, 13)	11.1	9,586	There are just under 900 unique examiners in this data. Typically, examiners are present in six PLAB2 exams in the dataset. In total, there are 9,586 (=862 × 11.1) examiner/station combinations.
Stations	390	17 (8, 29)	20.2	7,878	There are 390 unique stations in this data. Typically, stations are administered in 17 separate exams in this dataset. In total, there are 7,878 (=390 × 20.2) separate station administrations across the data.

^a Occasionally, a station might be removed from the examination due to poor psychometric performance. There are 372 of the 442 exams with the full 18 stations (84%).

Table 2. Outline of main modelling approach for station-level domain scores and pass/fail decisions.

In pseudo-code, the mixed model equation for domain scores is:

DOMAIN_SCORE ∼ 1 + (1 | CANDIDATE) + (1 | EXAMINER) + (1 | STATION)

(this uses the lmer function in the R package lme4)

For pass/fail decisions, we similarly have:

PASS_FAIL ∼ 1 + (1 | CANDIDATE) + (1 | EXAMINER) + (1 | STATION)

(this use the glmer function in the R package lme4 and the option: family = binomial (link = ‘logit’)

Table 3. Brief overview of each potential method of setting MNSP.

MNSP method	Description of standard	Exemplification/further details
M1	Equal to exam-level cut-score percentage	In (say) an 18 station OSCE, the cut score, set by borderline regression (BRM), is (say) 55%. The MNSP is then 18 × 0.55 = 9.9, i.e. 10 stations with rounding
M2	Set at the typical number of stations passed by a ‘borderline’ group (at exam level)	In a cohort of 50 students, 5 are identified as borderline—being ‘near’ the exam cut-score (set by BRM, for example and using the classical standard error of measurement (Hays et al. 2008) to provide an appropriate borderline interval). The median (or mean) number of stations passed by this group is the MNSP.
M3	Calculated using logistic regression at station level and aggregated to exam level	The modelled proportion of borderline candidates passing each station is calculated using logistic regression of passing/failing by global grade. These proportions are added across the exam to give the MNSP (rounded as appropriate).
M4	Calculated via regression of no. of stations passed on total score	The line of best fit is calculated having plotted the total number of stations passed by each candidate (y) against total score in the exam (x). At the exam cut-score (e.g. calculated by BRM), the predicted number of stations passed is the MNSP (rounded as appropriate).

Table 4. Variance components for domain scores and pass/fail decision in a station.

Facet	Total domain score in station		Station outcome (pass/fail)
	Variance	Percentage	Variance	Percentage
Candidate	0.701	11.6	0.482	10.6
Examiner	0.961	16.0	0.495	10.9
Station	0.347	5.8	0.279	6.1
Residual	4.011	66.6	3.290^a	72.4
Total	6.019	100.0	4.545	100.0

^a According to common practice, this residual variance is fixed at π²/3 (Hox et al. 2017, p. 121).

Table 5. Overall reliability/SEM estimates for an 18 station PLAB2 OSCE.

Index of reliability	Exam-level candidate outcome
	Total domain score	Total number of stations passed
Generalizability coefficient (G)	0.759	0.725
Relative SEM	0.472 (3.9%)	0.428^a
Dependability coefficient (Phi)	0.703	0.681
Absolute SEM	0.544 (4.5%)	0.475^5

^a Due to rescaling effects in mixed effects logistic regression models (Fielding 2004), creating interpretable SEM estimates as a percentage in these mixed models is a challenge . These ‘raw’ SEM estimates are included only for completeness.

Figure 1. Calculation of the M4 standard for a particular examination with 35 candidates.

Table 6. Candidate level pass/fail breakdown across all data.

Pass/fail profile	Frequency	Percent
Fail both BRM and M4 standard	2,397	13.0
Fail BRM standard only	463	2.5
Fail M4 standard only	978	5.3
Pass both BRM and M4 standards	14,652	79.2
Total	18,490	100

Figure 2. The M4 minimum station standard (18 station PLAB2 exams only, rounded) (n = 372).

Table A1. Breakdown of problems occurring with calculating M3 in PLAB2 station level data.

Problem with calculating M3 standard at station level	Number of stations	Percentage of stations
All candidates pass^a – this means logistic regression cannot be applied as there is only one level of the outcome variable being predicted (rather than two).	202	2.6
Complete separation – this means that all candidates below a certain grade fail, and all above that grade pass. This means that the usual estimation method for logistic regression fails (Albert and Anderson 1984).	487	6.2
No problem – the M3 standard can be calculated in the station.	7,189	91.3
Total	7,878	100

^a Note that it is possible for all candidates to fail, and the same problem would occur. This problem was not observed in the PLAB2 data, but might in other contexts.

Hays R, Gupta TS, Veitch J. 2008. The practical value of the standard error of measurement in borderline pass/fail decisions. Med Educ. 42(8):810–815.

 PubMed Web of Science ®Google Scholar

Hox JJ, Moerbeek M, Schoot R v d 2017. Multilevel analysis: techniques and applications, Third Edition. 3rd edition. New York, NY: Routledge.

 Google Scholar

Fielding A. 2004. Scaling for residual variance components of ordered category responses in generalised linear mixed multilevel models. Qual Quant. 38(4):425–433.

 Google Scholar

Albert A, Anderson JA. 1984. On the existence of maximum likelihood estimates in logistic regression models. Biometrika. 71(1):1–10.

 Web of Science ®Google Scholar