A competency based selection procedure for Dutch postgraduate GP training: A pilot study on validity and reliability

Figures & data

Table 1. Domains, Millers’ pyramid levels and instruments of the current and the new competency based selection procedure.

	Domains	Millers’ level	Instrument; personal qualities, competencies and (overall) score (range)	Duration (in minutes)
Current	Personal qualities	Not applicable	Semi-structured interview (SSI)	30–45
			Motivation (1–3)
			Orientation on the job (1–3)
			Learning needs (1–3)
			Personal attributes (1–3)
			Overall score (1–3)^a
New procedure	Knowledge	‘Knows’	National general practice knowledge test (LHK) Score (0–100%)	150
	Cognitions on how to deal effectively with practical dilemmas	‘Knows how’	Situational Judgement Test (SJT) Score (0–100%)	60
	Self-reported competencies	‘Shows how’	Patterned behaviour descriptive interview (PBDI)	60
			Empathy (1–5)
			Collaborative skills (1–5)
			Coping with pressure (1–5)
			Respect (1–5)
			Self-care (1–)
			Reflection (1–5)
			Overall score (1–5)^b
	General clinical competencies	‘Shows how’	Simulated encounter (SIM)	30
			Medical expertise (1–5)
			Doctor-patient communication (1–5)
			Professionalism (1–5)
			Overall score (1–5)^c

^aAverage of the scores of the four qualities.

^bAverage of the scores of the six competencies.

^cAverage of the scores of the three competencies.

Table 2. Individual characteristics; difference, (δ, 95% confidence interval, CI) between study population and remaining candidates.

	Study population n = 47^a	Remaining candidates n = 76	Difference δ (95% CI)	All n = 123
Gender, % male	29.8	25.0	4.8 (−11.5–21.1)	26.8
Age in years, mean (SD)	28.4 (2.8)	29.4 (5.2)	−1.0 (−1.9–0.9)	29.0 (4.5)
Past performance, % < 1 year clinical experience	38.3	36.8	1.5 (−16.2–19.1)	37.4
Region of medical school, % north-western Europe	100	88.2	11.8 (4.6–19.1)	92.7
Times of application, % first time	83.0	78.9	4.4 (−13.1–21. 9)	80.5
Admitted, %	78.7	73.7	5.0 (−10.3–20.4)	75.6

^aNijmegen n = 23; Utrecht n = 24.

Table 3. Mean and overall scores (SD) of the instruments in the current (SSI) and the new procedure (LHK, SJT, PBDI, SIM) (n = 47).

Instruments (n items)	Min–max	Mean (SD)	Number with poor assessments, score ‘1’ or below threshold (%)
Semi-structured Interview SSI overall (4)	1.1–2,8	2.2 (0.3)	Not applicable
Motivation^a	1.0–3.0	2.2 (0.4)	4 (8.7%)
Orientation on the job^a	1.0–3.0	2.1 (0.4)	5 (10.9%)
Learning needs^a	1.0–3.0	2.2 (0.4)	3 (6.5%)
Personal attitude^a	1.0–3.0	2.1 (0.5)	5 (10.9%)
National general practice knowledge test LHK^a (114)	6.7–43.3	21.9 (8.7)	10 (21.7%)
Situational judgement test SJT (80)	75.1–91.0	83.8 (3.1)	6 (12.8%)
Patterned behaviour descriptive interview PBDI overall (6)	1.3–4.7	3.7 (0.5)	6 (12.8%)
Empathy	1–5	3.8 (0.9)	1 (2.1%)
Collaboration	1–5	3.7 (0.8)	1 (2.1%)
Coping with pressure^a	1–5	3.8 (0.8)	1 (2.2%)
Respect^b	1–5	3.6 (0.8)	2 (4.7%)
Self-care^c	1–5	3.5 (0.9)	2 (4.4%)
Reflection^d	2–5	3.6 (0.7)	0
Simulates encounter SIM overall (3)	1.3–4.7	2.9 (0.6)	12 (25.5%)
Medical expertise	1–4	2.4 (0.8)	7 (14.9%)
Doctor–patient communication	2–5	3.1 (0.7)	0
Professionalism	1–5	3.1 (0.7)	1 (2.1%)

^an = 46

^bn = 43

^cn = 45

^dn = 44

Table 4. Candidates’ evaluation scores of the instruments of the current and new procedure on 5 point scale, ranging from 1 (strongly disagree) to 5 (strongly agree) n = 46.

	Current	New procedure
	SSI	LHK n = 45	SJT	PBDI	SIM
Content instrument is relevant for postgraduate GP training mean (SD)	4.2 (0.6)	4.5 (0.7)^a	4.2 (0.7)	4.2 (0.6)	3.7 (0.8)^a
Instrument enables to show competence for postgraduate GP training mean (SD)	3.3 (0.8)	2.8 (0.9)^a	3.5 (0.9)	3.9 (0.8)^a	3.6 (0.7)
Instrument is fair to candidates % yes	63%	64%	96%^a	74%	78%^a

SSI, semi-structured interview; LHK, national general practice knowledge test; SJT, situational judgement test; PBDI, patterned behaviour descriptive interview; SIM, simulated encounter.

^aP < 0.05.