Selection for Dutch postgraduate GP training; time for improvement

Abstract

Background: In the Netherlands we select candidates for the postgraduate GP training by assessing personal qualities in interviews. Because of differences in the ratio of number of candidates and number of vacancies between the eight departments of GP training we questioned whether the risk of being rejected diverged amongst them.

Objective: The research question of this study was to which degree department of choice, candidates’ characteristics and qualities assessed during interviews explain admission into GP training.

Methods: A nationwide observational study was conducted of all candidates who applied for postgraduate GP training in 2009/ 2010. Application ratio per department, candidates’ characteristics (gender, age, region of medical school and times of application) and qualities (motivation, orientation on the job, personal attributes and learning needs) were collected. Outcome measures were admission to interview and admission to GP training.

Results: The study population addressed 542 candidates. Sixty three candidates were rejected on application letter (11.6%). So 479 candidates were admitted to the interview, of which 340 were admitted to the GP training (71%). Gender and region of medical school outside North West Europe were associated with admission to the interview. Department of choice had a strong association with admission in both stages (RR: 0.30 to 0.74; 0.20 to 0.79 respectively), while candidates’ qualities explained admission (RR: 1.09– 1.25) as well.

Conclusion:The influence of department of choice yields doubts about fairness of the procedure. So advantages and disadvantages of a national procedure are discussed as well as those of a competency based procedure.

Key words::

• Post graduate GP training
• Selection
• Fairness
• Europe
• General Practice

KEY MESSAGE(S)

• Interview assessments are the core of the selection procedure for the Dutch postgraduate GP training.

• The strong eff ect of department of choice yields doubts about fairness of the selection procedure.

• This result and EURACT recommendations for competency based curricula advocate a Dutch competency based selection procedure.

INTRODUCTION

Historically, medical doctors in European countries were allowed to work as a General Practitioner (GP) or family doctor after their formal registration as MD. With the emergence of postgraduate GP training in the latter decennia of the 20th century selection procedures were developed in order to improve the quality of physicians in primary care. Selection procedures vary substantially in Europe, however (Table 1). Some countries (e.g. Austria and Finland) check minimum criteria as formal registration as MD. Most countries, eg Iceland, Sweden and the Netherlands, carry out a selection procedure driven by disciplined based training; they aim to select best doctors by assessments of medical knowledge and personal qualities. (1) These personal qualities, considered suitable for completing the postgraduate GP training, are assessed by interviews.

Table I. European overview of entrance procedure for post graduate GP training. Based on correspondence with key persons of EURACT.^a

	Registration and Selection procedure
Registration Checking minimum criteria: MD	Discipline based assessment procedure		Competency based assessment procedure
No additional criteria	In case of sufficient vacations, all candidates are admitted	Threshold scores on knowledge and or personal qualities	Threshold scores on competencies (and knowledge tests)
Austria, Bosnia & Herzegovina, Croatia, Finland, Flanders, Greece, Montenegro ( >n2 yr experience as MD), Norway, Slovakia	Czech, Bulgaria, Latvia, Macedonia, Poland, Portugal, Serbia, Slovenia, Switzerland	Cyprus, Estonia, Hungary, Iceland, Ireland, Italy, the Netherlands, Romania, Spain, Sweden	Denmark, United Kingdom

^aUnknown or no GP certification: Albania, France, Germany, Lithuania, Wallonia.

In general, the reliability of interview methods to assess candidates’ personal qualities is moderate to sufficient with two or more well-trained interviewers (2). However, the predictive validity of personal interviews with a view on academic and clinical performance is equivocal (3–5). Rather recently, the UK has introduced a competency-based selection after an extended job- analysis. Six competencies were targeted for selection: empathy and sensitivity, communication skills, clinical expertise, problem solving, professional integrity and coping with pressure (6). In Denmark, a competency based selection procedure containing Multiple Mini Interviews (MMIs) has been introduced. MMIs apply principles of OSCE to the interview context, which have shown good predictive correlations with future performance (7,8).

In the Netherlands, the three-year postgraduate GP trainings follow national regulations on selection with personal quality assessments by interviews being the core of the selection procedure (9). However, there are differences in the number of candidates in comparison with the number of vacancies amongst the eight training departments. Therefore, we questioned whether such a decentralized selection procedure is fair to candidates. The aim of this study was to investigate to which degree the department of choice, and candidates’ individual characteristics and qualities explain admission to GP training. The study addressed routinely registered data; so the factual procedure was investigated.

METHODS

Design

A nationwide observational study, in cooperation with all eight departments, of all candidates who applied for the GP training was conducted in October 2009 in seven departments, and for the remaining department in April 2010. Candidates were registered centrally to avoid double applications; the selection procedure was carried out decentralized in the department of candidates’ choice. Four departments followed an informed consent procedure.

Selection procedure

First, the local selection committees decide which candidates are invited to the interview stage using criteria as mastery of the Dutch language and quality of motivation, expressed in their respective letters of application. Second, three members of the selection committee, staff member, GP trainer and GP trainee, independently assess candidates’ qualities as motivation, orientation on the job, personal attributes and learning needs with semi-structured interviews (30–45 min). Relevance of curriculum vitae and candidates’ own perception of being a future GP, are somewhat differently assessed by the departments. The final conclusion on admission into GP training is based on a consensus procedure with the selection committees considering all aspects.

Data collection

Number of candidates, number of vacancies per department and individual characteristics was derived from the national Dutch postgraduate GP training (Huisarts Opleiding Nederland). All data were clerically depersonalized before data processing.

Individual characteristics contain age when selected (in years), gender (male versus female), region of medical school (North West Europe versus elsewhere) and the number of times of application (first time versus second time or more). Candidates’ qualities (motivation, orientation on the job, personal attributes and learning needs) were assessed by the three members of the selection committee on ordinal scales with two extremes (poor/ insufficient and very good/excellent), varying from three points to 10 points at the respective departments. The three independent scores per quality were averaged.

Outcome measures were admission to the interview (stage 1) and admission to the GP training (stage 2).

Analysis

The associations between the determinants and admission to the interview and admission to the GP training were estimated by relative risks (RRs) with Log binomial models, followed by subgroup analysis (10). As there was a difference between the number of applicants and number of vacancies per department, we calculated the application ratio and included this as an offset in the model to correct these differences. The department with the lowest application ratio was used as a reference group. By using Z scores, results of different rating scales could be compared. All analyses were done in SAS (version 9.1).

RESULTS

In total, 597 candidates applied for the selection procedure and 375 (62.8%) were admitted to the postgraduate GP training (Table II). Nine per cent of the candidates (55/597) were not evaluable, mostly because of lack of informed consent. Therefore, the study population consisted of 542 candidates. One third of the candidates was male (n = 181, 33.4%). Mean age was 29.9 years (SD 5.2, minimum 22, maximum 52 years old); 506 (93.4%) followed their medical school in North West Europe and 392 (72.6%) applied for the first time. 63 were rejected on application letter (11.6%). Therefore, 479 candidates were admitted to the interview, of which 340 were eventually admitted to the GP training (71%).

Table II. Number of candidates, vacancies, admission and evaluable candidates, baseline characteristics of evaluable candidates per department.

Department	1	2	3	4	5	6	7	8	Total
Number of candidates	27	34	38	62	66	134	155	81	597
Number of vacancies	36	36	36	36	36	72	80	36	368
Ratio candidate/vacancy	0.75	0.94	1.03	1.72	1.83	1.86	1.91	2.25	1.62
Number admitted	23	27	28	44	46	80	87	40	375
Percentage admitted	85.2	79.4	73.7	71.0	69.7	59.7	56.1	49.4	62.8
Number evaluable	26^a	24^b	37^c	29^d	64^e	131^f	150^g	81	542
Male, n (%)	11 (42.3)	9 (37.5)	17 (45.9)	12 (41.4)	15 (23.4)	42 (32.1)	53 (35.3)	22 (27.2)	181 (33.4)
Age in years mean (SD)	32.0 (8.5)	32.1 (7.7)	29.8 (5.5)	28.2 (4.8)	28.8 (4.8)	30.3 (5.0)	29.9 (4.6)	29.0 (4.2)	29.9 (5.2)
Medical school NW-Europe, n (%)	25 (96.2)	19 (79.2)	33 (89.2)	27 (93.1)	59 (92.2)	125 (95.4)	140 (93.3)	78 (96.3)	506 (93.4)
First time of application, n (%)	19 (73.1)	12 (50.0)	30 (81.1)	22 (75.9)	54 (84.4)	84 (64.1)	108 (72.0)	63 (77.8)	392 (72.6)

^aWithdrawal during procedure: 1.

^bNo informed consent: 10.

^cNo informed consent: 1.

^dNo informed consent: 33.

^eWithdrawal during procedure: 2.

^fWithdrawal during procedure: 3.

^gMissing 3, no informed consent: 2.

In the first stage, 88% of the candidates were selected for the interview stage. Taking the department with the lowest application ratio as a reference, there was an independent association between the department of choice and admission to the interview, which means that the more candidates per vacancies, the less probability to be admitted to the interview stage (Table III). Male candidates and those who followed their medical education outside North West Europe had a smaller probability of admission to the interview too.

Table III Univariate and multivariate relative risks (RR (95% CI)) of being admitted to the interview stage (Stage 1) and of being admitted to the GP training (Stage 2).

	Admission to interview (stage 1) n = 542		Admission to GP training (stage 2) n = 479
	Univariate RR (95% CI)	Multivariate RR (95% CI)	Univariate RR (95% CI)	Multivariate RR (95% CI)
Department 1 (= ref.)^a	1.00	1.00	1.00	1.00
Department 2	0.69 (0.60–0.81)	0.74 (0.63–0.87)	0.77 (0.62–0.96)	0.79 (0.65–0.97)
Department 3	0.63 (0.57–0.71)	0.63 (0.57–0.70)	0.68 (0.56–0.83)	0.68 (0.56–0.81)
Department 4	0.38 (0.32–0.43)	0.36 (0.32–0.42)	0.43 (0.35–0.53)	0.45 (0.36–0.56)
Department 5	0.40 (0.38–0.41)	0.38 (0.35–0.41)	0.34 (0.28–0.42)	0.34 (0.29–0.41)
Department 6	0.33 (0.30–0.36)	0.32 (0.29–0.36)	0.34 (0.29–0.41)	0.34 (0.29–0.40)
Department 7	0.33 (0.31–0.36)	0.33 (0.30–0.36)	0.30 (0.25–0.36)	0.29 (0.25–0.35)
Department 8	0.32 (0.30–0.33)	0.30 (0.28–0.33)	0.20 (0.15–0.25)	0.20 (0.16–0.24)
Gender (male = ref.)	1.11 (1.01–1.21)	1.08 (1.01–1.16)	1.00 (0.87–1.16)	0.95 (0.86–1.06)
Age (in years)	0.98 (0.97–0.99)	0.99 (0.98–1.00)	0.98 (0.96–1.00)^a	1.00 (0.99–1.01)
Region medical school (NW Europe = ref.)	0.63 (0.45–0.88)	0.66 (0.48–0.90)	0.48 (0.24–0.95)	0.68 (0.37–1.27)
Times of application (first time = ref.)	0.94 (0.89–1.00)	0.96 (0.92–1.02)	0.94 (0.84–1.04)	1.00 (0.93–1.08)
Motivation			1.51 (1.39–1.63)	1.20 (1.10–1.31)
Orientation on the job			1.41 (1.31–1.52)	1.09 (1.02–1.17)
Personal attributes			1.54 (1.42–1.67)	1.25 (1.14–1.36)
Learning needs			1.49 (1.38–1.61)	1.09 (0.99–1.19)

^aref.: reference.

In the second stage, the probability of eventual admission was strongly related to the application ratio too. Candidates for departments 4, 5, 6, 7 and 8 had a far more low risk (RR; 0.20–0.45; Table III) to be admitted than those for the reference. In addition, higher ratings on motivation, orientation on the job and personal attributes explained independently eventual admission; these findings were the case for all eight departments.

DISCUSSION

Summary of main findings

Department of choice was a strong predictor for admission in both stages. Candidates’ qualities assessed during the interview explained eventual admission too. The impact of the differences in application ratio between the respective departments of choice questions fairness and public defensibility of the decentralized Dutch selection procedure.

Strengths and limitations of the study

As a result of strong collaboration between departments, it was possible to collect data of selection procedures in 2009/2010. Informed consent procedure at four departments has led to some missing data; selection bias might be possible. However, we assume low bias on observed associations, because results of departments with and without an informed consent procedure did not differ. Data on assessors were not available, so we could not check possible assessor bias.

Interpretation

Gender and region of medical school outside North West Europe was associated with admission to the interview. The association with gender is probably caused by the fact that women express their motivation better than men do. The impact of the region of medical school might be caused by insufficient mastery of Dutch language. Personal attributes, motivation and orientation on the job were determinants of eventual admission to the GP training, which is in line with the aim of the procedure. The department of choice is the strongest predictor for admission. This might be caused by the fact that the respective departments practice relative criteria partly based on the ratio between candidates and vacancies. So candidates who have been rejected in one department might have been admitted elsewhere, and vice versa. This latter result yields doubts about fairness of the current procedure.

Local versus national procedure

A decentralized procedure like the Dutch seems to cause inequality and unfairness to the candidates, which can be hardly prevented by training of local selected committees. A national procedure with assessors blinded for selection on behalf of their own department could resolve this problem. While fairness to candidates and public defensibility are important advantages of such a national selection procedure, there are disadvantages too. A national selection procedure will lead to less involvement of local staff and trainers in the selection of their own trainees. In addition, such a national selection procedure will yield fewer possibilities to develop local identity of departments by preference of type of candidates.

Discipline versus competency based

The reconstruction of a historically more discipline based Dutch GP training into a competency-based curriculum in 2005 and the doubtful predictive validity of discipline based selection procedures argue for an improvement of the procedure. Provisional results from the UK are promising in showing that trainees recruited by competency based methods performed better on key competencies after three months in practice than those recruited through traditional selection procedures (11). MMI's are chosen in Denmark to select trainees on clear competency based criteria. These are nowadays considered rather appropriate for such an aim (7,8).

The doubtful predictive validity of the current discipline based procedure and the promising results mentioned above advocate a competency-based selection procedure, which has several advantages. First, it gives the opportunity to assess candidates on relevant competencies. Second, incompetent candidates can be selected out on rather clear arguments. Last, competency based selection can provide an individual training/educational plan for admitted trainees. The individual educational plan can be assessed during the GP training, which is a good preparation for future monitoring of functioning as GP. How far competency based selection procedures is more time consuming and so more expensive than discipline based procedures depends on many factors such as costs of the baseline situation, number of instruments and number of assessors (12). Using machine-marked tests (like knowledge and situational judgement tests) for pre-selection might reduce costs.

Conclusion

Although poor performance and attrition rate of GP trainees in the Netherlands is relatively low, we have a public responsibility to perform a fair and defensible selection procedure (13). With a view on the results of this study, we would plead for a national procedure. In addition, with the reconstruction into a competency-based curriculum in 2005 and the promising results of competency-based selection, the Dutch GP training has to commit itself to a competency-based selection procedure based on critical GP competencies. The selection procedure has to be part of a more complex strategy to strengthen primary care and prevent wasting capacity of competent doctors.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.