The clinical skills assessment for international medical graduates in The Netherlands

Abstract

Aim: A need was felt to improve the quality of admission and licensing procedures for international medical graduates in The Netherlands.

Method: A clinical skills assessment was designed as part of a new procedure to realize a high-stakes, fair, transparent, and a time-limited path of admission for international medical graduates to the Dutch health care system. Additionally, it should provide a well-founded advice about length and content of additional medical training, should this be indicated by the outcome of the assessment.

Results: The clinical skills assessment procedure was developed as a Dutch variant of the “Step 2 Clinical Skills examination” of the Educational Commission for Foreign Medical Graduates (ECFMG) in collaboration with the United States National Board of Medical Examiners, which has a well-documented validity and reliability. The experience with the new procedure is yet limited, but enough to warrant a report.

Discussion: Worldwide, a number of countries have developed such high-stake assessment procedures, but they show little uniformity and transparency. By describing the design and development of our procedure, we do not pretend to set a standard, but we hope to contribute to more fair, accurate and uniform approaches for doctors moving from one country to another.

Introduction

Many western countries face considerable number of foreign doctors seeking certification as independent health care providers. Some countries partly rely on these international medical graduates (IMGs) to alleviate the shortage of locally trained doctors. The influx of foreign doctors into the national healthcare system, however, is often troubled. The level of competence and education does not always meet national standards. IMGs usually pass one or more tests before being granted certification and the ability to work unsupervised. However, only few countries have developed uniform and transparent tests. Assessment procedures are considered long and expensive and so is additional medical training. Many IMGs seek other employment, usually below their actual potential. In this way, much of the investments in their education and training, both economical and personal, are lost (Spike 2006; Srivastava 2008).

The situation in The Netherlands was, until the late nineties, also far from ideal. The Dutch assessment procedure consisted of verification of the diploma and, when this was considered unequal to the Dutch medical licensing diploma, additional training in a University Medical Centre (UMC) was prescribed. The length and content of this obligatory training period was determined by the UMC the candidate was allocated to, and usually no assessment of the applicant's actual medical competence was carried out. Between 1999 and 2002 the number of IMGs entering The Netherlands and applying for recognition of their diplomas increased significantly, from 180 to 400 per year, and medical schools as well as the government were increasingly unsatisfied with the procedure (CIBG-Brochure 2004).

Table

Practice points

Due to growing numbers of International Medical Graduates (IMGs) applying for recognition of their medical diplomas and increasing discontent with existing assessment procedures, the Dutch government ordered the creation of a new high stakes assessment procedure for IMGs. The new Dutch assessment procedure is time-limited, treats all candidates equally, is reliable and content-validated. It was launched into practice, December 1, 2005. After registration in one country of the European Economic Area (EEA), IMGs have access as an independent health care provider to all other countries of the EEA without further national assessment of their skills. Few countries within the EEA have a validated and reliable procedure for the assessment of IMGs seeking recognition and registration. This might lead to the undesirable situation that insufficiently qualified independent providers take a place in the health care system. By describing the design and development of our procedure, we do not pretend to set a standard, but rather intend to contribute to transparency and discussions in the field.

The time had come to develop a new procedure. This new procedure was to provide equal treatment for all medical graduates from outside the European Economical Area (EEA, European Union (EU) member states plus Norway, Iceland, Switzerland and Liechtenstein), as within the EEA there is an open labour market and the assessment of professional competence cannot be enforced. The new procedure should lead to a shorter track before admission to employment in the Dutch health care system and also provide a justified and well-founded advice for additional medical training in case the assessment showed the need for it, taking the medical professional background and postgraduate training of the applicants into account (Splinter et al. 2003).

Such a high-stake assessment procedure would probably not only serve as a screening of incoming doctors’ medical competence, but would most likely also help to improve their ability, as they will probably seriously prepare to pass these examinations (Newble & Jaeger 1984; Wilkinson et al. 2007).

The USA and Canada have developed robust and validated procedures for the assessment of (foreign) doctors on a national level (Cohen et al. 1988; Conn & Cody 1989; Sutnnick et al. 1994; Friedman Ben-David et al. 1999). Specifically, the USA procedures, carried out by the Educational Commission for Foreign Medical Graduates (ECFMG), served as an exemplary focus in the design of procedures that could be used in The Netherlands. The ECFMG has a long standing and well-known experience with the assessment of foreign medical graduates. The validity of the ECFMG assessment procedure is reflected in the predictive properties of these tests for the future practice of the candidates (Chambers et al. 2000; Boulet et al. 2002; Austin et al. 2003; Papadakis et al. 2004).

The Dutch assessment procedure was designed after carefully studying the ECFMG procedures. The reason to match ECFMG procedures was to contribute to an international standard and to enhance the acceptance of procedures. This is of particular importance as the EU has an open labour market. A medical diploma, recognized by one member state is, in most cases, automatically recognized by all other member states, due to European regulations. By describing the development of the clinical skills assessment, we hope to be followed by other countries. With the formation of a multi-national community of interest around fairness in the testing of IMGs, hopefully there will be more transparency in this field.

The Dutch assessment of medical competence of foreign medical graduates

The new Dutch assessment of medical competence of foreign medical graduates (DAMCFG) procedure consists of (a) a portfolio, in which education and work experience are explained, (b) tests of general skills necessary to work in a Dutch health care environment, including Dutch medical language proficiency, knowledge of the organization of Dutch health care and English reading proficiency and (c) a series of assessments of medical competence, including a computer-based assessment of the knowledge of basic and clinical sciences and a hands-on assessment of clinical skills (Table 1).

Table 1.  Steps in the admission of foreign medical graduates

Initial track under responsibility of Department of Health			University responsibility	Department of Health responsibility
STEP I	STEP IIA	STEP IIB	STEP III	STEP IV
• Initial diploma screening • Basic language assessment • Composition of a portfolio of past experience	Assessment of • Dutch medical language proficiency • Knowledge of Dutch Health care organization • English reading proficiency	Assessment of • Knowledge of basic sciences • Clinical knowledge • Clinical skills	• Supplementary medical training and assessment of professional attitude	• Licensing as MD • Residency training • Licensing as specialist

The general skills tests have been developed by the University Medical Centre of Groningen and are administered at the James Boswell Institute in Utrecht. The computer-based assessments have been developed and held in Maastricht. Depending on the number of applicants, the tests can be held every month. The clinical skills assessment procedure has been developed by the University Medical Centre of Nijmegen and is the focus of this article. It takes place at least once in every 3 months in the clinical skills centre in Nijmegen. Candidates must first pass the general skills tests (Step IIA) before they are allowed to proceed to the clinical tests (Step IIB).

Clinical skills assessment: Development of a blueprint

To assess clinical and communicative skills an Objective Structured Clinical Examination (OSCE) has been developed in analogy with the Step 2 Clinical Skills of the ECFMG examination, which has been proven to be valid and reliable. The OSCE consists of 10 stations, nine of which involve standardized patients. Candidates are asked to take a history and perform a physical examination as they would do in real practice. In the tenth station, mastery of specific procedures must be shown. The time per station is 20 min. After that the candidate has 10 min available to formulate a problem list and a differential diagnosis and select additional diagnostic procedures and therapy.

A structured procedure was carried out to establish the content validity of the OSCE. First, two established sources were consulted to generate a list of domains representing organ systems, disease categories and elements of professional behaviour. One is the so-called Inter-university Progress Test, originally designed at Maastricht University, now applied at five Dutch medical schools, and intended to cover the complete range of relevant objectives of undergraduate medical training (Verhoeven et al. 2002). The second source was the United States Medical Licensing Examination Step 2 Clinical Knowledge test, as used in ECFMG procedures (Swanson et al. 1987; Gary et al. 1997; Hallock & Kostis 2006; USMLE Clinical Knowledge Test 2008). This yielded a list of 20 domains (Table 2).

Table 2.  List of the 20 domains of the clinical skills assessment blueprint

Blood and lymph	Cardiovascular system
Digestive system	Endocrine and metabolic system
Health care prevention	Immunology
Infectious diseases	Mental disorders
Molecular and cellular aspects	Musculoskeletal system
Neoplasm	Nervous system
Personal and social aspects	Phases of life
Renal and urinary system	Reproductive system
Respiratory system	Science and methodology
Skin and subcutaneous tissue	Special senses

The next step was to constitute an expert panel to rank these domains. Thirty-three practicing physicians from 20 different disciplines, involved in the training of medical students and residents in academic as well as affiliated centres were selected. Those physicians were well equipped to judge the importance of clinical problems for this new assessment, because of their experience with the level of competency of newly graduated doctors in daily practice. All physicians were invited to select from the list of 20 domains, the 10 most important ones. In addition, they were asked to identify 10 patient problems or conditions from their own discipline that could be used as material to build cases. This generated a list of problems and conditions that was cross-validated against the so-called Dutch Blueprint of Objectives of undergraduate medical training. This document describes all requirements that graduated doctors should meet in terms of knowledge, skills and attitude and includes a list of clinical problems. This blueprint has been established by the Royal Dutch Medical Association and organizations of clinical physicians and University Medical Centers and was put into effect as a law in The Netherlands in 1994 and revised in 2001 (Schade & Sminia 1995; Blueprint of objectives of undergraduate medical education in the Netherlands 2001; Metz et al. 2001).

From the 20 domains of the list, 18 were identified as important to test, although some domains were rated as more important than others. This information was used to give weight to the domains. As a consequence, 18 domains are to be tested in the clinical skills assessment, although some domains will be tested more often than others. The 18 domains were linked to medical disciplines and put into a multidimensional matrix. This matrix contains the general competencies to be tested, the disciplines on which these have to be demonstrated, and the specific problems or conditions that ought to be assessed within these disciplines. This clinical skills blueprint has been used to select appropriate case samples for subsequent assessments. Given the high stakes nature of the OSCE and the need to securely guard its content, we cannot provide more detailed information about the blueprint here.

Constructing stations and training standardized patients

After designing the OSCE blueprint, a series of actual cases was constructed. Each case consisted of (a) the objectives of the assessment, (b) the patient characteristics and (c) the setting (e.g. family practice, emergency department). Next, the patient's history was meticulously described. Not only relevant elements of the chief complaint, but all normal and abnormal findings within the various systems were included. The pieces of information that could be presented directly, as well as information that is only to be revealed as answers to specific questions were defined. Expert clinical faculty members created, reviewed and repeatedly revised the cases. Finally, checklists and rating scales were constructed by clinical and educational experts and based on checklists and rating scales from the literature and examples from the UMC Nijmegen (Jacobs et al. 2004). For each case, separate checklists were developed to score history taking, physical examination and professional behaviour. To improve the quality of the items, the checklists were revised several times by various experts. Particularly, the formulation of the items was the subject of extensive discussion. The standardized patients score the performance of the candidate by using the checklists. Every encounter is recorded on tape and at random these are scored by clinicians. The written information (problem list, differential diagnosis and additional diagnostic and therapeutic procedures) is scored by a clinician on the basis of a constructed rating scale.

After the case material and checklists had been described in detail, the next phase was the training of the standardized patients. The approximate training time per standardized patient was 25 h per case. This included – among other things – reading the case together, role playing and education and training in observing the physical examination. Various practice sessions were organized with medical students. These sessions were recorded on tape, which were reviewed by standardized patients as well as by their trainers. During this training, emphasis was put on the provision of the correct information at the right moment to assure the standardization of standardized patients’ performance. Additional training in observing and rating candidates followed, mostly using the video tapes and new practice sessions. After this training, high levels of inter-rater agreement were reached (>90%) (Pelgrim et al. 2008) and formal assessment sessions with a reference group were then organized. The reference groups consisted of recently graduated medical students of the UMCs of Nijmegen, Maastricht and Utrecht. They had a maximum of 6 months of clinical experience after graduation.

Establishing a found advice per IMG

The station scores of the clinical skills assessment are derived from the checklists with total test scores being an aggregate of the individual station scores. To evaluate the candidates’ test scores, no definite criterion has yet been set. The absence of performance criteria is due to a lack of insight in the actual levels of difficulty of the test at this moment. Until sufficient information is collected to develop performance criteria, the scores of the IMGs are compared with the scores of a reference group. Scores less than one standard deviation below the mean of this reference group are considered to indicate deficiencies in the skills of the candidates. The clinical knowledge tests scores are valued in the same way.

The results of the clinical assessment and the information provided by the IMGs through their portfolio are weighted by a committee established by the Ministry of Healthcare. All members, except the chairman, are senior faculty from the eight Dutch University Medical Centers. They have been selected by their respective institutions because of their clinical and educational expertise. The chairman is a representative of the Ministry of Health Care, a physician with a longstanding career in the legislation and regulations concerning the certification of IMGs. The committee advises the Minister of Health Care whether or not a candidate needs additional training and how long this training should be. After each decision the Minister of Health Care takes, the Committee communicates this to the IMG.

First results

Due to the political pressure and adjustment of regulations, the influx of foreigners in the Netherlands, in particular asylum seekers, has shown a marked decrease in the first decade of the twenty-first century. Consequently, the number of IMGs applying for the new assessment procedure also decreased rather unexpectedly, leaving only small numbers available for our first evaluation. We decided that nevertheless communication of our procedure is worth while at this moment and we will report more extensively in the future. After an extensive preparation, the new assessment procedure was launched in practice in December 2005. Since then, a total number of 200 IMGs entered the assessment procedure. The first step in the assessment procedure, namely the general skills tests, was failed by 161 candidates, mainly because of insufficient mastery of the Dutch language, in terms of speaking, reading and writing. As a consequence, only 39 candidates took the assessment of medical competence (Step IIB, see Table 1) between April 2006 and October 2008.

One IMG was granted a license without further demands after a successful assessment. The results of nine were considered equal to those of the reference groups and those candidates got their license after a 12 weeks period of supervised working in health care. This period was used to assess their professional behaviour in daily practice. The remaining 29 candidates all received the decision that additional training at a University Medical Centre to acquire their license was necessary. Two candidates were to take one half a year of additional medical training, nine to take 1 year, 14 to take 2 years and one to take 3 years. One candidate had to take 6 years of additional training, which is a complete medical curriculum.

Discussion

A new high-stakes, transparent assessment procedure for the certification of foreign medical graduates in the Netherlands has been developed and introduced. With the description of the development of the clinical assessment procedures, an example is provided to show how such an assessment procedure can be conceived and put into practice.

The new Dutch assessment procedure was developed as a means to select those IMGs who do not meet the standards of medical practice in the Netherlands. The DAMCFG consists of a series of assessments on different elements of medical competencies. The basic and clinical knowledge domains are covered by a computer-based assessment. Clinical skills and professional behaviour are assessed through a 10-station OSCE. However, performance in real practice is not just the sum of knowledge, skills and attitude, but also depends on other factors, in particular time and efficiency (Rethans et al. 1991). In the 1970s, Senior and Lloyd already distinguished between the competence and performance of physicians. They defined competence as what a doctor is capable of doing and performance as what he or she actually does in day-to-day practice (Senior 1976; Lloyd 1979). Naturally, what we really want to know is the level of performance of the IMGs when working in the Netherlands. However, before actual patients are to be taking care of, those with a competency below a minimal threshold should be detected. With this goal in mind, a valid and reliable clinical skills test was to be developed. The most important step in developing such an assessment must be the construction of a clinically relevant blueprint, reflecting at best the actual clinical tasks and problems the candidates are required to be able to cope with according to national standards. Various studies have shown that the reliability of measurements of clinical competence is hindered by the fact that competence is content specific. Good performance in one area is a very poor predictor of performance in another area, even if these are closely related. Wide sampling across problems is required if an adequate level of content validity and reliability is to be achieved (Swanson et al. 1987; Van der Vleuten et al. 1991; Newble et al. 1994; Newble 2004; Hallock & Kostis 2006). This requires many cases and a very long testing time. The simplest solution to minimize the practical difficulties raised by case specificity is to combine the OSCE with other test formats that provide more efficient sampling of content, for example multiple choice questions. When all test components are based on the same blueprint, this seems to be a justifiable approach. Within the DAMFCG, we use this approach by combining the OSCE with a multiple choice test and an open-question clinical cases test.

There is a large body of evidence indicating that the use of standardized patients in high-stake assessments yields reliable results (Colliver et al. 1989; Van der Vleuten & Swanson 1990; Boulet et al. 1998; Williams et al. 1999; Chambers et al. 2000; Whelan et al. 2005).

Research on the USMLE clinical skills examinations, on which the DAMFCG test procedures is based, demonstrates that test scores of candidates correlate with their subsequent practice patterns (Chambers et al. 2000; Boulet et al. 2002). Quebec Licensing Examination scores, including a clinical skills examination, were shown to predict the future practice patterns of physicians. For example, in contrast to high performers, lower performers tended to prescribe more symptomatic and contra-indicated drugs (Norcini et al. 2002; Tamblyn et al. 2002). In another study, IMGs who had passed the clinical skills assessment of the ECFMG outperformed those United States medical graduates who had not passed such a test, showing higher levels of physical examination skills (Boulet et al. 2002). External validity is of the greatest importance for licensing examinations. It is not only a quality measure for the test, but also the only way to predict the future performance of candidates. By studying the future prospects of the IMGs in the Netherlands, we try to determine the predictive value of our assessment.

With this new assessment, justified, transparent and adequate advice for additional training can be given or the lack of need for additional training can be determined. This is of great importance because it gives the IMGs the opportunity to start training, which in the end will lead to entrance to the healthcare system as an independent provider. This way, all previous education, training and practice is put into use again. From this, both society and the IMGs will profit.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

Marye J. Sonderen

MARYE SONDEREN is a resident in the Internal Medicine. Before her residency she coordinated the development of the clinical skills assessment.

Eddie Denessen

EDDIE DENESSEN is an assistant professor in the Department of Educational Sciences and member of the Behavioural Science Institute at Radboud University Nijmegen. His research interests include culture differences in education, teacher beliefs and teaching behaviour, assessment of teaching, research methods, and psychometrics.

Olle Th.J. Ten Cate

OLLE TH. TEN CATE is professor of Medical Education and director of the Center of Research & Development of Education, UMC Utrecht School of Medical Sciences.

Ted A.W. Splinter

T. SPLINTER is a medical oncologist, professor of medical education and chairman of the project group for the assessment of international medical graduates

Cornelis T. Postma

CORNELIS T. POSTMA is associate professor of medicine in the Department of Medicine of the University of Nijmegen Medical Centre. His main medical education interests are in the training of practical medical education and medical competence and in the field of clinical assessment.