Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA®)

References

• ALBANESE, M. (2000) The decline and fall of humanism in medical education, Medical Education, 34, pp. 596–597.
   Google Scholar
• ALLEN, P.F. & Loci, D. (1997) Do item weights matter? An assessment using the oral health impact profile, Community Dental Health, 14, pp. 133–138.
   Google Scholar
• BARZANSKY, B. & ETZEL, S.I. (2004) Educational programs in US medical schools, 2003-2004, Journal of the American Medical Association, 292, pp. 1025–1031.
   Google Scholar
• BECK, R.S., DAUGHTRIDGE, R. & SLOANE, P.D. (2002) Physician—patient communication in the primary care office: a systematic review, Journal of the American Board of Family Practice, 15, pp. 25–38.
   Google Scholar
• BEN DAVID, M.F., BOULET, J.R., BuRoick, W.P., Ziv, A., HAmELEToN, R.K. & GARY, N.E. (1997) Issues of validity and reliability concerning who scores the post-encounter patient-progress note, Academic Medicine, 72(10, Suppl. 1), S79—S81.
   Google Scholar
• BOULET, J.R., BEN DAVID, M.F., Ziv, A., BURDICK, W.P., CURTIS, M., PErrzmAN, S. & GARY, N.E. (1998a) Using standardized patients to assess the interpersonal skills of physicians, Academic Medicine, 73(10 Suppl.), pp. S94—S96.
   Google Scholar
• BOULET, J.R., McKiNLET, D.W., NORCINI, J.J. & WHELAN, G.P. (2002) Assessing the comparability of standardized patient and physician evaluations of clinical skills, Advances in Health Sciences Education, Theory & Practice, 7, pp. 85–97.
   Google Scholar
• BOULET, J.R., McKiNLEy, D.W., WHELAN, G.P. & HAmELEToN, R.K. (2003a) Quality assurance methods for performance-based assess-ments, Advances in Health Sciences Education, Theory & Practice, 8, pp. 27–47.
   Google Scholar
• BOULET, J.R., McKiNLEy, D.W., WHELAN, G.P. & HAmELEToN, R.K. (2003b) The effect of task exposure on repeat candidate scores in a high-stakes standardized patient assessment, Teaching and Learning in Medicine, 15, pp. 227–232.
   Google Scholar
• BOULET, J.R., REBBECCHI, T.A., DENToN, E.G., McKiNLEy, D.W. & WHELAN, G.P. (2004) Assessing the written communication skills of medical school graduates, Advances in Health Sciences Education, Theory & Practice, 9, pp. 47–60.
   Google Scholar
• BOULET, J.R., VAN ZANTEN, M., McKiNLEy, D.W. & GARY, N.E. (2001) Evaluating the spoken English proficiency of graduates of foreign medical schools, Medical Education, 35, pp. 767–773.
   Google Scholar
• BOULET, J.R., FRIEDMAN BEN-DAVID, M., HANIELEToN, R.K, BURDICK, W.P., ZET, A. & GARY, N.E. (1998b) An investigation of the sources of measurement error in the post-encounter written scores from standardized patient examinations, Advances in Health Sciences Education, Theory & Practice, 3, pp. 89–100.
   Google Scholar
• BOULET, J.R., FRIEDMAN BEN-DAVID, M., ZIV, A., BURDICK, W.P. & GARY, N.E. (2000) The use of holistic scoring for post-encounter written exercises, in: Melnick. D (Ed.) Proceedings of the Eighth Ottawa Conference of Medical Education and Assessment, pp. 254–260 (Philadelphia, National Board of Medical Examiners).
   Google Scholar
• BOULET, J.R. & McKiNLEy, D.W. (2005) Investigating gender-related construct-irrelevant components of scores on a written assessment exercise of a high-stakes certification assessment, Advances in Health Sciences Education, Theory & Practice, in press.
   Google Scholar
• BOULET, J., DE CHAMPLAIN, A. & McKiNLEy, D. (2003) Setting defensible performance standards on OSCEs and standardized patient examina-tions, Medical Teacher, 25, pp. 245–249.
   Google Scholar
• CEGALA, D.J. & LENZMEIER, B.S. (2002) Physician communication skills training: a review of theoretical backgrounds, objectives and skills, Medical Education, 36, pp. 1004–1016.
   Google Scholar
• COLUVER, J.A. (1995) Validation of standardized-patient assessment: a meaning for clinical competence, Academic Medicine, 70, pp. 1062–1064.
   Google Scholar
• CuNNiNoToN, J.P.W., NEVE IF, A.J. & NORMAN, G.R. (1997) The risks of thoroughness: reliability and validity of global ratings and checklists in an OSCE, Advances in Health Sciences Education, 1, pp. 227–233. ECFMG (1999) Clinical Skills Assessment (CSA) Candidate Orientation Manual, pp. 1–32 (Philadelphia, PA, ECFMG).
   Google Scholar
• EDUCATIONAL COMMISSION FOR FOREIGN MEDICAL GRADUATES, (2004) ECFMG Certification Fact Sheet (Philadelphia, PA, ECFMG).
   Google Scholar
• FEDERATION OF STATE MEDICAL BOARDS, I. & NATIONAL Bo ARD OF MEDICAL EXAMINERS (2003) 2004 USMLE Step 2 CS Content Description and General Information Booklet (Philadelphia, FSMB and NBME). BINKBINER, R.G., FLETCHER, E.A., Om, N. & KLASS, D.J. (1994) Question format and scoring methods for standardized patient interstation exercises, in: A.I. Rothman & R. Cohen (Eds) Proceedings of the Sixth Ottawa Conference on Medical Education, pp. 343–345 (Toronto, University of Toronto Bookstore).
   Google Scholar
• FRIEDMAN, M., SUTNICK, A.I., STILLMAN, P.L., NORCINI, J.J., ANDERSON, S.M., WILLIAMS, R.G., HENNING, G. & REEVES, M.J. (1991) The use of standardized patients to evaluate the spoken-English proficiency of foreign medical graduates, Academic Medicine, 66(9, Suppl.), pp. S61—S63.
   Google Scholar
• FRIEDMAN, M., SUTNICK, A.I., STILLMAN, P.L., REGAN, M.B. & NORCINI, J.J. (1993) The relationship of spoken-English proficiencies of foreign medical school graduates to their clinical competence, Academic Medicine, 68(10, Suppl.), pp. S1—S3.
   Google Scholar
• GIMPEL, J.R., BOULET, D.O. & EmucHEIII, A.M. (2003) Evaluating theclinical skills of osteopathic medical students, Journal of the American Osteopathic Association, 103, pp. 267–279.
   Google Scholar
• GORTER, S., RETHANS, J.J., SCHERPBIER, A., VAN DER HELUDE, D., HOUBEN, H., VAN DER VLEUTEN, C. & VAN DER LINDEN, L.S. (2000) Developing case-specific checklists for standardized-patient-based assessments in internal medicine: a review of the literature, Academic Medicine, 75, pp. 1130–1137.
   Google Scholar
• GOVAERTS, M.J., VAN DER VLEUTEN, C.P. & SCHUWIRTH, L.W. (2002) Optimising the reproducibility of a performance—based assessment test in midwifery education, Advances in Health Sciences Education, Theory & Practice, 7, pp. 133–145.
   Google Scholar
• HARDEN, R.M. & GLEESON, F.A. (1979) Assessment of clinical competence using an objective structured clinical examination (OSCE), Medical Education, 13, pp. 41–54.
   Google Scholar
• HARDEN, RM., STEVENSON, M., DOWNIE, W.W. & WILSON, G.U. (1975) Assessment of clinical competence using objective structured examina-tion, British Medical Journal, 1, pp. 447–451.
   Google Scholar
• LIAISON CommiTTEE ON MEDICAL EDUCATION (2003) Functions and Structure of a Medical School: standards for Accreditation of Medical Education Programs leading to the MD Degree (Washington, DC, Association of American Medical Colleges).
   Google Scholar
• LYONS, T.F. & BAYNE, B.C. (1975) The use of item importance weights in assessing physician performance with predetermined criteria indices, Medical Care, 13, pp. 432–439.
   Google Scholar
• MAZOR, K.M., OCKENE, J.K, ROGERS, J., CARLIN, M.M. & QuiRR, M.E. (2005) The relationship between checklist scores on a communication OSCE and analogue patients' perceptions of communication, Advances in Health Sciences Education, Theory & Practice, in press.
   Google Scholar
• MEDICAL COUNCIL of CANADA (2002) Qualifying Examination Part II, Information Pamphlet (Ottawa, Ontario, Canada, Medical Council of Canada).
   Google Scholar
• MELNICK, D.E., DILLoN, G.F. & SWANSON, D.B. (2002) Medical licensing examinations in the United States, Journal of Dental Education, 66, 595–599.
   Google Scholar
• NEWBLE, D.I. & SWANSON, D.B. (1988) Psychometric characteristics of the objective structured clinical examination, Medical Education, 22, pp. 325–334.
   Google Scholar
• NORCINI, J. & BOULET, J. (2003) Methodological issues in the use of standardized patients for assessment, Teaching and Learning in Medicine, 15, pp. 293–297.
   Google Scholar
• NORCINI, J.J., STILLMAN, P.L., SUTNICK, A.I., REGAN, M.B., HALEY, H.-L.A., WILLIAMS, R.G. & FRIEDMAN, M. (1993) Scoring and standard setting with standardized patients, Evaluation and the Health Professions, 16, pp. 322–332.
   Google Scholar
• NORMAN, G.R., VAN DER VLEUTEN, C.P. & DE GRAAFF, E. (1991) Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability, Medical Education, 25, pp. 119–126.
   Google Scholar
• ONG, L.M., DE HAEs, J.C., Hoos, A.M. & LAMMES, F.B. (1995) Doctor—patient communication: a review of the literature, Social Science Medicine, 40, pp. 903–918.
   Web of Science ®Google Scholar
• BANGARO, L.N., WORTH-DICKSTEIN, H., MACMILLAN, M.K., KLASS, D.J. & SHATZER, J.H. (1997) Performance of standardized examinees in a standardized-patient examination of clinical skills, Academic Medicine, 72, pp. 1008–1011.
   Google Scholar
• ROSS, K.V. & ROTHMAN, A.I. (1985) The assessment of clinical skills in general medical residents—comparison of the objective structured clinical examination to a conventional oral examination, Annals RCPSC, 18, pp. 235–238.
   Google Scholar
• SLAIW, S.C. & BouLET, J.R. (2001) Predicting holistic ratings of written performance assessments from analytic scoring, Advances in Health Sciences Education, Theory & Practice, 6, pp. 103–119.
   Google Scholar
• STILLMAN, P.L, REGAN, M.B., HALEY, H.L., NORCINI, FRIEDMAN, M. & Sul-mac, A.I. (1992) The use of a patient note to evaluate clinical skills of first-year residents who are graduates of foreign medical schools, Academic Medicine, 67(10 Suppl.), pp. S57—S59.
   Google Scholar
• STILLMAN, P.L., BROWN, D.R., REDFIELD, D.L. & SABERS, D.L. (1977) Construct validation of the Arizona Clinical Interview Rating Scale, Educational and Psychological Measurement, 37, pp. 1031-1039. STREINER, D.L., GOLDBERG, J.O. & MILLER, H.R. (1993) MCMI-II item weights: their lack of effectiveness, Journal of Personality Assessment, 60, pp. 471–476.
   Google Scholar
• SUTNICK, A.I., FRIEDMAN, M., WILSON, M.P., RAZIN, S., SPERLING, 0., GurmAN, D. & Gucx, S. (1995) Use of ECFMG certifying examina-tions for national comparisons: performance of graduates of Israeli medical schools in basic and clinical science examinations, Israel journal of Medical Science, 31, pp. 250–254.
   Google Scholar
• SUTNICK, A.I., STILLMAN, P.L., NORCINI, H., FRIEDMAN, M., RAN, M.B., WILLIAMS, R.G., KACHUR, E.K, HAGGERTY, M.A. & Wnsox, M.P. (1993) ECFMG assessment of clinical competence of graduates of foreign medical schools. Educational Commission for Foreign Medical Graduates, Journal of the American Medical Association, 270, pp. 1041–1045.
   Google Scholar
• SUL-MAC, A.I. & Wnsox M.P. (1994) ECFMG approach to global assessment of clinical competence. All/ISE Newsletter, 8–10.
   Google Scholar
• SWANSON, D.B., CLAUSER, B.E. & CASE, S.M. (1999) Clinical skills assessment with standardized patients in high-stakes tests: a framework for thinking about score precision, equating, and security, Advances in Health Sciences Education, Theory & Practice, 4, pp. 67–106.
   Google Scholar
• SWANSON, D.B. & STILLMAN, P.L. (1990) Use of standardized patients for teaching and assessing clinical skills, Evaluation and the Health Professions, 13, pp. 79–103.
   Google Scholar
• TOMBLESON, P., Fox, R.A. & DACRE, J.A. (2000) Defining the content for the objective structured clinical examination component of the professional and linguistic assessments board examination: develop-ment of a blueprint, Medical Education, 34, pp. 566–572.
   Google Scholar
• VAN DER VLEUTEN, C.P., NORMAN, G.R. & DE GRAAFF, E. (1991) Pitfalls in the pursuit of objectivity: issues of reliability, Medical Education, 25, pp. 110–118.
   Google Scholar
• Vu, N.V. & BARROWS, H.S. (1994) Use of standardized patients in clinical assessments: Recent developments and measurement findings, Educational Researcher, 23, pp. 23–30.
   Google Scholar
• Vu, N.V., BARROWS, H.S., MARCY, M.L., VERHULST, S.J., COLLIVER, J.A. & TRAVIS, T. (1992) Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine, Academic Medicine, 67, pp. 42–50.
   Google Scholar
• Vu, N.V., MARCY, M.L., BARNHART, A.J., COLLIVER, J.A., HENKLE, J.Q., HODGSON, K., SCHRAGE, J.P. & TRAVIS, T.A. (1994) Further evidence of construct validity of standardized patient—based performance examina-tions, Teaching & Learning in Medicine, 6, pp. 255–259.
   Google Scholar
• Zw, A., BEN DAVID, M.F., Sumicx, A.I. & GARY, N.E. (1998) Lessons learned from six years of international administrations of the ECFMG's SP-based clinical skills assessment, Academic Medicine, 73, pp. 84–91.
   Google Scholar