BEME systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practice

References

• Accreditation Council for Graduate Medical Education (ACGME). Outcome Project: General Competencies. 2000, [Online]. Available at: http://www.acqme.org/Outcome/
   Google Scholar
• Alexander GL, Davis WK, Yan AC, Fantone JC. Following medical school graduates into practice: residency directors’ assessments after the first year of residency. Academic Medicine 2000; 75: S1517
   Web of Science ®Google Scholar
• Amos DE, Massagli TL. Medical school achievements as predictors of performance in a physical medicine and rehabilitation residency. Academic Medicine 1996; 71: 678–680
   PubMed Web of Science ®Google Scholar
• Arnold L, Willoughby TL. The empirical association between student and resident physician performances. Academic Medicine 1993; 68: S35–S40
   PubMed Web of Science ®Google Scholar
• Bell JG, Kanellitsas I, Shaffer L. Selection of obstetrics and gynecology residents on the basis of medical school performance. Am J Obstet Gynecol 2002; 186: 1091–1094
   PubMed Web of Science ®Google Scholar
• Best Evidence Medical Education, Centre for Medical Education, Dundee, UK. 2002, [Online]. Available at: http://www.bemecollaboration.org/
   Google Scholar
• Blacklow RS, Goepp CE, Hojat M. Further psychometric evaluations of a class–ranking model as a predictor of graduates’ clinical competence in the first year of residency. Academic Medicine 1993; 68: 295–297
   PubMed Web of Science ®Google Scholar
• Borowitz SM, Saulsbury FT, Wilson WG. Information collected during the residency match process does not predict clinical performance. Archives of Pediatric and Adolescent Medicine 2000; 154: 256–260
   PubMedGoogle Scholar
• Boyse TD, Patterson SK, Cohan RH, Korobkin M, Fitzgerald JT, Oh MS, et al. Does medical school performance predict radiology resident performance?. Academic Radiology 2002; 9: 437–445
   PubMed Web of Science ®Google Scholar
• Brailovsky C, Charlin B, Beausoleil S, Cote S, Van der Vleuten C. Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test. Medical Education 2001; 35: 430–436
   PubMed Web of Science ®Google Scholar
• Callahan CA, Erdmann JB, Hojat M, Veloski JJ, Rattner S, Nasca TJ, Gonnella JS. Validity of faculty ratings of students’ clinical competence in core clerkships in relation to scores on licensing examinations and supervisors’ ratings in residency’. Academic Medicine 2000; 75: S71–73
   PubMed Web of Science ®Google Scholar
• Case SM, Swanson DB. Validity of NBME Part I and Part II scores for selection of residents in orthopedic surgery, dermatology and preventive medicine. Academic Medicine 1993; 68: S51–S56
   PubMed Web of Science ®Google Scholar
• Clute KF. The General Practitioner: A Study of Medical Education and Practice in Ontario and Nova Scotia. Toronto, University of Toronto Press 1963
   Google Scholar
• Comprehensive Meta-Analysis Version 1.0.23 Biostat. 1998, [Online]. Available at: http://www.meta-analysis.com/
   Google Scholar
• Cox K. Examining and recording clinical performance: a critique and some recommendations. Education for Health 2000; 13: 45–52
   PubMedGoogle Scholar
• Donabedian A. The quality of care. Journal of the American medical Association 1998; 260: 1743–1748
   Google Scholar
• Dreyfus SE, Dreyfus HL. A five stage model of the mental activities involved in directed skill acquisition. 2001, Unpublished manuscript supported by the Air Force Office of Scientific Research under contract F49620-79-C-0063 with the University of California, Berkeley
   Google Scholar
• Epstein RM, Hundert EM. Defining and Assessing Professional Competence. Journal of the American Medical Association 2002; 287: 226–235
   PubMed Web of Science ®Google Scholar
• Erlandson EE, Calhoun JG, Barrack FM, Hull AL, Youmans LC, Davis WK, Bartlett RH. Resident selection: applicant selection criteria compared with performance. Surgery 1982; 92: 270–275
   PubMed Web of Science ®Google Scholar
• Executive Council, World Federation for Medical Education. International standards in medical education: assessment and accreditation of medical schools’ educational programs. A WFME position paper. Medical Education 1998; 32: 549–558
   PubMed Web of Science ®Google Scholar
• Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. British Medical Journal 2002; 324: 952–957
   PubMed Web of Science ®Google Scholar
• Fincher R-ME., Lewis LA, Kuske TT. Relationships of interns’ performances to their self-assessments of their preparedness for internship and to their academic performances in medical school. Academic Medicine 1993; 68: S47–S50
   PubMed Web of Science ®Google Scholar
• Fine PL, Hayward RA. Do the criteria of resident selection committees predict residents’ performances?. Academic Medicine 1995; 70: S34–S38
   Web of Science ®Google Scholar
• Fish DE, Radfar-Baublitz LS, Choi H, Felsenthal G. Correlation of standardized testing results with success on the 2001 American Board of Physical Medicine and Rehabilitation Part 1 Board Certificate Examination. American Journal of Physical Medicine and Rehabilitation 2003; 82: 686–691
   PubMed Web of Science ®Google Scholar
• General Medical Council. Tomorrow's Doctors: Recommendations on Undergraduate Medical Education. 1993, [Online]. Available at http://www.gmc-uk.org/med_ed/tomdoc.htm
   Google Scholar
• Gonnella JS, Hojat M. Relationship between performance in medical school and postgraduate competence. Medical Education 1983; 58: 679–685
   Web of Science ®Google Scholar
• Gonnella JS, Hojat M, Erdmann JB, Veloski JJ. A case of mistaken identity: Signal and noise in connecting performance assessments before and after graduation from medical school. Academic Medicine 1993; 68: S9–16
   PubMed Web of Science ®Google Scholar
• Gonnella JS, Erdmann JB, Hojat M. An empirical study of the predictive validity of number grades in medical school using 3 decades of longitudinal data: implications for a grading system. Medical Education 2004; 38: 425–434
   PubMed Web of Science ®Google Scholar
• Gunzburger LK, Fraizier RG, Yang LM, Rainey ML, Wronski T. Premedical and medical school performance in predicting first-year residency performance. Journal of Medical Education 1987; 62: 379–384
   PubMedGoogle Scholar
• Haig A, Dozier M. BEME Guide No 3: Systematic searching for evidence in medical education—Part 1: sources of information. Medical Teacher 2003; 25: 352–363
   PubMed Web of Science ®Google Scholar
• Harden RM, Grant J, Buckley G, Hart IR. BEME Guide No 1: Best Evidence Medical Education. Medical Teacher 1999; 21: 3–15
   Web of Science ®Google Scholar
• Heneman RL. The effects of time delay in rating and amount of information observed on performance rating accuracy. Academy of Management Journal 1983; 26: 677–686
   Web of Science ®Google Scholar
• Hermen MW, Velowski JJ, Hojat M. Validity and importance of low ratings given to medical graduates in non-cognitive areas. Journal of Medical Education 1983; 58: 837–843
   PubMedGoogle Scholar
• Hojat M, Gonnella JS, Erdmann JB, Veloski JJ. The fate of medical students with different levels of knowledge: are the basic sciences relevant to physician competence. Advances in Health Sciences Education 1997; 1: 179–196
   Web of Science ®Google Scholar
• Hojat M, Gonnella JS, Veloski JJ, Erdmann JB. Is the glass half full or half empty? A re-examination of the association between assessment measures during medical school and clinical competence after graduation. Academic Medicine 1993; 68: S69–S76
   PubMed Web of Science ®Google Scholar
• Hojat M, Veloski JJ, Borenstein BD. Components of clinical competence ratings: an empirical approach’. Education Psychology Measurements 1986; 46: 761–769
   Web of Science ®Google Scholar
• Holmboe ES, Hawkins RE. Methods of evaluating the clinical competence of residents in internal medicine: a review. Annals of Internal Medicine 1998; 129: 42–48
   PubMed Web of Science ®Google Scholar
• Kahn MJ, Merrill WW, Anderson DS, Szerlip HM. Residency program director evaluations do not correlate with performance on a required 4th-year objective structured clinical examination. Teaching & Learning in Medicine 2001; 13: 9–12
   PubMed Web of Science ®Google Scholar
• Kane MT. The assessment of professional competence. Evaluation & the Health Professions 1992; 15: 163–182
   PubMed Web of Science ®Google Scholar
• Kirkpatrick DI. Evaluation of Training. Training and Development Handbook, R Craig, I Mittel. McGraw Hill, New York 1967
   Google Scholar
• Korman M, Stubblefield RL. Medical schools evaluation and internship performance. Journal of Medical Education 1971; 46: 670–673
   PubMedGoogle Scholar
• Kron IL, Kaiser DL, Nolan SP, Rudolf LE, Muller WH, Jones RS. Can success in the surgical residence be predicted from pre–residency evaluation?. Annals of Surgery 1985; 202: 694–695
   PubMed Web of Science ®Google Scholar
• Loftus LS, Arnold L, Willoughby TL, Connolly A. First-year residents’ performances compared with their medical school class ranks as determined by three ranking systems. Academic Medicine 1992; 67: 319–323
   PubMed Web of Science ®Google Scholar
• Markert RJ. The relationship of academic measures in medical school to performance after graduation. Academic Medicine 1993; 68: S31–S34
   PubMed Web of Science ®Google Scholar
• Martin JA, Reznick RK, Rothman A, Tamblyn RM, Regehr G. BEME Systematic Review: Predictive values of measurements obtained in medical schools and future performance in medical practice. Medical Teacher 1996; 71: 170–175
   Google Scholar
• McManus IC, Smithers E, Partridge P, Keeling A, Fleming PR. A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. British Medical Journal 2003; 327: 139–142
   PubMed Web of Science ®Google Scholar
• METRO Project. 2004, [Online]. Available at: http://srv1.mvm.ed.ac.uk/metro/index/asp (accessed 7 June 2004)
   Google Scholar
• Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine 1990; 65: S63–S67
   PubMed Web of Science ®Google Scholar
• Paolo AM, Bonaminio GA. Measuring outcomes of undergraduate medical education: residency directors’ ratings of first-year residents. Academic Medicine 2003; 78: 90–95
   PubMed Web of Science ®Google Scholar
• Papadakis MA, Hodgson CS, Teherani A, Kohatsu ND. Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Academic Medicine 2004; 79: 244–249
   PubMed Web of Science ®Google Scholar
• Patterson F, Ferguson E, Lane P, Farrell K, Martlew J, Wells A. A competency model for general practice: implications for selection, training and development. British Journal of General Practice 2000; 50: 188–193
   PubMed Web of Science ®Google Scholar
• Pearson S-A, Rolfe IE, Henry RL. The relationship between assessment measures at Newcastle Medical School (Australia) and performance ratings during internship. Medical Education 1998; 32: 40–45
   PubMed Web of Science ®Google Scholar
• Peterson OL, Andrews LP, Spain RS, Greenberg BG. An analytical study of North Carolina general practice, Part 2. Journal of Medical Education 1956; 31: 1–165
   Google Scholar
• Price PB. Search for excellence. American Journal of Surgery 1969; 118: 815–821
   PubMed Web of Science ®Google Scholar
• Price PB, Taylor CW, Nelson DE, Lewis EG, Loughmiller GC, Mathiesen R, et al. Measurement and Predictors of Physician Performance: Two Decades of Intermittently Sustained Research. UT Aaron Press, Salt Lake City 1973
   Google Scholar
• Price PB, Taylor CW, Richards JM, Jacobsen TC. Measurement of Physician Performance. Journal of Medical Education 1964; 39: 203–211
   PubMedGoogle Scholar
• Prideaux D, Alexander H, Bower A, Dacre J, Haist S, Holly B, et al. Clinical teaching: maintaining an educational role for doctors in the new health care environment. Medical Education 2000; 34: 820–826
   PubMed Web of Science ®Google Scholar
• Probert CS, Cahill DJ, McCann GL, Ben-Shlomo Y. Traditional finals and OSCEs in predicting consultant and self-reported clinical skills of PRHOs: a pilot study. Medical Education 2003; 37: 597–602
   PubMed Web of Science ®Google Scholar
• Rabinowitz HK, Hojat MA. Comparison of the modified essay question and multiple choice question formats: their relationship to clinical performance. Family Medicine 1989; 21: 364–367
   PubMedGoogle Scholar
• Ram P. Comprehensive assessment of general practitioners: a study on validity, reliability and feasibility. Maastricht 1998, Thesis
   Google Scholar
• Renger R, Meadows L. Testing for predictive validity in health care education research: a critical review. Academic Medicine 1994; 69: 685–687
   PubMed Web of Science ®Google Scholar
• Rethans J-J. Does Competence Predict Performance? Standardized Patients as a Means to Investigate the Relationship between Competence and Performance of General Practitioners. Thesis. (Amsterdam: Thesis) Publishers. 1991
   Google Scholar
• Richards JM, Taylor CW, Price PB. The prediction of medical intern performance. Journal of Applied Psychology 1962; 46: 142–146
   Web of Science ®Google Scholar
• Rolfe IE, Andren JM, Pearson S, Hensley MK, Gordon JJ. Clinical competence of interns in New South Wales, Australia. Medical Education 1995; 29: 225–230
   PubMed Web of Science ®Google Scholar
• Ronai AK, Golmon ME, Shanks CA, Schafer MF, Brunner EA. Relationship between past academic performance and results of specialty in-training examinations. Journal of Medical Education 1984; 59: 341–344
   PubMedGoogle Scholar
• Royal College of Surgeons of Edinburgh. Guidelines for Basic Surgical Training. Royal College of Surgeons of Edinburgh, Edinburgh 1997
   Google Scholar
• Rutala PJ, Fulginiti JV, McGeagh AM, Leko EO, Koff NA, Witzke DB. Validity studies using standardized-patient examinations: standardized patient potpourri. Academic Medicine 1992; 67: S60–S62
   PubMed Web of Science ®Google Scholar
• Schmidt H, Norman G, Boshuizen HA. Cognitive perspective on medical expertise: theory and implications. Academic Medicine 1990; 65: 611–621
   PubMed Web of Science ®Google Scholar
• Smith SR. Correlations between graduates’ performances as first-year residents and their performances as medical students. Academic Medicine 1993; 68: 633–634
   PubMed Web of Science ®Google Scholar
• Sosenko J, Stekel KW, Soto R, Gelbard M. NBME Examination Part I as a predictor of clinical and ABIM certifying examination performances. Journal of General Internal Medicine 1993; 8: 86–88
   PubMed Web of Science ®Google Scholar
• Southgate L, Hays RB, Norcini J, Mulholland H, Ayers B, Wolliscroft J, et al. Setting performance standards for medical practice: a theoretical framework. Medical Education 2001; 35: 474–481
   PubMed Web of Science ®Google Scholar
• Stevens DP. Three questions for the LCME. Academic Medicine 2000; 75: 960–961
   PubMed Web of Science ®Google Scholar
• Tamblyn R, Abrahamoqicz M, Dauphinee WD, Hanley JA, Noricini J, Girard H, et al. Association between licensure examination scores and practice in primary care. Journal of the American Medical Association 2002; 288: 3019–3026
   PubMed Web of Science ®Google Scholar
• Tamblyn RM. Is the public being protected? Prevention of suboptimal medical practice through training programs and credentialing examinations. Evaluating Health Professions 1994; 17: 198–221
   PubMed Web of Science ®Google Scholar
• Tamblyn RM, Benaroya S, Snell L, McLeod P, Schnarch B, Abrahamowicz M. The feasibility and value of using patient satisfaction ratings to evaluate internal medicine residents. Journal of General Internal Medicine 1994; 9: 146–152
   PubMed Web of Science ®Google Scholar
• Taylor CW, Albo D. Measuring and predicting the performances of practicing physicians: an overview of two decades of research at the University of Utah. Academic Medicine 1993; 68: S65–S67
   PubMed Web of Science ®Google Scholar
• Van der Vleuten C. Validity of final examinations in undergraduate medical training. Bahrain Medical Journal 2000; 321: 1217–1219
   PubMed Web of Science ®Google Scholar
• Viswesvaran C, Ones D, Schmidt F. Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology 1996; 81: 557–574
   Web of Science ®Google Scholar
• Vu NV, Barrows HS, Marcy ML, Verhulst SJ, Colliver JA, Travis T. Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine. Academic Medicine 1992; 67: 42–50
   PubMed Web of Science ®Google Scholar
• Vu NV, Distlehorst LH, Verhulst SJ, Colliver JA. Clinical performance-based test sensitivity and specificity in predicting first-year residency performance’. Academic Medicine 1993; 68: S41–S45
   PubMed Web of Science ®Google Scholar
• Wass V, van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001; 357: 945–949
   PubMed Web of Science ®Google Scholar
• West PA. Calibre of people recruited to medicine may be too high for the job [letter]. British Medical Journal 2001; 322: 1361
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Frampton CM. Comprehensive undergraduate medical assessments improve prediction of clinical performance. Medical Education 2004; 38: 1111–1116
   PubMed Web of Science ®Google Scholar
• Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine 2003; 15: 270–292
   PubMed Web of Science ®Google Scholar
• Wingard JR, Williamson JW. Grades as predictors of physicians career performance: an evaluative literature review. Medical Education 1973; 48: 311–322
   Web of Science ®Google Scholar
• Wolf FM, Shea JA, Albanese MA. Toward setting a research agenda for systematic reviews of evidence of the effects of medical education. Teaching and Learning in Medicine 2001; 13: 54–60
   PubMed Web of Science ®Google Scholar
• World Federation for Medical Education. Postgraduate Medical Education. WFME Global Standards for Quality Improvement. WFME, Copenhagen 2003, [Online]. Available at: http://www.wfme.org/
   Google Scholar
• Yindra KJ, Rosenfield PS, Donnelly MB. Medical school achievements as predictors of residency performance. Journal Medical Education 1998; 63: 356–363
   Google Scholar
• Zu G, Veloski JJ, Hojat M. Board certification: associations with physicians’ demographics and performances during medical school and residency. Academic Medicine 1998; 73: 1283–1289
   PubMed Web of Science ®Google Scholar