Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

References

• Crosson FJ, Leu J, Roemer BM, Ross MN. Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Affairs 2011;30:2142–8.
   PubMed Web of Science ®Google Scholar
• Institute of Medicine. Graduate medical education that meets the nation's health needs. Washington, DC: The National Academies Press, 2014.
   Google Scholar
• Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system. New England Journal of Medicine 2012;366:1051–6.
   PubMed Web of Science ®Google Scholar
• Accreditation Council for Graduate Medical Education. ACMGE common program requirements. Available at:http://www.acgme.org/acgmeweb/tabid/429/programandi nstitutionalaccreditation/commonprogramrequirements.aspx. Accessed June 18, 2015.
   Google Scholar
• Dreyfus H, Dreyfus S. Mind over machine: the power of human intuitive expertise in the era of the computer. New York, NY: Free Press, 1986.
   Google Scholar
• Carraccio CL, Benson BJ, Nixon J, Derstine PL. From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills. Academic Medicine 2008;83:761–7.
   PubMed Web of Science ®Google Scholar
• Accreditation Council for Graduate Medical Education, American Board of Internal Medicine. The internal medicine milestone project. Available at: https://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/InternalMedicineMilestones.pdf. Accessed June 18, 2015.
   Google Scholar
• Swing SR, Cowley DS, Bentman A. Assessing resident performance on the psychiatric milestones. Academic Psychiatry 2014;38:294–302.
   PubMed Web of Science ®Google Scholar
• Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Academic Medicine 2009;84:301–9.
   PubMed Web of Science ®Google Scholar
• Boateng BE, Bass LD, Blaszak RT, Farrar HC. The development of a competency-based assessment rubric to measure resident milestones. Journal of Graduate Medical Education 2009;1:45–8.
   PubMedGoogle Scholar
• Tekian A, Hodges BD, Roberts TE, Schuwirth L, Norcini J. Assessing competencies using milestones along the way. Medical Teacher 2014;19:1–4.
   Web of Science ®Google Scholar
• Hawkins RE, Holmboe ES. Constructing an evaluation system for an educational program. In ES Holmboe, RE Hawkins (Eds.), Practical guide to the evaluation of clinical competence (pp. 216–38). Philadelphia, PA: Mosby, 2008.
   Google Scholar
• Holmboe ES. Realizing the promise of competency-based medical education. Academic Medicine 2015;90:411–3.
   PubMed Web of Science ®Google Scholar
• Silber CG, Nasca TJ, Paskin DL, Eiger G, Robeson M, Veloski JJ. Do global rating forms enable program directors to assess the ACGME competencies? Academic Medicine 2004;79:549–56.
   PubMed Web of Science ®Google Scholar
• Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE guide no. 31. Medical Teacher 2007;29:855–71.
   PubMed Web of Science ®Google Scholar
• Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings' post-rotation performance ratings detect residents' clinical performance deficiencies? Academic Medicine 2004;79:453–7.
   PubMed Web of Science ®Google Scholar
• Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MK. How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine 1989;4:202–8.
   PubMed Web of Science ®Google Scholar
• Siegel BS, Greenberg LW. Effective evaluation of residency education: how do we know it when we see it? Pediatrics 2000;105:964–5.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: a conceptual model. Medical Education 2011;45:1048–60.
   PubMed Web of Science ®Google Scholar
• Cleland JA, Knight LV, Rees CE, Tracey S, Bond CM. Is it me or is it them? Factors that influence the passing of underperforming students. Medical Education 2008;42:800–9.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputation? A study of the reliability and predictive ability of ITER scores and narrative comments. Academic Medicine 2013;88:1539–44.
   PubMed Web of Science ®Google Scholar
• Park YS, Riddle J, Tekian A. Validity evidence of resident competency ratings and the identification of problem residents. Medical Education 2014;48:614–22.
   PubMed Web of Science ®Google Scholar
• Friedman KA, Balwan S, Cacace F, Katona K, Sunday S, Chaudhry S. Impact on house staff evaluation scores when changing from a Dreyfus- to a milestone-based evaluation model: One internal medicine residency program's findings. Medical Education Online 2014;19:1–8.
   Web of Science ®Google Scholar
• Messick S. Standards of validity and the validity of standards in performance assessment. Educational Measurement 1995;14:5–8.
   Google Scholar
• Brennan RL. Generalizability theory. New York, NY: Springer-Verlag, 2001.
   Google Scholar
• Kreiter CD, Ferguson K, Lee WC, Brennan RL, Densen P. A generalizability study of a new standardized rating form used to evaluate students' clinical clerkship performances. Academic Medicine 1998;73:1294–8.
   PubMed Web of Science ®Google Scholar
• DeMars C. Estimating variance components from sparse data matrices in large-scale educational assessments. Applied Measurement in Education 2015;28:1–13.
   Web of Science ®Google Scholar
• Bollen KA. Structural equations with latent variables. New York, NY: Wiley, 1989.
   Google Scholar
• Hu L, Bentler PM. Fit indices in covariance structure modeling: Sensitivity to underparameterized model specification. Psychological Methods 1998;3:424–53.
   Web of Science ®Google Scholar
• Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS. Internal structure of mini-CEX scores for internal medicine residents: Factor analysis and generalizability. Advances in Health Sciences Education 2010;15:633–45.
   PubMed Web of Science ®Google Scholar
• Margolis MJ, Clauser BE, Cuddy MM, Ciccone A, Mee J, Harik P, et al.. Use of the Mini-Clinical Evaluation Exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine 2006;81:S56–S60.
   PubMed Web of Science ®Google Scholar
• Hill F, Kendall K, Galbraith K, Crossley J. Impelemting the undergraduate mini-CEX: A tailored approach at Southampton University. Medical Education 2009;43:326–34.
   PubMed Web of Science ®Google Scholar
• Archer JC, Norcini J, Davies HA. Use of SPRAT for peer review of paediatricians in training. British Medical Journal 2005;330:1251–3.
   PubMed Web of Science ®Google Scholar
• Nasca TJ, Gonnella JS, Hojat M, Veloski J, Erdmann JB, Robeson M, et al. Conceptualization and measurement of clinical competence of residents: A brief rating form and its psychometric properties. Medical Teacher 2002;24:299–303.
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Frampton CM. Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting? Medical Education 2003;37:233–40.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Regehr G, Eva KW. Rater-based assessment as social judgments: Rethinking the etiology of rater errors. Academic Medicine 2011;86:S1–7.
   PubMed Web of Science ®Google Scholar
• Verhulst SJ, Colliver JA, Paiva RE, Williams RG. A factor analysis study of first-year residents. Journal of Medical Education 1986;61:132–4.
   PubMedGoogle Scholar
• Association of American Medical Colleges. Core entrustable professional activities for entering residency: curriculum developers' guide. Available at: https://www.mededportal.org/icollaborative/resource/887. Accessed June 18, 2015.
   Google Scholar
• Kane M, Case SM. The reliability and validity of weighted composite scores. Applied Measurement in Education 2004;17:221–40.
   Web of Science ®Google Scholar
• Davies H, Archer J, Southgate L, Norcini J. Initial evaluation of the first year of the Foundation Assessment Programme. Medical Education 2009;43:74–81.
   PubMed Web of Science ®Google Scholar
• Smither JW, London M, Reilly RR. Does performance improve following multisource feedback? A theoretical model, meta-analysis and review of empirical findings. Personnel Psychology 2005;58:33–66.
   Web of Science ®Google Scholar
• Archer J, Swanwick T, Smith D, O'Keefe C, Cater N. Developing a multisource feedback tool for postgraduate medical educational supervisors. Medical Teacher 2013;35:145–54.
   PubMed Web of Science ®Google Scholar
• Haist SA, Thompson JS, Monroe AL, Clements MM, Norcini JJ. Nursing and faculty evaluating internal medicine housestaff. Teaching and Learning in Medicine 1997;9:174–80.
   Web of Science ®Google Scholar
• Govaerts MJB, Van de Wiel MWJ, Schuwirth LWT, Van der Vleuten CPM, Muijtjens AMM. Workplace-based assessment: Raters' performance theories and constructs. Advances in Health Sciences Education 2013;18:375–96.
   PubMed Web of Science ®Google Scholar
• Sherbino J, Kulasegaram K, Worster A, Norman GR. The reliability of encounter cards to assess the CanMEDS roles. Advances in Health Sciences Education 2013;18:987–96.
   PubMed Web of Science ®Google Scholar
• Kreiter CD, Ferguson KJ. Examining the generalizability of ratings across clerkships using a clinical evaluation form. Evaluation & the Health Professions 2011;24:36–46.
   Web of Science ®Google Scholar
• Woehr DJ, Huffcutt AI. Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology 1994;67:189–205.
   Web of Science ®Google Scholar
• Park YS, Chen J, Holtzman SL. Evaluating efforts to minimize rater bias in scoring classroom observations. In TJ Kane, KA Kerr, RC Pianta (Eds.), Designing teacher evaluation systems (pp. 383–414). San Francisco, CA: Wiley, 2014.
   Google Scholar
• Newble D, Dawson B, Dauphinee D, Page G, Macdonald M, Swanson D, et al. Guidelines for assessing clinical competence. Teaching and Learning in Medicine 1994;6:213–20.
   Google Scholar
• Boulet JR, Jeffries PR, Hatala RA, Korndorffer JR, Feinstein DM, Poche JP. Research regarding methods of assessing learning outcomes. Simulation in Healthcare 2011;6:S48–S51.
   PubMedGoogle Scholar
• Baker K. Determining resident clinical performance: Getting beyond the noise. Anesthesiology 2011;115:862–78.
   PubMed Web of Science ®Google Scholar
• Chaudhry SI, Holmboe E, Beasley BW. The state of evaluation in internal medicine residency. Journal of General Internal Medicine 2008;23:1010–5.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Bowen JL, Green M, Gregg J, DiFrancesco L, Reynolds E, et al. Reforming internal medicine residency training: A report from the society of general internal medicine's task force for residency reform. Journal of General Internal Medicine 2005;20:1165–72.
   PubMed Web of Science ®Google Scholar
• Lewis FR, Klingensmith ME. Issues in general surgery residency training—2012. Annals of Surgery 2012;256:553–9.
   PubMed Web of Science ®Google Scholar
• Beeson MS, Vozenilek JA. Specialty milestones and the next accreditation system: An opportunity for the simulation community. Simulation in Healthcare 2014;9:184–91.
   PubMedGoogle Scholar