A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation

References

• Albanese MA, Mejicano G, Anderson WM, Gruppen L. Building a competency-based curriculum: the agony and the ecstasy. Adv Health Sci Educ Theory Pract. 2010;15(3):439–454. doi:https://doi.org/10.1007/s10459-008-9118-2.
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell L, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–645. doi:https://doi.org/10.3109/0142159X.2010.501190.
   PubMed Web of Science ®Google Scholar
• Pusic MV, Kessler D, Szyld D, Kalet A, Pecaric M, Boutis K. Experience curves as an organizing framework for deliberate practice in emergency medicine learning. Acad Emerg Med. 2012;19(12):1476–1480. doi:https://doi.org/10.1111/acem.12043.
   PubMed Web of Science ®Google Scholar
• Carraccio C, Englander R, Van Melle E, et al. Advancing competency-based medical education: a charter for clinician-educators. Acad Med. 2016;91(5):645–649. doi:https://doi.org/10.1097/ACM.0000000000001048.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR. A call to action: the controversy of and rationale for competency-based medical education. Med Teach. 2017;39(6):574–581. doi:https://doi.org/10.1080/0142159X.2017.1315067.
   PubMed Web of Science ®Google Scholar
• Carraccio C, Englander R, Gilhooly J, et al. Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. Acad Med. 2017;92(3):324–330. doi:https://doi.org/10.1097/ACM.0000000000001141.
   PubMed Web of Science ®Google Scholar
• Nousiainen MT, Caverzagie KJ, Ferguson PC, Frank JR. Implementing competency-based medical education: what changes in curricular structure and processes are needed?Med Teach. 2017;39(6):594–598. doi:https://doi.org/10.1080/0142159X.2017.1315077.
   PubMed Web of Science ®Google Scholar
• Lineberry M, Park YS, Cook DA, Yudkowsky R. Making the case for mastery learning assessments: key issues in validation and justification. Acad Med. 2015;90(11):1445–1450. doi:https://doi.org/10.1097/ACM.0000000000000860.
   PubMed Web of Science ®Google Scholar
• Grant J. The incapacitating effects of competence: a critique. Adv Health Sci Educ Theory Pract. 1999;4(3):271–277. doi:https://doi.org/10.1023/A:1009845202352.
   PubMed Web of Science ®Google Scholar
• Talbot M. Monkey see, monkey do: a critique of the competency model in graduate medical education. Med Educ. 2004;38(6):587–592. doi:https://doi.org/10.1046/j.1365-2923.2004.01794.x.
   PubMed Web of Science ®Google Scholar
• Castro D, Yang J, Greer ML, et al. Competency based medical education—towards the development of a standardized pediatric radiology testing module. Acad Radiol. 2020;27(11):1622–1632. doi:https://doi.org/10.1016/j.acra.2019.12.005.
   PubMed Web of Science ®Google Scholar
• Deitte LA, Gordon LL, Zimmerman BD, et al. Entrustable professional activities: ten things radiologists do. Acad Radiol. 2016;23(3):374–381. doi:https://doi.org/10.1016/j.acra.2015.11.010.
   PubMed Web of Science ®Google Scholar
• Royal College of Physicians and Surgeons. Final in-Training Evalaution Report (FITER). http://www.royalcollege.ca/rcsite/documents/ibd/diagnostic_radiology_fiter_e.pdf. Accessed February 3, 2020.
   Google Scholar
• van Deven T, Wang S, Hibbert KM. Radiology Education: The Evaluation and Assessment of Clinical Competence. New York, NY: Springer; 2012:215.
   Google Scholar
• Chetlen AL, Mendiratta-Lala M, Probyn L, et al. Conventional medical education and the history of simulation in radiology. Acad Radiol. 2015;22(10):1252–1267. doi:https://doi.org/10.1016/j.acra.2015.07.003.
   PubMed Web of Science ®Google Scholar
• Boutis K, Pecaric M, Pusic M. Selecting radiographs for teaching the interpretation of X-rays based on ratings by the target population. Med Educ. 2009;43(5):434–441. doi:https://doi.org/10.1111/j.1365-2923.2009.03311.x.
   PubMed Web of Science ®Google Scholar
• Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005;80(10):S46–S54. doi:https://doi.org/10.1097/00001888-200510001-00015.
   PubMed Web of Science ®Google Scholar
• Hammond KS. The Essential Brunswik: Beginnings, Explications, Applications. Oxford, OA: Oxford University Press; 2001.
   Google Scholar
• Buijze GA, Guitton TG, van Dijk CN, Ring D. Training improves interobserver reliability for the diagnosis of scaphoid fracture displacement. Clin Orthop Relat Res. 2012;470(7):2029–2034. doi:https://doi.org/10.1007/s11999-012-2260-4.
   PubMed Web of Science ®Google Scholar
• Østergaard ML, Rue Nielsen K, Albrecht-Beste E, Kjaer Ersbøll A, Konge L, Bachmann Nielsen M. Simulator training improves ultrasound scanning performance on patients: a randomized controlled trial. Eur Radiol. 2019;29(6):3210–3218. doi:https://doi.org/10.1007/s00330-018-5923-z.
   PubMed Web of Science ®Google Scholar
• Soltanimehr E, Bahrampour E, Imani MM, Rahimi F, Almasi B, Moattari M. Effect of virtual versus traditional education on theoretical knowledge and reporting skills of dental students in radiographic interpretation of bony lesions of the jaw. BMC Med Educ. 2019;19(1):233. doi:https://doi.org/10.1186/s12909-019-1649-0.
   PubMedGoogle Scholar
• Todsen T, Melchiors J, Charabi B, et al. Competency-based assessment in surgeon-performed head and neck ultrasonography: a validity study. Laryngoscope. 2018;128(6):1346–1352. doi:https://doi.org/10.1002/lary.26841.
   PubMed Web of Science ®Google Scholar
• Wong V, Smith AJ, Hawkins NJ, et al. Adaptive tutorials versus web-based resources in radiology: a mixed methods comparison of efficacy and student engagement. Acad Radiol. 2015;22(10):1299–1307. doi:https://doi.org/10.1016/j.acra.2015.07.002.
   PubMed Web of Science ®Google Scholar
• Ericsson A, Pool R. The power of purposeful practice. In: Ericsson APR, ed. Peak: Secrets from the New Science of Expertise. Boston, MA: Houghton Mifflin Harcourt; Boston, MA, USA 2016:1–25.
   Google Scholar
• Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90(11):1471–1486. doi:https://doi.org/10.1097/ACM.0000000000000939.
   PubMed Web of Science ®Google Scholar
• Lee MS, Pusic M, Carriere B, Dixon A, Stimec J, Boutis K. Building emergency medicine trainee competency in pediatric musculoskeletal radiograph interpretation: a multicenter prospective cohort study. Acad Emerg Med Educ Train. 2019;3(3):269–279.
   Google Scholar
• Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49(6):560–575. doi:https://doi.org/10.1111/medu.12678.
   PubMed Web of Science ®Google Scholar
• Hambleton, RK, Pitoniak, M. Setting performance standards. In: Robert LB (ed.), Educational Measurement, 4th ed. Westport, CT: Praeger; 2014:433–470.
   Google Scholar
• Zieky M, Marianne P. A primer on setting cut scores on tests of educational achievement. https://www.ets.org/research/policy_research_reports/publications/publication/2006/dbkw.
   Google Scholar
• Mortaz Hejri S, Jalili M. Standard setting in medical education: fundamental concepts and emerging challenges. Med J Islam Repub Iran. 2014;19(28):34.
   Google Scholar
• Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychol Bull. 1955;52(4):281–302. doi:https://doi.org/10.1037/h0040957.
   PubMed Web of Science ®Google Scholar
• Cizek GJ, Bunch MB. Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. Thousand Oaks, California: SAGE Publications Inc.; 2006.
   Google Scholar
• Ebel RL. Essentials of Educational Measurement. 2nd ed. Upper Saddle River, New Jersey: Prentice Hall; 1972.
   Google Scholar
• Weiss DJ, Shanteau J. Empirical assessment of expertise. Hum Factors. 2003;45(1):104–114. doi:https://doi.org/10.1518/hfes.45.1.104.27233.
   PubMed Web of Science ®Google Scholar
• Cook DA, Lineberry M. Consequences validity evidence: evaluating the impact of educational assessments. Acad Med. 2016;91(6):785–795. doi:https://doi.org/10.1097/ACM.0000000000001114.
   PubMed Web of Science ®Google Scholar
• Yudkowsky R, Park YS, Lineberry M, Knox A, Ritter EM. Setting mastery learning standards. Acad Med. 2015;90(11):1495–1500. doi:https://doi.org/10.1097/ACM.0000000000000887.
   PubMed Web of Science ®Google Scholar
• Dixon A. Pediatric fractures - an educational needs assessment of Canadian pediatric emergency medicine residents. Open Access Emerg Med. 2015;7:25–29. doi:https://doi.org/10.2147/OAEM.S82393.
   PubMedGoogle Scholar
• Reeder BM, Lyne ED, Patel DR, Cucos DR. Referral patterns to a pediatric orthopedic clinic: implications for education and practice. Pediatrics. 2004;113(3):e163–719. doi:https://doi.org/10.1542/peds.113.3.e163.
   PubMed Web of Science ®Google Scholar
• Ryan LM, DePiero AD, Sadow KB, et al. Recognition and management of pediatric fractures by pediatric residents. Pediatrics. 2004;114(6):1530–1533. doi:https://doi.org/10.1542/peds.2004-0120.
   PubMed Web of Science ®Google Scholar
• Taras HL, Nader PR. Ten years of graduates evaluate a pediatric residency program. Am J Dis Child. 1990;144(10):1102–1105. doi:https://doi.org/10.1001/archpedi.1990.02150340046021.
   PubMedGoogle Scholar
• Trainor JL, Krug SE. The training of pediatric residents in the care of acutely ill and injured children. Arch Pediatr Adolesc Med. 2000;154(11):1154–1159. doi:https://doi.org/10.1001/archpedi.154.11.1154.
   PubMedGoogle Scholar
• Al‐Sani F, Prasad S, Panwar J, et al. Adverse events from emergency physician pediatric extremity radiograph interpretations: a prospective cohort study. Acad Emerg Med. 2020;27(2):128–138. doi:https://doi.org/10.1111/acem.13884.
   PubMed Web of Science ®Google Scholar
• van Merrienboer JJG, Kirschner PA, Kester L. Taking the load off a learner’s mind: instructional design for complex learning. Educ Psychol. 2010;38(1):5–13. doi:https://doi.org/10.1207/S15326985EP3801_2.
   Google Scholar
• Vandewaetere M, Manhaeve D, Aertgeerts B, Clarebout G, Van Merriënboer JJ, Roex A. 4C/ID in medical education: how to design an educational program based on whole-task learning: AMEE Guide No. 93. Med Teach. 2015;37(1):4–20. doi:https://doi.org/10.3109/0142159X.2014.928407.
   PubMed Web of Science ®Google Scholar
• Wickens CD, Hutchins S, Carolan T, Cumming J. Effectiveness of part-task training and increasing-difficulty training strategies: a meta-analysis approach. Hum Factors. 2013;55(2):461–470. doi:https://doi.org/10.1177/0018720812451994.
   PubMed Web of Science ®Google Scholar
• Pusic M, Pecaric M, Boutis K. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Acad Med. 2011;86(6):731–736. doi:https://doi.org/10.1097/ACM.0b013e3182178c3c.
   PubMed Web of Science ®Google Scholar
• Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med. 2006;18(1):50–57. doi:https://doi.org/10.1207/s15328015tlm1801_11.
   PubMed Web of Science ®Google Scholar
• Hejri SM, Jalili M. Standard setting in medical education: fundamental concepts and emergening challenges. Med J Islamic Republic Iran. 2014;28(34):1–6.
   PubMedGoogle Scholar
• Hurtz GM, Hertz NR. How many raters should be used for establishing cut-off scores with the Agnoff method? A generalizability theory study. Educ Psychol Meas. 1999;59(6):885–897. doi:https://doi.org/10.1177/00131649921970233.
   Web of Science ®Google Scholar
• Pusic MV, Andrews JS, Kessler DO, et al. Prevalence of abnormal cases in an image bank affects the learning of radiograph interpretation. Med Educ. 2012;46(3):289–298. doi:https://doi.org/10.1111/j.1365-2923.2011.04165.x.
   PubMed Web of Science ®Google Scholar
• Downing SM. Item response theory: applications of modern test theory in medical education. Med Educ. 2003;37(8):739–745. doi:https://doi.org/10.1046/j.1365-2923.2003.01587.x.
   PubMed Web of Science ®Google Scholar
• Boutis K, Howard A, Constantine E, Cuomo A, Somji F, Narayanan U. Evidence to practice: orthopedic management of common pediatric fractures. J Pediatr Orthop. 2015;35(1):18–23. doi:https://doi.org/10.1097/BPO.0000000000000223.
   PubMed Web of Science ®Google Scholar
• Boutis K, Willan AR, Babyn P, Alman B, Narayanan U, Schuh S. A randomized, controlled trial of a removable brace versus casting in children with low-risk ankle fractures. Pediatrics. 2007;119(6):e1256–e1263. doi:https://doi.org/10.1542/peds.2006-2958.
   PubMed Web of Science ®Google Scholar
• Flynn JM, Skaggs DL, Waters PM. Rockwood and Wilkins’ Fractures in Children. 7th ed. Lippincott-Raven Publishers; 2014:1252.
   Google Scholar
• Boutis K. Common pediatric fractures treated with minimal intervention. Pediatr Emerg Care. 2010;26(2):152–157. quiz 158-62. doi:https://doi.org/10.1097/PEC.0b013e3181ce310c.
   PubMed Web of Science ®Google Scholar
• Sabhaney V, Boutis K, Yang G, et al. Bone fractures in children: is there an association with obesity?J Pediatr. 2014;165(2):313–318. doi:https://doi.org/10.1016/j.jpeds.2014.04.006.
   PubMed Web of Science ®Google Scholar
• Cendan J, Wier D, Behrns K. A primer on standards setting as it applies to surgical education and credentialing. Surg Endosc. 2013;27(7):2631–2637. doi:https://doi.org/10.1007/s00464-012-2771-9.
   PubMed Web of Science ®Google Scholar
• Bond TG, Fox CM. Applying the Rasch Model: Fundamental Measurement in the Health Sciences. 2nd ed. Lawrence Erlbaum Associates, Inc; 2007:234.
   Google Scholar
• Shepard LA. The centrality of test use and consequences for test validity. Educ Meas Issues Pract. 2005;16(2):5–24. doi:https://doi.org/10.1111/j.1745-3992.1997.tb00585.x.
   Google Scholar
• Kramer MS, Roberts-Bräuer R, Williams RL. Bias and ‘overcall’ in interpreting chest radiographs in young febrile children. Pediatrics. 1992;90(1):11–13.
   PubMed Web of Science ®Google Scholar
• Zhao S, Wang S, Pan P, et al. Magnitude, risk factors, and factors associated with adenoma miss rate of tandem colonoscopy: a systematic review and meta-analysis. Gastroenterology. 2019;156(6):1661–1674.e11. doi:https://doi.org/10.1053/j.gastro.2019.01.260.
   PubMed Web of Science ®Google Scholar
• Searle J. Defining competency - the role of standard setting. Med Educ. 2000;34(5):363–366. doi:https://doi.org/10.1046/j.1365-2923.2000.00690.x.
   PubMed Web of Science ®Google Scholar
• Itri JN, Patel SH. Heuristics and cognitive error in medical imaging. AJR Am J Roentgenol. 2018;210(5):1097–1105. doi:https://doi.org/10.2214/AJR.17.18907.
   PubMed Web of Science ®Google Scholar
• Taylor PM. A review of research into the development of radiologic expertise: implications for computer-based training. Acad Radiol. 2007;14(10):1252–1263. doi:https://doi.org/10.1016/j.acra.2007.06.016.
   PubMed Web of Science ®Google Scholar
• Kwan C, Pusic M, Pecaric M, Weerdenburg K, Tessaro M, Boutis K. The variable journey in learning to interpret pediatric point-of-care ultrasound images: a multicenter prospective cohort study. Acad Emerg Med Educ Train. 2019;4(2):111–122.
   Google Scholar
• John-Matthews JS, Wallace MJ, Robinson L. The Delphi technique in radiography education research. Radiography. 2017;23(Suppl 1):S53–S57. doi:https://doi.org/10.1016/j.radi.2017.03.007.
   PubMedGoogle Scholar
• Wells GL, Windschitl PD. Stimulus sampling and social psychological experimentation. Pers Soc Psychol Bull. 1999;25(9):1115–1125. doi:https://doi.org/10.1177/01461672992512005.
   Web of Science ®Google Scholar
• Maher BA. Stimulus sampling in clinical research: representative design reviewed. J Consult Clin Psychol. 1978;46(4):643–647. doi:https://doi.org/10.1037//0022-006x.46.4.643.
   PubMed Web of Science ®Google Scholar
• Bourque J, Skinner H, Dupré J, et al. Performance of the Ebel standard-setting method in spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisted of multiple-choice questions. J Educ Eval Health Prof. 2020;17(12):12. doi:https://doi.org/10.3352/jeehp.2020.17.12.
   PubMedGoogle Scholar
• Mu EW, Pusic M, Coneybeare M, Meehan SA. Learning analytics to enhance dermatopathology education among dermatology residents. J Drugs Dermatol. 2019;18(12):1231–1236.
   PubMed Web of Science ®Google Scholar
• Smith EV, Jr, Stone GE. Criterion Referenced Testing: Practice Analysis to Score Reporting Using Rasch Measurement. Maple Grove, Minn: JAM Press; 2009.
   Google Scholar
• Diaz FN, Ulla M. Validation of an informatics tool to assess resident’s progress in developing reporting skills. Insights Imaging. 2019;10(1):96. doi:https://doi.org/10.1186/s13244-019-0772-0.
   PubMed Web of Science ®Google Scholar
• Kwan BYM, Mbanwi A, Cofie N, et al. Creating a competency-based medical education curriculum for Canadian Diagnostic Radiology Residency (Queen’s Fundamental Innovations in Residency Education)-Part 1: transition to discipline and foundation of discipline stages. Can Assoc Radiol J. 2020. doi:https://doi.org/10.1177/0846537119894723.
   Google Scholar
• Schmitt JE, Scanlon MH, Servaes S, Levin D, Cook TS. Milestones on a shoestring: a cost-effective, semi-automated implementation of the new ACGME requirements for radiology. Acad Radiol. 2015;22(10):1287–1293. doi:https://doi.org/10.1016/j.acra.2015.02.013.
   PubMed Web of Science ®Google Scholar
• van der Laan MJ, Rose S. Targeted Learning in Data Science. New York, NY: Springer; 2018.
   Google Scholar
• Boutis K, Cano S, Pecaric M, et al. Interpretation difficulty of normal versus abnormal radiographs using a pediatric example. Can Med Ed J. 2016;7(1):e68-77–e77. doi:https://doi.org/10.36834/cmej.36654.
   PubMedGoogle Scholar