Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

References

• Alofs L, Huiskes J, Heineman MJ, Buis C, Horsman M, van der Plank L, ten Cate O. 2015. User reception of a simple online multisource feedback tool for residents. Perspect Med Educ. 4(2):57–65.
   PubMedGoogle Scholar
• Bohnen JD, George BC, Williams RG, Schuller MC, DaRosa DA, Torbeck L, Mullen JT, Meyerson SL, Auyang ED, Chipman JG, et al. 2016. The feasibility of real-time intraoperative performance assessment with simpl (system for improving and measuring procedural learning): early experience from a multi-institutional trial. J Surg Educ. 73(6):e118–e130.
   PubMedGoogle Scholar
• Chen HC, ten Cate O. 2018. Assessment through entrustable professional activities. In: Delany, C., Molloy E., editors. Learning & Teaching in Clinical Contexts: A practical guide. Chatswood: Elsevier Australia, p. 286–304.
   Google Scholar
• Cheung K, Rogoza C, Chung AD, Kwan BYM. 2021. Analyzing the administrative burden of competency-based medical education. Can Assoc Radiol J. 73(2):299–304.
   Web of Science ®Google Scholar
• Cheung WJ, Patey AM, Frank JR, Mackay M, Boet S. 2019. Barriers and enablers to direct observation of trainees’ clinical performance: a qualitative study using the theoretical domains framework. Acad Med. 94(1):101–114.
   PubMed Web of Science ®Google Scholar
• Cleaning and Disinfecting Your Facility. 2021. Centers for Disease Control. https://www.cdc.gov/coronavirus/2019-ncov/community/disinfecting-building-facility.html [accessed 2022 Oct 25].
   Google Scholar
• Countries with the largest digital populations in the world as of January 2022. no date. Statista. [accessed 2022 Oct 25]. https://www.statista.com/statistics/262966/number-of-internet-users-in-selected-countries/
   Google Scholar
• Crossley J, Johnson G, Booth J, Wade W. 2011. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ. 45(6):560–569.
   PubMed Web of Science ®Google Scholar
• David MFB, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. 2001. AMEE medical education Guide No. 24: portfolios as a method of student assessment. Medical Teacher. 23(6):535–551.
   PubMed Web of Science ®Google Scholar
• Donoff MG. 2009. Field notes. Can Fam Physician. 55(12):1260–1262.
   PubMed Web of Science ®Google Scholar
• George BC, Bohnen JD, Schuller MC, Fryer JP. 2020. Using smartphones for trainee performance assessment: a SIMPL case study. Surgery. 167(6):903–906.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, Watling CJ, Schumacher DJ, Gingerich A, Hatala R. 2021. Numbers encapsulate, words elaborate: toward the best use of comments for assessment and feedback on entrustment ratings. Acad Med. 96(7S):S81–S86.
   PubMed Web of Science ®Google Scholar
• Gruppen LD, Ten Cate O, Lingard LA, Teunissen PW, Kogan JR. 2018. Enhanced requirements for assessment in a competency-based, time-variable medical education system. Acad Med. 93(3S Competency-Based, Time-Variable Education in the Health Professions):S17–S21.
   PubMed Web of Science ®Google Scholar
• Hanson MN, Pryor AD, Jeyarajah DR, Minter RM, Mattar SG, Scott DJ, Brunt LM, Cummings M, Vassiliou M, Feldman LS. 2022. Implementation of entrustable professional activities into fellowship council accredited programs: a pilot project. Surgical Endoscopy. DOI:10.1007/s00464-022-09502-5.
   Web of Science ®Google Scholar
• Hauer KE, O'Sullivan PS, Fitzhenry K, Boscardin C. 2018. Translating theory into practice: implementing a program of assessment. Acad Med. 93(3):444–450.
   PubMed Web of Science ®Google Scholar
• Holmboe E, Ginsburg S, Bernabeo E. 2011. The rotational approach to medical education: time to confront our assumptions? Med Educ. 45(1):69–80.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Durning SJ, Hawkins RE., editors. 2018. A practical guide to the evaluation of clinical competence. Philadelphia (PA, USA): Elsevier.
   Google Scholar
• Kamp MA, Malzkorn B, von Sass C, DiMeco F, Hadjipanayis CG, Senft C, Rapp M, Gepfner-Tuma I, Fountas K, Krieg SM, et al. 2021. Proposed definition of competencies for surgical neuro-oncology training. J Neurooncol. 153(1):121–131.
   PubMed Web of Science ®Google Scholar
• Kaur B, Taylor E. 2023. Development of a pediatric anesthesia fellowship curriculum in Australasia by the society for pediatric anesthesia of New Zealand and Australia (SPANZA) education sub-committee. Pediatr Anaesth. 33(2):100-106.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Hatala R, Hauer KE, Holmboe E. 2017. Guidelines: the do’s, don’ts and don’t knows of direct observation of clinical skills in medical education. Perspect Med Educ. 6(5):286–305.
   PubMed Web of Science ®Google Scholar
• LaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. 2017. Staging a performance: learners’ perceptions about direct observation during residency. Med Educ. 51(5):498–510.
   PubMed Web of Science ®Google Scholar
• Landreville JM, Wood TJ, Frank JR, Cheung WJ. 2022. Does direct observation influence the quality of workplace-based assessment documentation? AEM Educ Train. 6(4):e10781.
   PubMedGoogle Scholar
• Lockyer J, Carraccio C, Chan M-K, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR, 2017. Core principles of assessment in competency-based medical education. Med Teach. 39(6):609–616.
   PubMed Web of Science ®Google Scholar
• Lockyer J. 2013. Multisource feedback: can it meet criteria for good assessment? J Contin Educ Health Prof. 33(2):89–98.
   PubMed Web of Science ®Google Scholar
• Lurie SJ. 2012. History and practice of competency-based assessment. Med Educ. 46(1):49–57.
   PubMed Web of Science ®Google Scholar
• Marty AP, Braun J, Schick C, Zalunardo MP, Spahn DR, Breckwoldt J. 2022. A mobile application to facilitate implementation of programmatic assessment in anaesthesia training. Br J Anaesth. 128(6):990–996.
   PubMed Web of Science ®Google Scholar
• Massie J, Ali JM. 2016. Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract. 21(2):455–473.
   PubMed Web of Science ®Google Scholar
• Maudsley G, Taylor D, Allam O, Garner J, Calinici T, Linkman K. 2019. A best evidence medical education (BEME) systematic review of: what works best for health professions students using mobile (hand-held) devices for educational support on clinical placements? BEME Guide No. 52. Med Teach. 41(2):125–140.
   PubMed Web of Science ®Google Scholar
• Mezirow J. 2009. An overview on transformative learning. In: Illeris K, editor. Contemporary Theories of Learning. Oxon: Routledge, p. 90–_105.
   Google Scholar
• Miller GE. 1990. The assessment of clinical skills/competence/performance. Acad Med. 65(9 Suppl):S63–S67.
   PubMed Web of Science ®Google Scholar
• Nikou SA, Economides AA. 2018. Mobile-based assessment: a literature review of publications in major referred journals from 2009 to 2018. Computers and Education. Elsevier, 125:101–119.
   Web of Science ®Google Scholar
• NN 2018. What is GDPR, the EU’s new data protection law? General Data Protection Regulation. [accessed 2022 Oct 25]. https://gdpr.eu/what-is-gdpr/.
   Google Scholar
• Norcini J, Burch V. 2007. Workplace-based assessment as an educational tool: AMEE Guide No. Med Teach. 29(9):855–871.
   PubMed Web of Science ®Google Scholar
• Norcini J. 2020. Is it time for a new model of education in the health professions? Med Educ. 54(8):687–690.
   PubMed Web of Science ®Google Scholar
• Number of internet users worldwide as of 2022, by region no date. Statista. [accessed 2022 Oct 25]. https://www.statista.com/statistics/249562/number-of-worldwide-internet-users-by-region/.
   Google Scholar
• Ochoa R, Lach S, Masaki T, Rodríguez-Castelán C. 2021. Why aren’t more people using mobile internet in West Africa?, World Bank Blogs. [accessed 2022 Oct 25]. https://blogs.worldbank.org/digital-development/why-arent-more-people-using-mobile-internet-west-africa
   Google Scholar
• Ott MC, Pack R, Cristancho S, Chin M, Van Koughnett JA, Ott M. 2022. ‘The most crushing thing’: understanding resident assessment burden in a competency-based curriculum. J Grad Med Educ. 14(5):583–592.
   PubMedGoogle Scholar
• Oudkerk Pool A, Govaerts MJB, Jaarsma DADC, Driessen EW. 2018. From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Adv Health Sci Educ Theory Pract. 23(2):275–287.
   Google Scholar
• Oxford English Dictionary. no date. Oxford University Press. https://www.oed.com.
   Google Scholar
• Schut S, Driessen E, van Tartwijk J, van der Vleuten C, Heeneman S. 2018. Stakes in the eye of the beholder: an international study of learners’ perceptions within programmatic assessment. Med Educ. 52(6):654–663.
   PubMed Web of Science ®Google Scholar
• Schut S, Maggio LA, Heeneman S, van Tartwijk J, van der Vleuten C, Driessen E. 2021. Where the rubber meets the road—An integrative review of programmatic assessment in health care professions education. Perspect Med Educ. 10(1):6–13.
   PubMed Web of Science ®Google Scholar
• Sukhera J, Wodzinski M, Milne A, Teunissen PW, Lingard L, Watling C. 2019. Implicit bias and the feedback paradox: exploring how health professionals engage with feedback while questioning its credibility. Acad Med. 94(8):1204–1210.
   PubMed Web of Science ®Google Scholar
• Telio S, Regehr G, Ajjawi R. 2016. Feedback and the educational alliance: examining credibility judgements and their consequences. Med Educ. 50(9):933–942.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Chen HC. 2020. The ingredients of a rich entrustment decision. Med Teach. 42(12):1413–1420.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Hoff RG. 2017. From case-based to entrustment-based discussions. Clin Teach. 14(6):385–389.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Scheele F. 2007. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 82(6):542–547.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Schwartz A, Chen HC. 2020. Assessing trainees and making entrustment decisions: on the nature and use of entrustment-supervision scales. Acad Med. 95(11):1662–1669.
   PubMed Web of Science ®Google Scholar
• ten Cate O. 2016. Entrustment as assessment: recognizing the ability, the right, and the duty to act. J Grad Med Educ. 8(2):261–262.
   PubMedGoogle Scholar
• ten Cate O. 2022. How can entrustable professional activities serve the quality of health care provision through licensing and certification? Can Med Educ J. 13(4):8–14.
   PubMedGoogle Scholar
• ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. 2015. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE Guide No. 99. Med Teach. 37(11):983–1002.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Dahdal S, Lambert T, Neubauer F, Pless A, Pohlmann PF, van Rijen H, Gurtner C. 2020. Ten caveats of learning analytics in health professions education: a consumer’s perspective. Med Teach. 42(6):673–678.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. 2015. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE Guide No. 99. Med Teach. 37(11):983–1002.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Hart D, Ankel F, Busari J, Englander R, Glasgow N, Holmboe E, Iobst W, Lovell E, Snell LS, et al. 2016. Entrustment decision making in clinical training. Acad Med. 91(2):191–198.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Regehr G. 2019. The power of subjectivity in the assessment of medical trainees. Acad Med. 94(3):333–337.
   PubMed Web of Science ®Google Scholar
• ten Cate O. 2014. AM last page: what entrustable professional activities add to a competency-based curriculum. Acad Med. 89(4):691.
   PubMed Web of Science ®Google Scholar
• Touchie C, Kinnear B, Schumacher D, Caretta-Weyer H, Hamstra SJ, Hart D, Gruppen L, Ross S, Warm E, Ten Cate O, I, et al. 2021. On the validity of summative entrustment decisions. Med Teach. 43(7):780–787.
   PubMed Web of Science ®Google Scholar
• van der Schaaf M, et al. 2017. Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Educ Technol Res Dev. 65(2):359–380.
   Web of Science ®Google Scholar
• van der Vleuten C, Heeneman S, Schut S. 2020. Programmatic assessment: an avenue to a different assessment culture. In: Yudkowsky R, Park YS, Downing SM, editors. Assessent in Health Professions Education. 2nd edn. New York(NY): Routledge, p. 245–256.
   Google Scholar
• Van Der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. 2015. Twelve tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, et al. 2016. Entrusting observable practice activities and milestones over the 36 months of an internal medicine residency. Acad Med. 91(10):1398–1405.
   PubMed Web of Science ®Google Scholar
• Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, et al. 2014. Entrustment and mapping of observable practice activities for resident assessment. J Gen Intern Med. 29(8):1177–1182.
   PubMed Web of Science ®Google Scholar
• Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. 2016. “Sometimes the work just needs to be done”: socio-cultural influences on direct observation in medical training. Med Educ. 50(10):1054–1064.
   PubMed Web of Science ®Google Scholar
• Weller JM, Castanelli DJ, Chen Y, Jolly B. 2017. ‘Making robust assessments of specialist trainees’ workplace performance. Br J Anaesth. 118(2):207–214.
   PubMed Web of Science ®Google Scholar
• Weller JM, et al. 2014. Can i leave the theatre? A key to more reliable workplace-based assessment. British Journal of Anaesthesia. 112(6):1083–1091.
   PubMed Web of Science ®Google Scholar
• Williams RG, Chen XP, Sanfey H, Markwell SJ, Mellinger JD, Dunnington GL. 2014. The measured effect of delay in completing operative performance ratings on clarity and detail of ratings assigned. J Surg Educ. 71(6):e132–e138.
   PubMedGoogle Scholar
• Williams RG, George BC, Bohnen JD, Dunnington GL, Fryer JP, Klamen DL, Meyerson SL, Swanson DB, Mellinger JD. 2021. A proposed blueprint for operative performance training, assessment, and certification. Ann Surg. 273(4):701–708.
   PubMed Web of Science ®Google Scholar
• Young JQ, McClure M. 2020a. Fast, easy, and good: assessing entrustable professional activities in psychiatry residents with a mobile app. Acad Med. 95(10):1546–1549.
   PubMed Web of Science ®Google Scholar
• Young JQ, McClure M. 2020b. Fast, easy, and good: assessing entrustable professional activities in psychiatry residents with a mobile app. Acad Med. 95(10):210–219.
   Web of Science ®Google Scholar
• Young JQ, Sugarman R, Schwartz J, McClure M, O'Sullivan PS. 2020. A mobile app to capture EPA assessment data: utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement. Perspect Med Educ. 9(4):210–219.
   PubMed Web of Science ®Google Scholar
• Young JQ, Sugarman R, Schwartz J, O'Sullivan PS. 2020a. Faculty and resident engagement with a workplace-based assessment tool: use of implementation science to explore enablers and barriers. Acad Med. 95(12):1937–1944.
   PubMed Web of Science ®Google Scholar
• Young JQ, Sugarman R, Schwartz J, O'Sullivan PS. 2020b. Overcoming the challenges of direct observation and feedback programs: a qualitative exploration of resident and faculty experiences. Teach Learn Med. 32(5):541–551.
   PubMed Web of Science ®Google Scholar
• Yudkowsky R, Park YS, Downing SM, editors. 2020. Assessment in Health Professions Education. 2nd ed. New York: taylor & Francis.
   Google Scholar