Impact of rating demands on rater-based assessments of clinical competence

References

• Huwendiek S, Mennin S, Dern P et al (2010) Expertise, needs and challanges of medical educators: results of an international web survey. Medical Teacher 32 (11): 912–18.
   PubMed Web of Science ®Google Scholar
• Downing SM (2004) Reliability: on the reproducibility of assessment data. Medical Education 38 (9): 1006–12.
   PubMed Web of Science ®Google Scholar
• Downing SM (2005) Threats to the validity of clinical teaching assessments: what about rater error? Medical Education 39 (4): 353–5.
   PubMed Web of Science ®Google Scholar
• Haber RJ and Avins AL (1994) Do ratings on the American Board of Internal Medicine Resident Evaluation Form detect differences in clinical competence? Journal of General Internal Medicine 9 (3): 140–5.
   PubMed Web of Science ®Google Scholar
• Lurie SJ, Mooney CJ and Lyness JM (2009) Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Academic Medicine 84 (3): 301–9.
   PubMed Web of Science ®Google Scholar
• Thompson WG, Lipkin M Jr, Gilbert DA, Guzzo RA and Roberson L (1990) Evaluating evaluation: assessment of the American Board of Internal Medicine Resident Evaluation Form. Journal of General Internal Medicine 5 (3): 214–17.
   PubMed Web of Science ®Google Scholar
• Williams RG, Klamen DA, and McGaghie WC (2003) Special article: Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine 15 (4): 270–92.
   PubMed Web of Science ®Google Scholar
• Yeates P, O'Neill P, Mann K and Eva KW (2012) Seeing the same thing differently. Advances in Health Sciences Education 18 (3): 325–41.
   PubMed Web of Science ®Google Scholar
• Green ML and Holmboe E (2010) Perspective: the ACGME toolbox: half empty or half full? Academic Medicine 85 (5): 787–90.
   PubMed Web of Science ®Google Scholar
• Tavares W and Eva KW (2012) Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education 18 (2): 291–303.
   PubMed Web of Science ®Google Scholar
• Govaerts M, van de Wiel M, Schuwirth L, van der Vleuten C and Muijtjens A (2013) Workplace-based assessment: rater's performance theories and constructs. Advances in Health Sciences Education 18 (3): 375–96.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Conforti L, Bernabeo E, Lobst W and Holmboe E (2011) Opening the black box of clinical skills assessment via observation: a conceptual model. Medical Education 45 (10): 1048–60.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, McIlroy J, Oulanova O, Eva KW and Regehr G (2010) Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Academic Medicine 85 (5): 780–6.
   PubMed Web of Science ®Google Scholar
• Crossley J, Johnson G, Booth J and Wade W (2011) Good questions, good answers: construct alignment improves the performance of workplace based assessment scales. Medical Education 45 (6): 560–9.
   PubMed Web of Science ®Google Scholar
• Eva KW and Hodges BD (2012) Scylla or Charybdis? Can we navigate between objectification and judgement in assessment? Medical Education 46 (9): 914–19.
   PubMed Web of Science ®Google Scholar
• Hodges B (2013) Assessment in the post-psychometric era: learning to love the subjective and collective. Medical Teacher 35 (7): 564–8.
   PubMed Web of Science ®Google Scholar
• Norcini J, Blank L, Duffy F and Fortna G (2003) The mini-CEX: a method for assessing clinical skills. Annals of Internal Medicine 138 (6): 476–81.
   PubMed Web of Science ®Google Scholar
• Baddeley (1992) Working Memory. Science, 255 (5044): 556–9.
   PubMed Web of Science ®Google Scholar
• Navon D and Gopher D (1979) On the economy of the human-processing system. Psychological Review 86 (3): 214–55.
   Web of Science ®Google Scholar
• Salvucci DD and Taatgen NA (2008) Threaded cognition: an integrated theory of concurrent multitasking. Psychological Review 115 (1): 101–30.
   PubMed Web of Science ®Google Scholar
• Wickens CD (2002) Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science 3 (2): 159–77.
   Google Scholar
• Tamblyn RM, Klass DJ, Gail K, Schnabl K and Kopelow ML (1991) Sources of unreliability and bias in standardized-patient rating. Teaching and Learning in Medicine 3 (2): 74–85.
   Google Scholar
• Vu Nu Viet, Marcy M, Colliver JA, Verhulst SJ, Travis TA and Barrows HS (1992) Standardized (simulated) patients' accuracy in recording clinical performance check-list items. Medical Education 26 (2): 99–104.
   PubMed Web of Science ®Google Scholar
• Byrne A, Tweed N and Halligan C (2014) A pilot study of the mental workload of objective structured clinical examiners. Medical Education 48 (3): 262–7.
   PubMed Web of Science ®Google Scholar
• Norcini J (2003) Peer assessment of competence. Medical Education 37 (6): 539–43.
   PubMed Web of Science ®Google Scholar
• Dannefer EF, Henson LC, Bierer SB et al (2005) Peer assessment of professional competence. Medical Education 39 (7): 713–22.
   PubMed Web of Science ®Google Scholar
• Boet S, Bould MD, Sharma B et al (2013) Within-team debriefing versus instructor led debriefing for simulation based education: a randomized controlled trial. Annals of Surgery 258 (1): 53–8.
   PubMed Web of Science ®Google Scholar
• Pelgrim E, Kramer A, Mokkink H, Van den Elsen L, Grol R and Van der Vleuten C (2011) In-training assessment using direct observation of single-patient encounters: a literature review. Advances in Health Sciences Education 16 (1): 131–42.
   PubMed Web of Science ®Google Scholar
• Tavares W, Boet S, Theriault R, Mallette T and Eva KW (2012) Global rating scale for the assessment of paramedic clinical competence. Prehospital Emergency Care 17 (1): 57–67.
   PubMed Web of Science ®Google Scholar
• Streiner DL and Norman GR (2008) Health Measurement Scales: a practical guide to their development and use (4e). Oxford University Press: New York.
   Google Scholar
• Chi M (1997) Quantifying qualitative analyses of verbal data: a practical guide. Journal of the Learning Sciences 6 (3): 271–315.
   Web of Science ®Google Scholar
• Byrne AJ, Oliver M, Bodger O et al (2010) Novel method of measuring the mental workload of anaesthetists during clinical practice. British Journal of Anaesthesia 105 (6): 767–71.
   PubMed Web of Science ®Google Scholar
• Paas F, Tuovinen JE, Tabbers H and Van Gerven PWM (2003) Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist 38 (1): 63–71.
   Web of Science ®Google Scholar
• Hart SG (2006) NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 904–8). Santa Monica CA: Human Factors & Ergonomics.
   Google Scholar
• Hart SG and Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Human Mental Workload. Elsevier: Amsterdam, pp. 139–83.
   Google Scholar
• Brennan RL (2001) Generalizability Theory. Springer Verlag: New York, USA.
   Google Scholar
• Bloch R and Norman G (2012) Generalizability theory for the perplexed: a practical introduction and guide: AMEE Guide No. 68. Medical Teacher 34 (11): 960–92.
   PubMed Web of Science ®Google Scholar
• Alexander AL, Nygren TE and Vidulich MA (2000) Examining the relationship between mental workload and situation awareness in a simulated air combat task AFRL-HE-WPTR-2000–0094. Air Force Research Laboratory, Human Effectiveness Directorate, Wright-Patterson AFB, OH.
   Google Scholar
• Cantin V, Lavallière M, Simoneau M and Teasdale N (2009) Mental workload when driving in a simulator: effects of age and driving complexity. Accident Analysis & Prevention 41 (4): 763–71.
   PubMed Web of Science ®Google Scholar
• Tsang PS and Vidulich MA (2006) Mental Workload and Situation Awareness. Wiley: Hoboken, NJ.
   Google Scholar
• Yurko YY, Scerbo MW, Prabhu AS, Acker CE and Stefanidis D (2010) Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simulation in Healthcare 5 (5): 267–71.
   PubMedGoogle Scholar
• Zheng B, Cassera MA, Martinec DV, Spaun GO and Swanström LL (2010) Measuring mental workload during the performance of advanced laparoscopic tasks. Surgical Endoscopy 24 (1): 45–50.
   PubMed Web of Science ®Google Scholar
• LaMantia J, Rennie W, Risucci DA et al (1999) Interobserver variability among faculty in evaluations of residents clinical skills. Academic Emergency Medicine 6 (1): 38–44.
   PubMed Web of Science ®Google Scholar
• Wickens CD and Carswell C (2006). Handbook of Human Factors and Ergonomics, Third Edition. Hoboken, NJ: Wiley
   Google Scholar
• Fletcher G, Flin R, McGeorge P, Glavin R, Maran N and Patey R (2003) Anaesthetists' non-technical skills (ANTS): evaluation of a behavioural marker system. British Journal of Anaesthesia 90 (5): 580–8.
   PubMed Web of Science ®Google Scholar
• Kim J, Neilipovitz D, Cardinal P, Chiu M and Clinch J (2006) A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Critical Care Medicine 34 (8): 2167–74.
   PubMed Web of Science ®Google Scholar
• Norcini J, Blank L, Arnold G and Kimball H (1995) The mini-CEX (clinical evaluation exercise): a preliminary investigation. Annals of Internal Medicine 123 (10): 795–9.
   PubMed Web of Science ®Google Scholar
• Cook DA, Zendejas B, Hamstra SJ, Hatala R and Brydges R (2014) What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Advances in Health Sciences Education 19 (2): 233–50.
   PubMed Web of Science ®Google Scholar
• Tavares W and Eva KW (2013) Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education 18 (2): 291–303.
   PubMed Web of Science ®Google Scholar
• Holmboe ES (2004) Faculty and the observation of trainees' clinical skills: problems and opportunities. Academic Medicine 79 (1): 16–22.
   PubMed Web of Science ®Google Scholar