Identifying the response process validity of clinical vignette-type multiple choice questions: An eye-tracking study

References

• Agarwal PK. 2019. Retrieval practice & Bloom’s taxonomy: do students need fact knowledge before higher order learning? J Educ Psychol. 111(2):189–209.
   Web of Science ®Google Scholar
• Anderson JR. 1982. Acquisition of cognitive skill. Psychol Ver. 89:369–406.
   Google Scholar
• Anderson JR, Bothell D, Byrne MD, Douglass S, Lebiere C, Qin YL. 2004. An integrated theory of the mind. Psychol Ver. 111:1036–1060.
   Google Scholar
• Bartolomé J, Garaizar P, Bastida L. 2020. Validating item response processes in digital competence assessment through eye-tracking techniques. In: García-Peñalvo FJ, García-Holgado A, editors. TEEM'20: Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality; Oct 21–23; Salamanca, Spain. New York (NY): Association for Computing Machinery.
   Google Scholar
• Bissell AN, Lemons PP. 2006. A new method for assessing critical thinking in the classroom. Bioscience. 56(1):66–72.
   Web of Science ®Google Scholar
• Bloom BS. 1956. Taxonomy of educational objectives: the classification of education goals by a committee of college and university Examiners. Ann Arbor (MI): David McKay.
   Google Scholar
• Bordage G, Page G. 2018. The key-features approach to assess clinical decisions: validity evidence to date. Adv Health Sci Educ. 23(5):1005–1036.
   PubMed Web of Science ®Google Scholar
• Borys M, Plechawska-Wojcik M, Wawrzyk M, Wesolowska K. 2017. Classifying cognitive workload using eye activity and EEG features in arithmetic tasks. Inf Softw Techno. 756:90–105.
   Google Scholar
• Cecilio-Fernandes D, Kerdijk W, Bremers AJ, Aalders W, Tio RA. 2018. Comparison of the level of cognitive processing between case-based items and non-case-based items on the Interuniversity Progress Test of Medicine in the Netherlands. J Educ Eval Health Prof. 15:28.
   PubMed Web of Science ®Google Scholar
• Cecilio-Fernandes D, Kerdijk W, Jaarsma ADC, Tio RA. 2016. Development of cognitive processing and judgments of knowledge in medical students: analysis of progress test results. Med Teach. 38(11):1125–1129.
   PubMed Web of Science ®Google Scholar
• Chen F, Zhou J, Wang Y, Yu K, Arsha S, Khawaji A, Conway D. 2016. Robust multimodal cognitive load measurement. New York (NY): Springer Cham; p. 13–32.
   Google Scholar
• Cohen J. 1969. Statistical power analysis for the behavioral sciences. 1st ed. New York (NY): Academic Press.
   Google Scholar
• Connor DM, Durning SJ, Rencic JJ. 2020. Clinical reasoning as a core competency. Acad Med. 95(8):1166–1171.
   PubMed Web of Science ®Google Scholar
• Cook DA, Beckman TJ. 2006. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 119(2):166.e7–166.e16.
   PubMedGoogle Scholar
• Crowe A, Dirks C, Wenderoth MP. 2008. Biology in Bloom: implementing Bloom’s taxonomy to enhance student learning in biology. LSE. 7(4):368–381.
   Google Scholar
• Daniel M, Rencic J, Durning SJ, Holmboe E, Santen SA, Lang V, Ratcliffe T, Gordon D, Heist B, Lubarsky S, et al. 2019. Clinical reasoning assessment methods: a scoping review and practical guidance. Acad Med. 94(6):902–912.
   PubMed Web of Science ®Google Scholar
• Dias RD, Ngo-Howard MC, Boskovski MT, Zenati MA, Yule SJ. 2018. Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. Br J Surg. 105(5):491–501.
   PubMed Web of Science ®Google Scholar
• Duchowski A, Mccormick BH. 1995. Simple multiresolution approach for representing multiple regions of interest (ROIs). In: Wu LT, editor. 1995 Conference on Visual Communications and Imaging Processing (1995 VCIP); May 24–26; Taipei, Taiwan: SPIE; p. 175–186.
   Google Scholar
• Dunn OJ. 1961. Multiple comparisons among means. J Am Stat Assoc. 56(293):52–64.
   Web of Science ®Google Scholar
• Goldberg J, Kotval X. 1999. Computer interface evaluation using eye movements: methods and constructs. Int J Ind Ergon. 24(6):631–645.
   Web of Science ®Google Scholar
• Gruppen LD,. 2017. Clinical reasoning: defining it, teaching it, assessing it, studying it. WestJEM. 18(1):4–7.
   Google Scholar
• Hamamoto PT, Silva E, Ribeiro ZMT, Hafner M, Cecilio-Fernandes D, Bicudo AM. 2020. Relationships between Bloom’s taxonomy, judges’ estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study. Sao Paulo Med J. 138(1):33–39.
   PubMed Web of Science ®Google Scholar
• Heist BS, Gonzalo JD, Durning S, Torre D, Elnicki DM. 2014. Exploring clinical reasoning strategies and test-taking behaviors during clinical vignette style multiple-choice examinations: a mixed methods study. J Grad Med Educ. 6(4):709–714.
   PubMedGoogle Scholar
• Holmqvist K, Örbom SL, Hooge IT, Niehorster DC, Alexander RG, Andersson R, Benjamins JS, Blignaut P, Brouwer AM, Chuang LL. 2022. Eye tracking: empirical foundations for a minimal reporting guideline. Behav Res. 55:1–53.
   Google Scholar
• Just MA, Carpenter PA. 1980. A theory of reading: from eye fixations to comprehension. Psychol Rev. 87(4):329–354.
   PubMed Web of Science ®Google Scholar
• Korbach A, Brünken R, Park B. 2017. Measurement of cognitive load in multimedia learning: a comparison of different objective measures. Instr Sci. 45(4):515–536.
   Web of Science ®Google Scholar
• Korbach A, Brünken R, Park B. 2018. Differentiating different types of cognitive load: a comparison of different measures. Educ Psychol Rev. 30(2):503–529.
   Google Scholar
• Krathwohl DR. 2002. A revision of Bloom’s taxonomy: an overview. Theory Pract. 41(4):212–218.
   Web of Science ®Google Scholar
• Lang VJ, Berman NB, Bronander K, Hingle S, Holthouser A, Leizman D, Packer CD, Park YS, Vu TR, Yudkowsky R. 2019. Validity evidence for a brief online key features examination in the internal medicine clerkship. Acad Med. 94(2):259–266.
   PubMed Web of Science ®Google Scholar
• Mayer RE. 2010. Applying the science of learning to medical education. Med Educ. 44(6):543–549.
   PubMed Web of Science ®Google Scholar
• Naismith LM, Cheung JJ, Ringsted C, Cavalcanti RB. 2015. Limitations of subjective cognitive load measures in simulation‐based procedural training. Med Educ. 49(8):805–814.
   PubMed Web of Science ®Google Scholar
• Nakayama M, Takahasi Y. 2008. Estimation of certainty for responses to multiple-choice questionnaires using eye movements. ACM Trans Multimedia Comput Commun Appl. 5(2):1–18.
   Web of Science ®Google Scholar
• Olejnik S, Algina J. 2003. Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychol Methods. 8(4):434–447.
   PubMed Web of Science ®Google Scholar
• Padilla GJL, Benítez BI. 2014. Validity evidence based on response processes. Psicothema. 26(1):136–144.
   PubMed Web of Science ®Google Scholar
• Quen MTZ, Mountstephens J, Teh YG, Teo J. 2021. Medical image interpretation training with a low‐cost eye tracking and feedback system: a preliminary study. Healthc Technol Lett. 8(4):97–103.
   Web of Science ®Google Scholar
• Shankar S, St‐Onge C, Young ME. 2022. When I say… response process validity evidence. Med Educ. 56(9):878–880.
   PubMed Web of Science ®Google Scholar
• Sweller J. 2018. Measuring cognitive load. Perspect Med Educ. 7(1):1–2.
   PubMed Web of Science ®Google Scholar
• Tien T, Pucher PH, Sodergren MH, Sriskandarajah K, Yang GZ, Darzi A. 2014. Eye tracking for skills assessment and training: a systematic review. J Surg Res. 191(1):169–178.
   PubMed Web of Science ®Google Scholar
• Tippey KG, Longnecker MT. 2016. An ad hoc method for computing pseudo-effect size for mixed models. In: Alonzo L, Mendez L, editors. Proceedings of South Central SAS Users Group Forum; Nov 6–8; San Antonio, Texas. Dallas (TX): South Central SAS Users Group (SCSUG). p. 1–7.
   Google Scholar
• Valliappan N, Dai N, Steinberg E, He J, Rogers K, Ramachandran V, Xu P, Shojaeizadeh M, Guo L, Kohlhoff K, et al. 2020. Accelerating eye movement research via accurate and affordable smartphone eye tracking. Nat Commun. 11(1):1–12.
   PubMed Web of Science ®Google Scholar
• Wingelaar TT, Wagter JM, Arnold AE. 2012. Students’ educational needs for clinical reasoning in first clerkships. Perspect Med Educ. 1(2):56–66.
   PubMedGoogle Scholar
• Wood TJ, Cunnington JP, Norman GR. 2000. Assessing the measurement properties of a clinical reasoning exercise. Teach Learn Med. 12(4):196–200.
   PubMed Web of Science ®Google Scholar
• Yamada K, Kise K, Augereau O. 2017. Estimation of confidence based on eye gaze: an application to multiple-choice questions. In: Lee SC, Takayama L, Truong K, editors. Proceedings of UbiComp '17: The 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers; Sep 11–15; Maui, Hawaii. New York (NY): Association for Computing Machinery. p. 217–220.
   Google Scholar
• Yaneva V, Clauser BE, Morales A, Paniagua M. 2021. Using eye‐tracking data as part of the validity argument for multiple‐choice questions: a demonstration. J Educ Meas. 58(4):515–537.
   Web of Science ®Google Scholar