Identifying the response process validity of clinical vignette-type multiple choice questions: An eye-tracking study

Abstract

Introduction

Clinical vignette-type multiple choice questions (CV-MCQs) are widely used in assessment and identifying the response process validity (RPV) of questions with low and high integration of knowledge is essential. Answering CV-MCQs of different levels of knowledge application and integration can be understood from a cognitive workload perspective and this can be identified by using eye-tracking. The aim of the pilot study was to identify the cognitive workload and RPV of CV-MCQs of different levels of knowledge application and integration by the use eye-tracking.

Methods

Fourteen fourth-year medical students answered a test with 40 CV-MCQs, which were equally divided into low-level and high-level complexity (knowledge application and integration). Cognitive workload was measured using screen-based eye tracking, with the number of fixations and revisitations for each area of interest.

Results

We found a higher cognitive workload for high-level complexity (M = 121.74) compared with lower-level complexity questions (M = 51.94) and also for participants who answered questions incorrectly (M = 94.31) compared with correctly (M = 79.36).

Conclusion

Eye-tracking has the potential to become a useful and practical approach for helping to identify the RPV of CV-MCQs. This approach can be used for improving the design and development of CV-MCQs, and to provide feedback to inform teaching and learning.

Practice points

• Clinical vignette- type multiple choice questions (CV-MCQs) are widely used in assessment.

• Identifying the response process validity (RPV) of CV-MCQs with low and high complexity is essential.

• Low and high complexity CV-MCQs have different cognitive workloads that can be practically identified using eye-tracking.

• Eye-tracking has the potential for identifying the RPV of CV-MCQs.

Keywords:

• Eye tracking
• cognitive workload
• medical education
• assessment
• multiple choice questions

Acknowledgements

The authors would like to thank Pedro Tadao Hamamoto Filho for helping with the graphics and being a friendly review. The authors would also like to thank the statistics department of the Faculty of Medical Sciences for the support to calculate effect sizes. The authors would like to thank the students than accept to participate in this research.

Ethical approval

This study was approved by the University Ethics Committee (CAAE number 28709220.4.0000.5404 and protocol number 3.941.717), and all methods were performed in accordance with the relevant guidelines and regulations. The participation in this research was voluntary and participants could withdraw from the research at any moment. Written informed consent was obtained from all participants.

Author contributions

FCSJ, EA and DC-F were responsible for the conception and design of the study. FCSJ collected the data. FCSJ and DC-F were responsible for the data analysis. All authors were responsible for the interpretation of data. FCSJ was responsible for the first draft. All authors were responsible for revising it critically, and final approval of the version.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Data availability statement

Since raw data is considered personal data under Brazilian law and we did not ask their consents to publicly publish the data, the datasets generated and/or analyzed during the current study are not publicly available but are available from the corresponding author on reasonable request.

Additional information

Funding

This research was partially funded by FAPESP – São Paulo Research Foundation [Young Investigator Grant number 2018/15642-1] awarded to Dario Cecilio-Fernandes and Francisco Carlos Specian Junior [grant number 2020/01671-0]. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Notes on contributors

Francisco Carlos Specian Junior

Francisco Carlos Specian Junior, MD, School of Medical Sciences, University of Campinas, Campinas, São Paulo, Brazil.

Thiago Martins Santos

Thiago Martins Santos, MD, PhD, Department of Internal Medicine, School of Medical Sciences, University of Campinas, Campinas, São Paulo, Brazil.

John Sandars

John Sandars, MB, MSc, MD, MRCP, MRCGP, FAcad, MEd, Health Research Institute, Edge Hill University, Ormskirk, UK.

Eliana Martorano Amaral

Eliana Martorano Amaral, MD, PhD, Department of Obstetrics & Gynecology, University of Campinas, Campinas, São Paulo, Brazil.

Dario Cecilio-Fernandes

Dario Cecilio-Fernandes, MSc, PhD, AFAMEE, Department of Medical Psychology and Psychiatry, School of Medical Sciences, University of Campinas, Campinas, São Paulo, Brazil.