Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Abstract

Background: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available.

Aims: We examined the validity of an instrument for assessing the perception of VP design by learners.

Methods: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis.

Results: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors.

Conclusion: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.

Acknowledgements

We thank all of the other former eVIP partners and VP experts (http://www.virtualpatients.eu/) who are not co-authors of this paper, namely, Chara Balasubramaniam, David A. Davies, Martin Fischer, Valentin Muntean, Terry Poulton, Uno Fors and Nabil Zary for their support in developing this instrument and Sarah Mannion de Hernandez for the English language correction of the manuscript. Furthermore, we would like to express our gratitude to all of the students who filled in the questionnaires.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.