Using virtual standardized patients to accurately assess information gathering skills in medical students

Abstract

Introduction: Practicing a medical history using standardized patients is an essential component of medical school curricula. Recent advances in technology now allow for newer approaches for practicing and assessing communication skills. We describe herein a virtual standardized patient (VSP) system that allows students to practice their history taking skills and receive immediate feedback.

Methods: Our VSPs consist of artificially intelligent, emotionally responsive 3D characters which communicate with students using natural language. The system categorizes the input questions according to specific domains and summarizes the encounter. Automated assessment by the computer was compared to manual assessment by trained raters to assess accuracy of the grading system.

Results: Twenty dialogs chosen randomly from 102 total encounters were analyzed by three human and one computer rater. Overall scores calculated by the computer were not different than those provided by the human raters, and overall accuracy of the computer system was 87%, compared with 90% for human raters. Inter-rater reliability was high across 19 of 21 categories.

Conclusions: We have developed a virtual standardized patient system that can understand, respond, categorize, and assess student performance in gathering information during a typical medical history, thus enabling students to practice their history-taking skills and receive immediate feedback.

Acknowledgments

Individuals with interest in evaluating the Virtual Standardized Patient System for their own use should contact the corresponding author.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Glossary

Virtual Standardized Patient: Avatar representation of a human standardized patient that can communicate using natural language.

Objective Structured Clinical Exam (OSCE): Formal structured assessment of history taking and/or physical exam skills.

Convolutional Neural Network: A type of artificial neural network that discovers patterns in data that are useful for classifying inputs into one of a set of predefined categories. When applied to text, they can be used for Natural Language Understanding applications such as paraphrase detection, question answering, and machine translation.

Additional information

Funding

This project was funded (in part) by a National Board of Medical Examiners (NBME) Edward J. Stemmler, MD Medical Education Research Fund grant [NBME 1112-064]. The project does not necessarily reflect NBME policy, and NBME support provides no official endorsement. This project was also supported by funding from the Department of Health and Human Services Health Resources and Services Administration [HRSA D56HP020687].

Notes on contributors

Kellen R. Maicher

Kellen Maicher, MFA, is a Learning and Development Consultant for the James Cancer Hospital and Solove Research Institute. His research interests include user experience design for virtual reality and affective virtual human development.

Laura Zimmerman

Laura Zimmerman, MS, is a Research Associate in the Department of Obstetrics and Gynecology.

Bruce Wilcox

Bruce Wilcox is the CEO of Brillig Understanding, Inc. and the author of the dialog management software, ChatScript.

Beth Liston

Beth Liston, MD, is an Assistant Professor of Hospital Medicine.

Holly Cronau

Holly Cronau, MD, is Emeritus Faculty in the Department of Family Medicine in the College of Medicine at the Ohio State University.

Allison Macerollo

Allison Macerollo, MD, is an Assistant Professor and Director of Medical Education. She is also Director of Patients within Populations and Co-Director of the Primary Care Track (three-year medical school curriculum). Her interests include patient communication and care of the underserved.

Lifeng Jin

Lifeng Jin is a PhD student in the Department of Linguistics. His research interests include using Bayesian and deep learning models for question classification and cognitively-inspired language acquisition modeling.

Evan Jaffe

Evan Jaffe is a PhD student in the Department of Linguistics.

Michael White

Michael White, PhD, is an Associate Professor in the Department of Linguistics. His primary research interests are in natural language generation, paraphrasing and dialog systems.

Eric Fosler-Lussier

Eric Fosler-Lussier, PhD, is a Professor in the Department of Computer Science and Engineering.

William Schuler

William Schuler, PhD, is a Professor in the Department of Linguistics. His research interests involve using cognitive models of human sentence processing to improve artificial sentence processing systems.

David P. Way

David P. Way, MEd is a Senior Educational Research Associate in the Department of Emergency Medicine. His research interests involve survey methods, Rasch measurement, and psychometrics.

Douglas R. Danforth

Douglas Danforth, PhD, is an Associate Professor in the Department of Obstetrics and Gynecology and is the Academic Program Director for years 1 and 2 of the medical school curriculum. His research interests involve using virtual patients and virtual reality for medical education.