Abstract
Introduction: Practicing a medical history using standardized patients is an essential component of medical school curricula. Recent advances in technology now allow for newer approaches for practicing and assessing communication skills. We describe herein a virtual standardized patient (VSP) system that allows students to practice their history taking skills and receive immediate feedback.
Methods: Our VSPs consist of artificially intelligent, emotionally responsive 3D characters which communicate with students using natural language. The system categorizes the input questions according to specific domains and summarizes the encounter. Automated assessment by the computer was compared to manual assessment by trained raters to assess accuracy of the grading system.
Results: Twenty dialogs chosen randomly from 102 total encounters were analyzed by three human and one computer rater. Overall scores calculated by the computer were not different than those provided by the human raters, and overall accuracy of the computer system was 87%, compared with 90% for human raters. Inter-rater reliability was high across 19 of 21 categories.
Conclusions: We have developed a virtual standardized patient system that can understand, respond, categorize, and assess student performance in gathering information during a typical medical history, thus enabling students to practice their history-taking skills and receive immediate feedback.
Acknowledgments
Individuals with interest in evaluating the Virtual Standardized Patient System for their own use should contact the corresponding author.
Disclosure statement
The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.
Glossary
Virtual Standardized Patient: Avatar representation of a human standardized patient that can communicate using natural language.
Objective Structured Clinical Exam (OSCE): Formal structured assessment of history taking and/or physical exam skills.
Convolutional Neural Network: A type of artificial neural network that discovers patterns in data that are useful for classifying inputs into one of a set of predefined categories. When applied to text, they can be used for Natural Language Understanding applications such as paraphrase detection, question answering, and machine translation.
Additional information
Funding
Notes on contributors
Kellen R. Maicher
Kellen Maicher, MFA, is a Learning and Development Consultant for the James Cancer Hospital and Solove Research Institute. His research interests include user experience design for virtual reality and affective virtual human development.
Laura Zimmerman
Laura Zimmerman, MS, is a Research Associate in the Department of Obstetrics and Gynecology.
Bruce Wilcox
Bruce Wilcox is the CEO of Brillig Understanding, Inc. and the author of the dialog management software, ChatScript.
Beth Liston
Beth Liston, MD, is an Assistant Professor of Hospital Medicine.
Holly Cronau
Holly Cronau, MD, is Emeritus Faculty in the Department of Family Medicine in the College of Medicine at the Ohio State University.
Allison Macerollo
Allison Macerollo, MD, is an Assistant Professor and Director of Medical Education. She is also Director of Patients within Populations and Co-Director of the Primary Care Track (three-year medical school curriculum). Her interests include patient communication and care of the underserved.
Lifeng Jin
Lifeng Jin is a PhD student in the Department of Linguistics. His research interests include using Bayesian and deep learning models for question classification and cognitively-inspired language acquisition modeling.
Evan Jaffe
Evan Jaffe is a PhD student in the Department of Linguistics.
Michael White
Michael White, PhD, is an Associate Professor in the Department of Linguistics. His primary research interests are in natural language generation, paraphrasing and dialog systems.
Eric Fosler-Lussier
Eric Fosler-Lussier, PhD, is a Professor in the Department of Computer Science and Engineering.
William Schuler
William Schuler, PhD, is a Professor in the Department of Linguistics. His research interests involve using cognitive models of human sentence processing to improve artificial sentence processing systems.
David P. Way
David P. Way, MEd is a Senior Educational Research Associate in the Department of Emergency Medicine. His research interests involve survey methods, Rasch measurement, and psychometrics.
Douglas R. Danforth
Douglas Danforth, PhD, is an Associate Professor in the Department of Obstetrics and Gynecology and is the Academic Program Director for years 1 and 2 of the medical school curriculum. His research interests involve using virtual patients and virtual reality for medical education.