Understanding Pitfalls and Opportunities of Applying Heuristic Evaluation Methods to VR Training Systems: An Empirical Study

Abstract

The usability of virtual reality (VR) training applications is crucial for their success, but examining the usability in the early development stages remains challenging. A realistic and plausible solution would be revisiting and reconciling Heuristics Evaluation (HE) methods among the most widely used usability inspection methods in the human-computer interaction (HCI) domain. While research on studying and using HE methods is growing within the VR domain, few studies have considered the novel VR environment challenges new requirements for fitting HE methods to the context and applying them effectively. To this end, we conducted a user study with 14 evaluators using the standard HE methods to complete two HE sessions for a VR training application. We identified five critical challenges that evaluators encountered in the HE process by observing and interviewing them. Based on our findings, we discuss the importance of considering an easy-to-use heuristic set, how we can facilitate the HE procedures in the VR context, and the opportunities for developing HE-supporting tools.

Acknowledgement

We thank the participants of our study for sharing their insights and challenges with us.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Partial funding for this research was provided by the National Institutes of Health [grant No. 1R25EY029127-01].

Notes on contributors

Xiaolei Guo

Xiaolei Guo is a Ph.D. student in the Department of Computer Graphics Technology at Purdue University. Her research focuses on Virtual Reality and Human-Computer Interactions.

Kushal Kumar Nerella

Kushal Kumar Nerella is a graduate student in the Department of Computer Graphics Technology at Purdue University. He has been working with Dr. Chen on VR system evaluation research.

Jiahui Dong

Jiahui Dong is a Ph.D. student in the Department of Computer Graphics Technology at Purdue University. His research focuses on VR training systems and VR interaction techniques.

Zhenyu Qian

Zhenyu Qian is a Full Professor of Interaction Design in Industrial Design at Purdue University. Dr. Qian is interested in studying and developing cognitive systems to enrich knowledge, employing interdisciplinary research methodologies to improve design quality, and adopting innovative technologies to accommodate user experience.

Yingjie Chen

Yingjie Chen is an Associate Professor in the Computer Graphics Technology Department at Purdue University. His research covers interdisciplinary domains of Information Visualization, Virtual Reality, and Human-Computer Interactions.