Understanding Pitfalls and Opportunities of Applying Heuristic Evaluation Methods to VR Training Systems: An Empirical Study

References

• Afacan, Y., & Erbug, C. (2009). An interdisciplinary heuristic evaluation method for universal building design. Applied Ergonomics, 40(4), 731–744.
   PubMed Web of Science ®Google Scholar
• Almohaimeed, N., Al-Wabil., & A., Fazal-e-Amin. (2017). Virtual reality-based surgical simulation training usability heuristics. Journal of Medical Imaging and Health Informatics, 7(6), 1338–1345. https://doi.org/10.1166/jmihi.2017.2200
   Web of Science ®Google Scholar
• Alonso-Rıos, D., Mosqueira-Rey, E., & Moret-Bonillo, V. (2018). A systematic and generalizable approach to the heuristic evaluation of user interfaces. International Journal of Human–Computer Interaction, 34(12), 1169–1182. https://doi.org/10.1080/10447318.2018.1424101
   Web of Science ®Google Scholar
• Ashtari, N., Bunt, A., McGrenere, J., Nebeling, M., Chilana, P. (2020). Creating augmented and virtual reality applications: Current practices, challenges, and opportunities. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–13). https://doi.org/10.1145/3313831.3376722
   Google Scholar
• Baniasadi, T., Ayyoubzadeh, S., & Mohammadzadeh, N. (2020). Challenges and practical considerations in applying virtual reality in medical education and treatment. Oman Medical Journal, 35(3), e125. https://doi.org/10.5001/omj.2020.43
   PubMedGoogle Scholar
• Braun, V., & Clarke, V. (2012). Thematic analysis. American Psychological Association.
   Google Scholar
• Buttussi, F., & Chittaro, L. (2017). Effects of different types of virtual reality display on presence and learning in a safety training scenario. IEEE Transactions on Visualization and Computer Graphics, 24(2), 1063–1076. https://doi.org/10.1109/TVCG.2017.2653117
   PubMed Web of Science ®Google Scholar
• Chen, S., & Macredie, R. (2005). The assessment of usability of electronic shopping: A heuristic evaluation. International Journal of Information Management, 25(6), 516–532. https://doi.org/10.1016/j.ijinfomgt.2005.08.008
   Web of Science ®Google Scholar
• Clarke, V., Braun, V., & Hayfield, N. (2015). Thematic analysis. In J. A. Smith (Ed.), Qualitative psychology: A practical guide to research methods (pp. 222–248). SAGE Publications.
   Google Scholar
• Cockton, G., & Woolrych, A. (2001). Understanding inspection methods: Lessons from an assessment of heuristic evaluation. People and Computers XV—Interaction without Frontiers (pp. 171–191).
   Google Scholar
• Dey, A., Billinghurst, M., Lindeman, R., & Swan, J. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37. https://doi.org/10.3389/frobt.2018.00037
   PubMed Web of Science ®Google Scholar
• Dong, J., Li, D., Ozcan, K., Wan, D., Jiang, W., & Chen, Y. (2022). Development of CryoVR, a virtual reality training system for hands-on cryoEM operations. Acta Crystallographica Section D, 78(7). 903–910. https://doi.org/10.1107/S2059798322005654
   Google Scholar
• Dong, J., Zhang, J., Ma, X., Ren, P., Qian, Z., & Chen, Y. (2019). Virtual reality training with passive haptic feedback for CryoEM sample preparation. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 892–893). https://doi.org/10.1109/VR.2019.8797918
   Google Scholar
• Duncan, I., Miller, A., & Jiang, S. (2012). A taxonomy of virtual worlds usage in education. British Journal of Educational Technology, 43(6), 949–964. https://doi.org/10.1111/j.1467-8535.2011.01263.x
   Web of Science ®Google Scholar
• Fernández, J., Macıas, J. (2021). Heuristic-based usability evaluation support: A systematic literature review and comparative study. In Proceedings of the XXI International Conference on Human Computer Interaction (pp. 1–9).
   Google Scholar
• Følstad, A., Law, E., Hornbæk, K. (2012). Analysis in practical usability evaluation: A survey study. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2127–2136). https://doi.org/10.1145/2207676.2208365
   Google Scholar
• Gabbard, J., Hix, D., & Swan, J. (1999). User-centered design and evaluation of virtual environments. IEEE Computer Graphics and Applications, 19(6), 51–59. https://doi.org/10.1109/38.799740
   Web of Science ®Google Scholar
• Hartson, R., & Pyla, P. (2012). The UX Book: Process and guidelines for ensuring a quality user experience. Elsevier.
   Google Scholar
• Hepperle, D., Dienlin, T., & Wölfel, M. (2021). Reducing the human factor in virtual reality research to increase reproducibility and replicability. In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (pp. 100–105). https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00030
   Google Scholar
• Hermawati, S., & Lawson, G. (2016). Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? Applied Ergonomics, 56, 34–51. https://doi.org/10.1016/j.apergo.2015.11.016
   PubMed Web of Science ®Google Scholar
• Hvannberg, E. T., Law, E. L.-C., & Lárusdóttir, M. K. (2007). Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers, 19(2), 225–240. https://doi.org/10.1016/j.intcom.2006.10.001
   Web of Science ®Google Scholar
• Jerald, J. (2015). The VR book: Human-centered design for virtual reality. Morgan & Claypool.
   Google Scholar
• Joyce, A. (2021). 10 Usability Heuristics Applied to Virtual Reality. Nielsen Norman Group. https://www.nngroup.com/articles/usability-heuristics-virtual-reality/
   Google Scholar
• Kabassi, K., Amelio, A., Komianos, V., & Oikonomou, K. (2019). Evaluating museum virtual tours: The case study of Italy. Information, 10(11), 351. https://doi.org/10.3390/info10110351
   Google Scholar
• Karre, S., Mathur, N., & Reddy, Y. (2020). Understanding usability evaluation setup for VR products in industry: A review study. ACM SIGAPP Applied Computing Review, 19(4), 17–27. https://doi.org/10.1145/3381307.3381309
   Google Scholar
• Kim, Y., Rhiu, I., & Yun, M. (2020). A systematic review of a virtual reality system from the perspective of user experience. International Journal of Human–Computer Interaction, 36(10), 893–910. https://doi.org/10.1080/10447318.2019.1699746
   Web of Science ®Google Scholar
• Krauß, V., Boden, A., Oppermann, L., Reiners, R. (2021). Current practices, challenges, and design implications for collaborative ar/vr application development. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–15).
   Google Scholar
• Langevin, R., Lordon, R., Avrahami, T., Cowan, B., Hirsch, T., Hsieh, G. (2021). Heuristic evaluation of conversational agents. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–15).
   Google Scholar
• Lavery, D., Cockton, G., & Atkinson, M. (1997). Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16(4–5), 246–266. https://doi.org/10.1080/014492997119824
   Web of Science ®Google Scholar
• Law, E., Hvannberg, E. (2004). Analysis of strategies for improving and estimating the effectiveness of heuristic evaluation. Proceedings of the Third Nordic Conference on Human-Computer Interaction (pp. 241–250).
   Google Scholar
• Lee, E., Wong, K., & Fung, C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education, 55, 1424–1442.
   Web of Science ®Google Scholar
• Liberatore, M., & Wagner, W. (2021). Virtual, mixed, and augmented reality: A systematic review for immersive systems research. Virtual Reality, 25(3), 773–799. https://doi.org/10.1007/s10055-020-00492-0
   Web of Science ®Google Scholar
• Ling, C., & Salvendy, G. (2005). Extension of heuristic evaluation method: A review and reappraisal. Ergonomia, 27(3), 179–197.
   Google Scholar
• MacArthur, C., Grinberg, A., Harley, D., Hancock, M. (2021). You’re making me sick: A systematic review of how virtual reality research considers gender & cybersickness. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–15).
   Google Scholar
• Mankoff, J., Dey, A., Hsieh, G., Kientz, J., Lederer, S., Ames, M. (2003). Heuristic evaluation of ambient displays. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 169–176).
   Google Scholar
• Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., & Weiskopf, D. (2020). Evaluating mixed and augmented reality: A systematic literature review. 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (2009–2019) (pp. 438–451).
   Google Scholar
• Molina-Carmona, R., Pertegal-Felices, M., Jimeno-Morenilla, A., & Mora-Mora, H. (2018). Virtual reality learning activities for multimedia students to enhance spatial ability. Sustainability, 10(4), 1074. https://doi.org/10.3390/su10041074
   Web of Science ®Google Scholar
• Mulhall, A. (2003). In the field: Notes on observation in qualitative research. Journal of Advanced Nursing, 41(3), 306–313. https://doi.org/10.1046/j.1365-2648.2003.02514.x
   PubMed Web of Science ®Google Scholar
• Muñoz, R., Barcelos, T., Chalegre, V. (2011). Defining and validating virtual worlds usability heuristics. 2011 30th International Conference of the Chilean Computer Science Society (pp. 171–178).
   Google Scholar
• Murtza, R., Monroe, S., Youmans, R. (2017). Heuristic evaluation for virtual reality systems. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61 (pp. 2067–2071).
   Google Scholar
• Nebeling, M., Speicher, M., Wang, X., Rajaram, S., Hall, B., Xie, Z., Raistrick, A., Aebersold, M., Happ, E., Wang, J. (2020). MRAT: The mixed reality analytics toolkit. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–12). https://doi.org/10.1145/3313831.3376330
   Google Scholar
• Nielsen, J., Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 249–256). https://doi.org/10.1145/97243.97281
   Google Scholar
• Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 373–380). https://doi.org/10.1145/142750.142834
   Google Scholar
• Nielsen, J. (1994). Usability engineering Morgan Kaufmann.
   Google Scholar
• Nielsen, J. (1995). How to conduct a heuristic evaluation. Nielsen Norman Group 1, 8.
   Google Scholar
• Oliveira, E., Simões, F., Correia, W. (2017). Heuristics Evaluation and Improvements for Low-Cost Virtual Reality. 2017 19th Symposium on Virtual and Augmented Reality (SVR) (pp. 178–187).
   Google Scholar
• Pettersson, I., Lachner, F., Frison, A., Riener, A., Butz, A. (2018). A bermuda triangle? A review of method application and triangulation in user experience evaluation. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–16).
   Google Scholar
• Pinelle, D., Wong, N., Stach, T. (2008). Heuristic evaluation for games: Usability principles for video game design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1453–1462).
   Google Scholar
• Quiñones, D., & Rusu, C. (2017). How to develop usability heuristics: A systematic literature review. Computer Standards & Interfaces, 53, 89–122. https://doi.org/10.1016/j.csi.2017.03.009
   Web of Science ®Google Scholar
• Quiñones, D., Rusu, C., & Rusu, V. (2018). A methodology to develop usability/user experience heuristics. Computer Standards & Interfaces, 59, 109–129. https://doi.org/10.1016/j.csi.2018.03.002
   Web of Science ®Google Scholar
• Rivu, R., Mäkelä, V., Prange, S., Rodriguez, S., Piening, R., Zhou, Y., Köhle, K., Pfeuffer, K., Abdelrahman, Y., & Hoppe, M. (2021). Remote VR studies – A framework for running virtual reality studies remotely via participant-owned HMDs. ArXiv Preprint ArXiv:2102.11207.
   Google Scholar
• Rusu, C., Muñoz, R., Roncagliolo, S., Rudloff, S., Rusu, V., Figueroa, A. (2011). Usability heuristics for virtual worlds. Proceedings of the Third International Conference on Advances in Future Internet, Ser (pp. 16–19).
   Google Scholar
• Schwind, V., Knierim, P., Haas, N., Henze, N. (2019). Using presence questionnaires in virtual reality. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–12).
   Google Scholar
• Smith, D. (2021). Flux VR: The development and validation of a heuristic checklist for virtual reality game design supporting immersion, presence, and flow [Doctoral dissertation]. Wichita State University.
   Google Scholar
• Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., Guo, Y., Harris, D., Ioannou, A., Jeong, K-a (., Konomi, S., Krömker, H., Kurosu, M., Lewis, J. R., Marcus, A., … Zhou, J. (2019). Seven HCI grand challenges. International Journal of Human–Computer Interaction, 35(14), 1229–1269. https://doi.org/10.1080/10447318.2019.1619259
   Web of Science ®Google Scholar
• Speicher, M., Hall, B., Nebeling, M. (2019). What is mixed reality?. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–15).
   Google Scholar
• Sutcliffe, A., & Gault, B. (2004). Heuristic evaluation of virtual reality applications. Interacting with Computers, 16(4), 831–849. https://doi.org/10.1016/j.intcom.2004.05.001
   Web of Science ®Google Scholar
• Sutcliffe, A., Poullis, C., Gregoriades, A., Katsouri, I., Tzanavari, A., & Herakleous, K. (2019). Reflecting on the design process for virtual reality applications. International Journal of Human–Computer Interaction, 35(2), 168–179. https://doi.org/10.1080/10447318.2018.1443898
   Web of Science ®Google Scholar
• Wang, W., Cheng, J., & Guo, J. (2019). Usability of virtual reality application through the lens of the user community: A case study. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–6). https://doi.org/10.1145/3290607.3312816
   Google Scholar
• Xie, B., Liu, H., Alghofaili, R., Zhang, Y., Jiang, Y., Lobo, F. D., Li, C., Li, W., Huang, H., Akdere, M., Mousas, C., & Yu, L.-F. (2021). A review on virtual reality skill training applications. Frontiers in Virtual Reality, 2, 645153. https://doi.org/10.3389/frvir.2021.645153
   Google Scholar