From lab to industry: lessons learned from the evaluation of augmented and virtual reality use cases in the Austrian manufacturing industry

References

• Akçayır, M., Akçayır, G., Pektaş, H. M., & Ocak, M. A. (2016). Augmented reality in science laboratories: The effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories. Computers in Human Behavior, 57, 334–27. https://doi.org/10.1016/j.chb.2015.12.054
   Web of Science ®Google Scholar
• Arendarski, B., Termath, W., & Mecking, P. (2008). Maintenance of complex machines in electric power systems using Virtual reality techniques. Conference Record of the 2008 IEEE International Symposium on Electrical Insulation, 483–487. https://doi.org/10.1109/ELINSL.2008.4570378
   Google Scholar
• Aschauer, A., Reisner-Kollmann, I., & Wolfartsberger, J. (2021). Creating an open-source augmented reality remote support tool for industry: Challenges and Learnings. Procedia Computer Science, 180, 269–279. https://doi.org/10.1016/j.procs.2021.01.164
   Google Scholar
• Aschenbrenner, D., Leutert, F., Çençen, A., Verlinden, J., Schilling, K., Latoschik, M., & Lukosch, S. (2019). Comparing human factors for augmented reality supported single-user and collaborative repair operations of industrial robots. Frontiers in Robotics and AI, 6, 6. https://doi.org/10.3389/frobt.2019.00037
   PubMed Web of Science ®Google Scholar
• Azuma, R. T. (1997). A survey of augmented reality. Presence Teleoperators & Virtual Environments, 6(4), 355–385. https://doi.org/10.1162/pres.1997.6.4.355
   Web of Science ®Google Scholar
• Bacca, J., Baldiris, S., Fabregat, R., Graf, S., & Kinshuk D. (2014). Augmented reality trends in education: A systematic review of research and applications. J Educ Technol Soc, 17, 133–149.
   Web of Science ®Google Scholar
• Becker, T. (2008). Prozesse in Produktion und supply chain optimieren. Springer. https://doi.org/10.1007/978-3-540-77556-0
   Google Scholar
• Biondi, F. N., Saberi, B., Graf, F., Cort, J., Pillai, P., & Balasingam, B. (2023). Distracted worker: Using pupil size and blink rate to detect cognitive load during manufacturing tasks. Applied Ergonomics, 106, 103867. https://doi.org/10.1016/j.apergo.2022.103867
   PubMed Web of Science ®Google Scholar
• Bottani, E., & Vignali, G. (2019). Augmented reality technology in the manufacturing industry: A review of the last decade. IISE Transactions, 51(3), 284–310. https://doi.org/10.1080/24725854.2018.1493244
   Web of Science ®Google Scholar
• Brooke, J. (2013). SUS - a retrospective. Journal of Usability Studies, 8(2), 29–40.
   Google Scholar
• Byvaltsev, S. (2020). Review of the features of augmented reality application in the training of operators and maintenance staff. IOP Conference Series: Materials Science and Engineering, 966, 012121. https://doi.org/10.1088/1757-899X/966/1/012121
   Google Scholar
• Chan, W., Hanks, P., Sakr, G., Zhang, M., Zuo, H., T, V. D. L., F, M. H., & Croft, E. (2022). Design and evaluation of an augmented reality head-mounted display interface for human robot teams collaborating in physically shared manufacturing tasks. J Hum Robot Interact, 11(3), 1–19. https://doi.org/10.1145/3524082
   Google Scholar
• Danielsson, O., Syberfeldt, A., Holm, M., & Thorvald, P. (2023). Integration of augmented reality smart glasses as assembly support: A framework implementation in a quick evaluation tool. International Journal of Manufacturing Research, 18(2), 144–164. https://doi.org/10.1504/ijmr.2023.131583
   Google Scholar
• Danielsson, O., Syberfeldt, A., Holm, M., & Wang, L. (2018). Operators perspective on augmented reality as a support tool in engine assembly. Procedia CIRP, 72, 45–50. https://doi.org/10.1016/j.procir.2018.03.153
   Google Scholar
• Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
   Web of Science ®Google Scholar
• de Souza Cardoso, L. F., Mariano, F. C. M. Q., & Zorzal, E. R. (2020). A survey of industrial augmented reality. Computers & Industrial Engineering, 139, 106159. https://doi.org/10.1016/j.cie.2019.106159
   Web of Science ®Google Scholar
• Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. E. I. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 5. https://doi.org/10.3389/frobt.2018.00037
   PubMed Web of Science ®Google Scholar
• Erlach, K. (2010). Wertstromdesign: Der Weg zur schlanken Fabrik. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-89867-2
   Google Scholar
• Fast-Berglund, Å., Gong, L., & Li, D. (2018). Testing and validating extended reality (xR) technologies in manufacturing. Procedia Manufacturing, 25, 31–38. https://doi.org/10.1016/j.promfg.2018.06.054
   Google Scholar
• Fischer, C., Rupprecht, P., & Schlund, S. (2023). Different approaches of conducting ergonomic assessment utilizing digital human models and motion capture in industrial site assembly. Intelligent Human Systems Integration (IHSI 2023) Integrating People and Intelligent Systems. https://doi.org/10.54941/ahfe1002854
   Google Scholar
• Funk, M., Bächler, A., Bächler, L., Kosch, T., Heidenreich, T., & Schmidt, A. (2017). Working with augmented reality?: A long-term analysis of in-situ instructions at the assembly workplace. Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, 222–229. https://doi.org/10.1145/3056540.3056548
   Google Scholar
• Gattullo, M., Dammacco, L., Ruospo, F., Evangelista, A., Fiorentino, M., Schmitt, J., & Uva, A. E. (2020). Design preferences on industrial augmented reality: A survey with potential technical writers. In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), virtual (pp. 172–177). IEEE.
   Google Scholar
• Gavish, N., Gutiérrez, T., Webel, S., Rodríguez, J., Peveri, M., Bockholt, U., & Tecchia, F. (2015). Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments, 23(6), 778–798. https://doi.org/10.1080/10494820.2013.815221
   Web of Science ®Google Scholar
• Häfner, P., Häfner, V., & Ovtcharova, J. (2013). Teaching methodology for virtual reality practical course in engineering education. Procedia Computer Science, 25, 251–260. https://doi.org/10.1016/j.procs.2013.11.031
   Google Scholar
• Hart, S. G. (2006). Nasa-task load index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9), 904–908. https://doi.org/10.1177/154193120605000909
   Google Scholar
• Helin, K., Kuula, T., Vizzi, C., Karjalainen, J., & Vovk, A. (2018). User experience of augmented reality system for astronaut’s manual work support. Front Robot AI, 5, 106. https://doi.org/10.3389/frobt.2018.00106
   PubMed Web of Science ®Google Scholar
• Helsper, E. J., & Eynon, R. (2010). Digital natives: Where is the evidence? British Educational Research Journal, 36(3), 503–520. https://doi.org/10.1080/01411920902989227
   Web of Science ®Google Scholar
• Hirsch-Kreinsen, H., ten Hompel, M., & Kretschmer, V. (2019). Digitalisierung industrieller Arbeit. In M. ten Hompel, B. Vogel-Heuser, & T. Bauernhansl (Eds.), Handbuch Industrie 4.0: Produktion, automatisierung und Logistik (pp. 1–18). Springer. https://doi.org/10.1007/978-3-662-45537-1_21-2
   Google Scholar
• Ho, P. T., Albajez, J. A., Santolaria, J., & Yagüe-Fabra, J. A. (2022). Study of augmented reality based manufacturing for futher integration of quality control 4.0: A systematic literature review. Applied Sciences, 12(4), 1961. https://doi.org/10.3390/app12041961
   Google Scholar
• Hoda, R., Noble, J., & Marshall, S. (2010). How much is just enough?: Some documentation patterns on agile projects. Proceedings of the 15th European Conference on Pattern Languages of Programs - EuroPLoP ’10, 1. https://doi.org/10.1145/2328909.2328926
   Google Scholar
• Hoedt, S., Claeys, A., An Landeghem, H., & Cottyn, J. (2016). Evaluation framework for Virtual training within mixed-model manual assembly. IFAC-Papersonline, 49(12), 261–266. https://doi.org/10.1016/j.ifacol.2016.07.614
   Google Scholar
• Holly, F., Zigart, T., Maurer, M., Wolfartsberger, J., Brunnhofer, M., Sorko, S. R., Moser, T., & Schlager, A. (2022). Gaining impact with mixed reality in industry – a sustainable approach. 2022 8th International Conference on Computer Technology Applications, 128–134. https://doi.org/10.1145/3543712.3543729
   Google Scholar
• Kadir, B. A., Broberg, O., & da Conceição, S. C. (2019). Current research and future perspectives on human factors and ergonomics in industry 4.0. Computers & Industrial Engineering, 137, 106004. https://doi.org/10.1016/j.cie.2019.106004
   Web of Science ®Google Scholar
• Khokhlovskiy, V., Oleynikov, V., Kostenko, D., Onufriev, V., & Potekhin, V. (2019). Modernisation of a production process using multicriteria optimisation logic and augmented reality. In B. Katalinic (Ed.), DAAAM Proceedings (1st ed., Vol. 1, pp. 0500–0507). DAAAM International Vienna. https://doi.org/10.2507/30th.daaam.proceedings.067
   Google Scholar
• Kim, M., Choi, S. H., Park, K.-B., & Lee, J. Y. (2019). User interactions for augmented reality smart glasses: A comparative evaluation of visual contexts and interaction gestures. Applied Sciences, 9(15), 3171. https://doi.org/10.3390/app9153171
   Google Scholar
• Kortum, P. T., & Bangor, A. (2013). Usability ratings for everyday products measured with the system usability scale. International Journal of Human-Computer Interaction, 29(2), 67–76. https://doi.org/10.1080/10447318.2012.681221
   Web of Science ®Google Scholar
• Kosch, T., Kettner, R., Funk, M., & Schmidt, A. (2016). Comparing tactile, auditory, and visual assembly error-feedback for workers with cognitive impairments. Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, 53–60. https://doi.org/10.1145/2982142.2982157
   Google Scholar
• Kostolani, D., Wollendorfer, M., & Schlund, S. (2022). ErgoMaps: Towards interpretable and accessible automated ergonomic analysis. 2022 IEEE 3rd International Conference on Human-Machine Systems (ICHMS), 1–7. https://doi.org/10.1109/ICHMS56717.2022.9980741
   Google Scholar
• Kugler, M., Bierwirth, M., Schaub, K., Sinn-Behrendt, A., Feith, A., Ghezel-Ahmadi, K., & Bruder, R. (2010). Ergonomie in der Industrie - aber wie? Handlungshilfe für den schrittweisen Aufbau eines einfachen Ergonomiemanagements (1. Aufl.). https://www.baua.de/DE/Themen/Arbeitsgestaltung-im-Betrieb/Physische-Belastung/Praevention/pdf/Muskel-Skelett-1.pdf?__blob=)
   Google Scholar
• Li, X., Yi, W., Chi, H.-L., Wang, X., & Chan, A. P. C. (2018). A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Automation in Construction, 86, 150–162. https://doi.org/10.1016/j.autcon.2017.11.003
   Web of Science ®Google Scholar
• Longo, L., & Federici, S. (2018). Experienced mental workload, perception of usability, their interaction and impact on task performance. PLOS ONE, 13(8), e0199661. https://doi.org/10.1371/journal.pone.0199661
   PubMed Web of Science ®Google Scholar
• Lovasz-Bukvova, H., Hölzl, M., Kormann-Hainzl, G., Moser, T., Zigart, T., & Schlund, S. (2021). Usability and task load of applications in augmented and Virtual reality. In M. Yilmaz, P. Clarke, R. Messnarz, & M. Reiner (Eds.), Systems, software and services process improvement (Vol. 1442, pp. 708–718). Springer. https://doi.org/10.1007/978-3-030-85521-5_48
   Google Scholar
• Ma, J., Jaradat, R., Ashour, O., Hamilton, M., Jones, P., & Dayarathna, V. L. (2018). Efficacy investigation of virtual reality teaching module in manufacturing system Design course. ASME J Mech Des, 141(1), 012002. https://doi.org/10.1115/1.4041428. January 2019.
   Web of Science ®Google Scholar
• Marino, E., Barbieri, L., Colacino, B., Fleri, A. K., & Bruno, F. (2021). An augmented reality inspection tool to support workers in industry 4.0 environments. Computers in Industry, 127, 103412. https://doi.org/10.1016/j.compind.2021.103412
   Web of Science ®Google Scholar
• Mark, B., Gualtieri, L., Rauch, E., Rojas, R., Buakum, D., & Matt, D. (2019). Analysis of user groups for assistance systems in production 4.0. https://doi.org/10.1109/IEEM44572.2019.8978907
   Google Scholar
• Mark, B., Rauch, E., & Matt, D. T. (2022). Systematic selection methodology for worker assistance systems in manufacturing. Computers & Industrial Engineering, 166, 107982. https://doi.org/10.1016/j.cie.2022.107982
   Web of Science ®Google Scholar
• Moser, T., Hohlagschwantner, M., Kormann-Hainzl, G., Pölzlbauer, S., & Wolfartsberger, J. (2019). Mixed reality applications in industry: Challenges and research areas. 11th International Conference SWQD, Vienna (pp. 95–105.
   Google Scholar
• Moser, T., Zigart, T., Kormann-Hainzl, G., & Lovasz-Bukvova, H. (2020). Assistenzsysteme der zukunft schon heute. Aktuelle anwendungsfälle von Mixed Reality in der Produktion. WINGBusiness, 53(2), 18–24. https://issuu.com/beablond/docs/heft_02_2020_end/18
   Google Scholar
• Niedermayr, D., & Wolfartsberger, J. (2021). Design and evaluation of a virtual training environment for industrial assembly tasks. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3862367
   Google Scholar
• Othman, M. K., Nogoibaeva, A., Leong, L. S., & Barawl, M., H. (2022). Usability evaluation of a virtual reality smartphone app for a living museum. Univ Access Inf Soc, 21(4), 995–1012. https://doi.org/10.1007/s10209-021-00820-4
   Web of Science ®Google Scholar
• Pringle, A., Hutka, S., Mom, J., van Esch, R., Heffernan, N., & Chen, P. (2019). Ethnographic study of a commercially available augmented reality HMD app for industry work instruction. Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, 389–397. https://doi.org/10.1145/3316782.3322752
   Google Scholar
• Quandt, M., Ait Alla, A., Meyer, L., & Freitag, M. (2017). Success factors for the development of augmented reality-based assistance systems for maintenance services. 7. WGP-Jahreskongress, Aachen (pp. 175–182).
   Google Scholar
• Quint, F., Loch, F., Orfgen, M., & Zühlke, D. (2016). A system architecture for assistance in manual tasks. Intelligent Environments. https://doi.org/10.3233/978-1-61499-690-3-43
   Google Scholar
• Rapaccini, M., Porcelli, I., Espíndola, D. B., & Pereira, C. E. (2014). Evaluating the use of mobile collaborative augmented reality within field service networks: The case of océ italia – canon group. Production and Manufacturing Research, 2(1), 738–755. https://doi.org/10.1080/21693277.2014.943430
   Google Scholar
• Ras, E., Wild, F., Stahl, C., & Baudet, A. (2017). Bridging the skills gap of workers in industry 4.0 by human performance augmentation tools: Challenges and roadmap. Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, 428–432. https://doi.org/10.1145/3056540.3076192
   Google Scholar
• Riener, R., & Harders, M. (2012). Virtual reality in medicine. Springer London.
   Google Scholar
• Roßmann, J. (2011). From space to the forest and to construction sites: Virtual testbeds pave the way for new technologies. In D. Ma, X. Fan, J. Gausemeier, & M. Grafe (Eds.), Virtual reality & augmented reality in industry (pp. 39–54). Springer.
   Google Scholar
• Rupprecht, P., Kueffner-Mccauley, H., Trimmel, M., Hornacek, M., & Schlund, S. (2022). Advanced adaptive spatial augmented reality utilizing dynamic in-situ projection in industrial site assembly. Procedia CIRP, 107, 937–942. https://doi.org/10.1016/j.procir.2022.05.088
   Google Scholar
• Sahu, C. K., Young, C., & Rai, R. (2021). Artificial intelligence (AI) in augmented reality (AR)-assisted manufacturing applications: A review. International Journal of Production Research, 59(16), 4903–4959. https://doi.org/10.1080/00207543.2020.1859636
   Web of Science ®Google Scholar
• Sampaio, A. (2010). Virtual reality technology used in civil Engineering Education~!2010-02-18~!2010-06-15~!2010-09-02~! The Open Virtual Reality Journal, 2(1), 18–25. https://doi.org/10.2174/1875323X01002010018
   Google Scholar
• Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Elsevier, Morgan Kaufmann.
   Google Scholar
• Schumann, M., Schenk, M., & Bluemel, E. (2011). Numerically controlled virtual models for commissioning, testing and training. In D. Ma, X. Fan, J. Gausemeier, & M. Grafe (Eds.), Virtual reality & augmented reality in industry (pp. 163–170). Springer.
   Google Scholar
• Sırakaya, M., & Alsancak Sırakaya, D. (2018). Trends in Educational augmented reality studies: A systematic review. Malaysian Online Journal of Educational Technology, 6(2), 60–74. https://doi.org/10.17220/mojet.2018.02.005
   Google Scholar
• Stone, R. J., Panfilov, P. B., & Shukshunov, V. E. (2011). Evolution of aerospace simulation: From immersive virtual reality to serious games. Proceedings of 5th International Conference on Recent Advances in Space Technologies - RAST2011, 655–662. https://doi.org/10.1109/RAST.2011.5966921
   Google Scholar
• Terhoeven, J., Schiefelbein, F.-P., & Wischniewski, S. (2018). User expectations on smart glasses as work assistance in electronics manufacturing. Procedia CIRP, 72, 1028–1032. https://doi.org/10.1016/j.procir.2018.03.060
   Google Scholar
• Vacchetti, L., Lepetit, V., Ponder, M., Papagiannakis, G., Fua, P., Thalmann, D., & Thalmann, N. M. (2004). A stable real-time AR framework for training and planning in industrial Environments. In S. K. Ong & A. Y. C. Nee (Eds.), Virtual and augmented reality applications in manufacturing (pp. 129–145). Springer London. https://doi.org/10.1007/978-1-4471-3873-0_8
   Google Scholar
• Wang, X., & Dunston, P. S. (2007). Design, strategies, and issues towards an augmented reality-based construction training platform. ITcon, 12, 363–380.
   Google Scholar
• Webel, S., Bockholt, U., & Keil, J. (2011). Design criteria for AR-Based training of maintenance and assembly tasks. In R. Shumaker (Ed.), Virtual and mixed reality—new trends (pp. 123–132). Springer.
   Google Scholar
• Wenk, N., Penalver-Andres, J., Buetler, K. A., Nef, T., Müri, R. M., & Marchal-Crespo, L. (2023). Effect of immersive visualization technologies on cognitive load, motivation, usability, and embodiment. Virtual Reality, 27(1), 307–331. https://doi.org/10.1007/s10055-021-00565-8
   PubMed Web of Science ®Google Scholar
• Zhou, Y., Ji, S., Xu, T., & Wang, Z. (2018). Promoting knowledge construction: A model for using virtual reality interaction to enhance learning. Procedia Computer Science, 130, 239–246. https://doi.org/10.1016/j.procs.2018.04.035
   Google Scholar
• Zigart, T. (2022). Entwicklung eines multikriteriellen Evaluierungsmodells für industrielle Assistenzsysteme [TU Wien]. https://doi.org/10.34726/HSS.2022.98440
   Google Scholar
• Zigart, T., & Schlund, S. (2020). Evaluation of augmented reality technologies in manufacturing – a literature review. In I. L. Nunes (Ed.), Advances in human factors and systems interaction (pp. 75–82). Springer International Publishing. https://doi.org/10.1007/978-3-030-51369-6_11
   Google Scholar
• Zigart, T., & Schlund, S. (2022). Ready for industrial use? A user study of spatial augmented reality in industrial assembly. 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct(ISMAR-Adjunct) , 60–65. https://doi.org/10.1109/ISMAR-Adjunct57072.2022.00022
   Google Scholar