Context Matters: Understanding the Effect of Usage Contexts on Users’ Modality Selection in Multimodal Systems

References

• Arrabito, G. R., Ho, G., Aghaei, B., Burns, C., & Hou, M. (2015). Sustained attention in auditory and visual monitoring tasks: Evaluation of the administration of a rest break or exogenous vibrotactile signals. Human Factors, 57(8), 1403–1416. https://doi.org/10.1177/0018720815598433
   PubMed Web of Science ®Google Scholar
• Baxter, M., Bleakley, A., Edwards, J., Clark, L., Cowan, B. R., & Williamson, J. R. (2021). “You, Move There!”: Investigating the impact of feedback on voice control in virtual environments [Paper presentation]. CUI 2021–3rd Conference on Conversational User Interfaces, Bilbao (online), Spain. https://doi.org/10.1145/3469595.3469609
   Google Scholar
• Beckers, N., Schreiner, S., Bertrand, P., Reimer, B., Mehler, B., Munger, D., & Dobres, J. (2014). Comparing the demands of destination entry using Google glass and the Samsung Galaxy S4. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 2156–2160. https://doi.org/10.1177/1541931214581453
   Google Scholar
• Budiu, R. (2013). Interaction cost. https://www.nngroup.com/articles/interaction-cost-definition/
   Google Scholar
• Calisir, F., & Calisir, F. (2004). The relation of interface usability characteristics, perceived usefulness, and perceived ease of use to end-user satisfaction with enterprise resource planning (ERP) systems. Computers in Human Behavior, 20(4), 505–515. https://doi.org/10.1016/j.chb.2003.10.004
   Web of Science ®Google Scholar
• Card, S. K., Mackinlay, J. D., & Robertson, G. G. (1990, April). The design space of input devices [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Seattle, Washington, USA. https://doi.org/10.1145/97243.97263
   Google Scholar
• Card, S. K., Moran, T. P., & Newell, A. (1980). The keystroke-level model for user performance time with interactive systems. Communications of the ACM, 23(7), 396–410. https://doi.org/10.1145/358886.358895
   Web of Science ®Google Scholar
• Chen, X., & Tremaine, M. (2006). Patterns of multimodal input usage in non-visual information navigation [Paper presentation]. Proceedings of the Annual Hawaii International Conference on System Sciences, Kauai, HI, USA. https://doi.org/10.1109/HICSS.2006.377
   Google Scholar
• Cherubini, M., Anguera, X., Oliver, N., & de Oliveira, R. (2009). Text versus speech [Paper presentation]. Proceedings of the 11th International Conference on Human–Computer Interaction with Mobile Devices and Services, Bonn, Germany. https://doi.org/10.1145/1613858.1613860
   Google Scholar
• Detjen, H., Geisler, S., & Schneegass, S. (2020, October). Maneuver-based control interventions during automated driving: Comparing touch, voice, and mid-air gestures as input modalities [Paper presentation]. 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada. https://doi.org/10.1109/SMC42975.2020.9283431
   Google Scholar
• Ferrand, L. (2000). Reading aloud polysyllabic words and nonwords: The syllabic length effect reexamined. Psychonomic Bulletin & Review, 7(1), 142–148. https://doi.org/10.3758/BF03210733
   PubMed Web of Science ®Google Scholar
• Findlater, L., & McGrenere, J. (2008). Impact of screen size on performance, awareness, and user satisfaction with adaptive graphical user interfaces [Paper presentation]. Proceeding of the Twenty-Sixth Annual CHI Conference on Human Factors in Computing Systems–CHI, ’08, Florence, Italy. https://doi.org/10.1145/1357054.1357249
   Google Scholar
• Foley, M., Casiez, G., & Vogel, D. (2020). Comparing smartphone speech recognition and touchscreen typing for composition and transcription [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Honolulu, HI, USA. https://doi.org/10.1145/3313831.3376861
   Google Scholar
• Fujimura, O. (1975). Syllable as a unit of speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(1), 82–87. https://doi.org/10.1109/TASSP.1975.1162631
   Google Scholar
• Garde, A. H., Laursen, B., Jørgensen, A. H., & Jensen, B. R. (2002). Effects of mental and physical demands on heart rate variability during computer work. European Journal of Applied Physiology, 87(4–5), 456–461. https://doi.org/10.1007/s00421-002-0656-7
   PubMed Web of Science ®Google Scholar
• Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52(C), 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
   Google Scholar
• Hauptmann, A. G., & Rudnicky, A. (1990). A comparison of speech and typed input. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania. https://doi.org/10.3115/116580.116652
   Google Scholar
• Hoffmann, F., Tyroller, M.-I., Wende, F., & Henze, N. (2019). User-defined interaction for smart homes [Paper presentation]. Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia, Pisa, Italy. https://doi.org/10.1145/3365610.3365624
   Google Scholar
• Hua, Z., & Ng, W. L. (2010). Speech recognition interface design for in-vehicle system [Paper presentation]. Proceedings of the 2nd International Conference on Automotive User Interfaces and Interactive Vehicular Applications–AutomotiveUI ’10, AutomotiveUI, Pittsburgh, Pennsylvania, USA. https://doi.org/10.1145/1969773.1969780
   Google Scholar
• Huang, J., Qi, M., Mao, L., An, M., Ji, T., & Han, R. (2021). User-defined gestures for mid-air interaction: A comparison of upper limb muscle activity, wrist kinematics, and subjective preference. International Journal of Human–Computer Interaction, 37(16), 1516–1537. https://doi.org/10.1080/10447318.2021.1898825
   Web of Science ®Google Scholar
• Hwangbo, H., Yoon, S. H., Jin, B. S., Han, Y. S., & Ji, Y. G. (2013). A study of pointing performance of elderly users on smartphones. International Journal of Human-Computer Interaction, 29(9), 604–618. https://doi.org/10.1080/10447318.2012.729996
   Web of Science ®Google Scholar
• ISO. (1998). ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs): Part 11: Guidance on usability. https://www.iso.org/standard/16883.html
   Google Scholar
• Jameson, A., & Klöckner, K. (2005). User multitasking with mobile multimodal systems. In W. Minker, D. Bühler, & L. Dybkjær (Eds.), Spoken multimodal human–computer dialogue in mobile environments. Text, Speech and Language Technology (Vol. 28). Springer, Dordrecht. https://doi.org/10.1007/1-4020-3075-4_19
   Google Scholar
• Jeon, M., Gable, T. M., Davison, B. K., Nees, M. A., Wilson, J., & Walker, B. N. (2015). Menu navigation with in-vehicle technologies: Auditory menu cues improve dual task performance, preference, and workload. International Journal of Human-Computer Interaction, 31(1), 1–16. https://doi.org/10.1080/10447318.2014.925774
   Web of Science ®Google Scholar
• Kim, J., Jeong, M., & Lee, S. C. (2019). Why did this voice agent not understand me? [Paper presentation]. Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, Utrecht, Netherlands. https://doi.org/10.1145/3349263.3351513
   Google Scholar
• Kim, M. J., Hwangbo, H., & Ji, Y. G. (2020). Comparing flat and edge-screen smartphones operated on a one-hand-only basis: A video observation in laboratory settings. International Journal of Human-Computer Interaction, 36(18), 1756–1767. https://doi.org/10.1080/10447318.2020.1785153
   Web of Science ®Google Scholar
• Kim, Y., Reza, M., McGrenere, J., & Yoon, D. (2021). Designers characterize naturalness in voice user interfaces: Their goals, practices, and challenges [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Yokohama, Japan. https://doi.org/10.1145/3411764.3445579
   Google Scholar
• Kocaballi, A. B., Laranjo, L., & Coiera, E. (2019). Understanding and measuring user experience in conversational interfaces. Interacting with Computers, 31(2), 192–207. https://doi.org/10.1093/iwc/iwz015
   Web of Science ®Google Scholar
• Laureiti, C., Cordella, F., Di Luzio, F. S., Saccucci, S., Davalli, A., Sacchetti, R., & Zollo, L. (2017). Comparative performance analysis of M-IμEMG and voice user interfaces for assistive robots [Paper presentation]. IEEE International Conference on Rehabilitation Robotics, London, UK. https://doi.org/10.1109/ICORR.2017.8009380
   Google Scholar
• Lee, K. M., & Lai, J. (2005). Speech versus touch: A comparative study of the use of speech and DTMF keypad for navigation. International Journal of Human-Computer Interaction, 19(3), 343–360. https://doi.org/10.1207/s15327590ijhc1903_4
   Web of Science ®Google Scholar
• Lee, S. C., Yoon, S. H., & Ji, Y. G. (2019). Modeling task completion time of in-vehicle information systems while driving with keystroke level modeling. International Journal of Industrial Ergonomics, 72, 252–260. https://doi.org/10.1016/j.ergon.2019.06.001
   Web of Science ®Google Scholar
• Lemmelä, S. (2008). Selecting optimal modalities for multimodal interaction in mobile and pervasive environments. In Pervasive 2008 Workshop Proceedings (Sydney, Australia, May 18, 2008). IMUx (improved mobile user experience) (pp. 208–217).
   Google Scholar
• Lemmelä, S., Vetek, A., Mäkelä, K., & Trendafilov, D. (2008). Designing and evaluating multimodal interaction for mobile contexts. [Paper presentation]. Proceedings of the 10th International Conference on Multimodal Interfaces, Chania, Crete, Greece. https://doi.org/10.1145/1452392.1452447
   Google Scholar
• Liu, X., & Thomas, G. W. (2017, May). Gesture interfaces: Minor change in effort, major impact on appeal [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Denver, Colorado, USA. https://doi.org/10.1145/3025453.3025513
   Google Scholar
• Malaia, E. A., & Wilbur, R. B. (2020). Syllable as a unit of information transfer in linguistic communication: The entropy syllable parsing model. Wiley Interdisciplinary Reviews: Cognitive Science, 11(1), 1–16. https://doi.org/10.1002/wcs.1518
   Web of Science ®Google Scholar
• McGee, M. (2004). Master usability scaling: Magnitude estimation and master scaling applied to usability measurement. [Paper presentation]. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria. https://doi.org/10.1145/985692.985735
   Google Scholar
• Mustonen, T., Olkkonen, M., & Häkkinen, J. (2004, June). Examining mobile phone text legibility while walking [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Vienna, Austria. https://doi.org/10.1145/985921.986034
   Google Scholar
• Park, S., Kyung, G., Choi, D., Yi, J., Lee, S., Choi, B., & Lee, S. (2019). Effects of display curvature and task duration on proofreading performance, visual discomfort, visual fatigue, mental workload, and user satisfaction. Applied Ergonomics, 78, 26–36. https://doi.org/10.1016/j.apergo.2019.01.014
   PubMed Web of Science ®Google Scholar
• Perakakis, M., & Potamianos, A. (2008). A study in efficiency and modality usage in multimodal form filling systems. IEEE Transactions on Audio, Speech, and Language Processing, 16(6), 1194–1206. https://doi.org/10.1109/TASL.2008.2001389
   Google Scholar
• Qiu, L., & Benbasat, I. (2005). Online consumer trust and live help interfaces: The effects of text-to-speech voice and three-dimensional avatars. International Journal of Human-Computer Interaction, 19(1), 75–94. https://doi.org/10.1207/s15327590ijhc1901_6
   Web of Science ®Google Scholar
• Reicherts, L., Rogers, Y., Capra, L., Wood, E., Duong, T. D., & Sebire, N. (2022). It’s good to talk: A comparison of using voice versus screen-based interactions for agent-assisted tasks. ACM Transactions on Computer-Human Interaction, 29(3), 1–41. https://doi.org/10.1145/3484221
   Web of Science ®Google Scholar
• Ruan, S., Wobbrock, J. O., Liou, K., Ng, A., & Landay, J. A. (2018). Comparing speech and keyboard text entry for short messages in two languages on touchscreen phones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4), 1–23. https://doi.org/10.1145/3161187
   Google Scholar
• Sauro, J., & Dumas, J. S. (2009). Comparison of three one-question, post-task usability questionnaires [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Boston, MA, USA. https://doi.org/10.1145/1518701.1518946
   Google Scholar
• Schaffer, S., Jöckel, B., Wechsung, I., Schleicher, R., & Möller, S. (2011, August). Modality selection and perceived mental effort in a mobile application. In Proceedings of the 12th Annual Conference of the International Speech Communication Association, Interspeech 2011, Florence, Italy. https://doi.org/10.21437/Interspeech.2011-599
   Google Scholar
• Schartmüller, C., & Riener, A. (2022). Multimodal error correction for speech-to-text in a mobile office automated vehicle: Results from a remote study [Paper presentation]. International Conference on Intelligent User Interfaces, Proceedings IUI, Helsinki, Finland. https://doi.org/10.1145/3490099.3511131
   Google Scholar
• Sendlneier, W. F. (1995). Feature, phoneme, syllable or word: How is speech mentally represented? Phonetica, 52(3), 131–143. https://doi.org/10.1159/000262128
   PubMed Web of Science ®Google Scholar
• Simpson, G. B., & Kang, H. (2004). Syllable processing in alphabetic Korean. Reading and Writing, 17(1/2), 137–151. https://doi.org/10.1023/B:READ.0000013808.65933.a1
   Google Scholar
• Suhm, B., Myers, B., & Waibel, A. (1999). Model-based and empirical evaluation of multimodal interactive error correction [Paper presentation]. Conference on Human Factors in Computing Systems–Proceedings, Pittsburgh, Pennsylvania, USA. https://doi.org/10.1145/302979.303165
   Google Scholar
• Tsimhoni, O., Smith, D., & Green, P. (2004). Address entry while driving: Speech recognition versus a touch-screen keyboard. Human Factors, 46(4), 600–610. https://doi.org/10.1518/hfes.46.4.600.56813
   PubMed Web of Science ®Google Scholar
• Turner, C. J., Chaparro, B. S., & He, J. (2021). Typing on a smartwatch while mobile: A comparison of input methods. Human Factors, 63(6), 974–986. https://doi.org/10.1177/0018720819891291
   PubMed Web of Science ®Google Scholar
• Wechsung, I., Naumann, A., & Möller, S. (2010, October 1–2). The influence of the usage mode on subjectively perceived quality. In Spoken Dialogue Systems for Ambient Environments: Second International Workshop on Spoken Dialogue Systems Technology, IWSDS 2010, Gotemba, Shizuoka, Japan. Proceedings (pp. 188–193). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-16202-2_20
   Google Scholar
• Wickens, C. D. (2008). Multiple resources and mental workload. Human Factors, 50(3), 449–455. https://doi.org/10.1518/001872008X288394
   PubMed Web of Science ®Google Scholar
• Wu, J., Chang, C. C., Boyle, L. N., & Jenness, J. (2015, January). Impact of in-vehicle voice control systems on driver distraction: Insights from contextual interviews. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 1583–1587. https://doi.org/10.1177/1541931215591342
   Google Scholar
• Yoon, S. H., Lee, S. C., & Ji, Y. G. (2021). Modeling takeover time based on non-driving-related task attributes in highly automated driving. Applied Ergonomics, 92, 103343. https://doi.org/10.1016/j.apergo.2020.103343
   PubMed Web of Science ®Google Scholar
• Zhao, M., Cui, W., Ramakrishnan, I., Zhai, S., & Bi, X. (2021). Voice and touch based error-tolerant multimodal text editing and correction for smartphones [Paper presentation]. The 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA. https://doi.org/10.1145/3472749.3474742
   Google Scholar
• Zijlstra, F. R. (1993, January). Efficiency in work behaviour: A design approach for modern tools (pp. 1–186). Delft University Press. http://www.csa.com/partners/viewrecord.php?requester=gs&collection=TRD&recid=N9516953AH
   Google Scholar