Context Matters: Understanding the Effect of Usage Contexts on Users’ Modality Selection in Multimodal Systems

Abstract

Users often select different modalities in multimodal systems based on context and function. This study aimed to investigate the impact of context on users’ modality selection in a multimodal system. A modality selection task was presented through the developed multimodal system, where the task’s difficulty was controlled based on the number of syllables and touches. The experiments were examined in four usage contexts (baseline, watching, reading, and driving), and the selected modality, physical and mental effort, and satisfaction were measured. A regression model for predicting modality selection was established using binomial logistic regression. The results showed that voice usage increased in the order of baseline, watching, reading, and driving contexts. It was revealed that these changes were attributable to differences in physical and mental interaction efforts of modalities according to contexts. Our finding provides valuable insights into users’ modality selection in different contexts, which could inform the design of more efficient multimodal systems.

Keywords:

• Multimodal
• modality selection
• usage context
• touch interaction
• voice modality

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Min Chul Cha

Min Chul Cha is a Postdoctoral Researcher in Industrial Engineering at Yonsei University, Seoul, Korea. He received his PhD from Yonsei University in 2023. His research interests include multimodal interface, voice user interaction, automotive user interface, and smart devices.

Yong Gu Ji

Yong Gu Ji is a Professor in the Department of Industrial Engineering at Yonsei University, where he directs the Interaction Design Laboratory. He received his PhD in Human Factors/HCI from Purdue University. His research interests include usability/UX in smart devices and self-driving vehicles.