EchoTap: Non-Verbal Sound Interaction with Knock and Tap Gestures

Abstract

The growing demand for highly accessible interaction technologies to effectively interact with smart devices has led to the increasing popularity of voice user interfaces (VUIs). However, VUIs face interpretation challenges stemming from the variability of natural language input, such as speech clarity issues, linguistic variability, and speech impediments. As an alternative, non-verbal sound-based interaction techniques emerge as highly advantageous for smart device control, mitigating the inherent challenges of VUIs. In this article, we introduce EchoTap, a novel audio interface that harnesses the distinctive sound responses generated by knock and tap gestures on target objects. Employing deep neural networks, EchoTap recognizes both the type and location of these gestures based on their unique sound signatures. Through offline evaluation, EchoTap demonstrated competitive classification accuracy (88% on average) and localization precision (93% on average). Moreover, a user study involving 12 participants validated EchoTap’s practical effectiveness and user-friendliness in real-world scenarios. This study highlights EchoTap’s potential for various daily interaction contexts and discusses further design implications for leveraging auditory interfaces based on simple gestures.

Keywords:

• Non-verbal sound
• sound interface
• gesture classification
• gesture localization
• usability

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was financially supported by Seoul National University of Science and Technology.

Notes on contributors

Jae-Yeop Jeong

Jae-Yeop Jeong received a B.E. degree in Computer Engineering from Kumoh National Institute of Technology (KIT), Gumi, South Korea, in 2021. He is currently a Ph.D. student in the Department of Data Science at the Seoul National University of Science and Technology, Seoul, South Korea. He is interested in human-computer interaction.

Daun Kim

Daun Kim received a B.E. degree in Industrial Engineering from Seoul National University of Science and Technology, Seoul, South Korea, in 2023. She is currently a M.S. student in the Department of Data Science at the Seoul National University of Science and Technology, Seoul, South Korea. She is interested in human-computer interaction.

Jin-Woo Jeong

Jin-Woo Jeong received his Ph.D. in Computer Science and Engineering from Hanyang University, South Korea, in 2013. Since 2024, he has been an Associate Professor at the Department of Industrial Engineering, Seoul National University of Science and Technology, South Korea. His research interests include AI, human-computer interaction, and its applications.