Integrating Mobile Multimodal Interactions based on Programming By Demonstration​

ABSTRACT

Mobile Multimodal Interaction aims at exploiting complementary aspects of human communication capacities and new mobile sensors. Recently, most mobile applications are limited to a basic interaction modality, namely touchscreen, which is subject to restricted interaction under certain situations. In this paper, we present On-the-Fly Interaction Editor (OFIE), an application that allows mobile end-users to define and integrate sensor-based unimodal and multimodal input interactions in their already installed applications according to their contexts. OFIE is based on the Event-Condition-Action rules and Programming By Demonstration approach that allows end-users to demonstrate their expected action simply by performing it on the application’s interface. We evaluated OFIE through a controlled user study. Our evaluation involves 15 participants distributed on 4 groups based on their programming experience. Each participant was invited to integrate six input interactions (three multimodal inputs). The initial results show that end-users are able to successfully integrate sensor-based input interactions using OFIE.

Notes

1. www.twofortyfouram.com

2. https://bit.ly/2TlnIKf

3. This scenario is also shown in the following video https://bit.ly/39DaH4O.

4. Kitchen Stories Application: https://bit.ly/34VXyQw

5. https://goo.gl/EEUSfu

6. https://github.com/nisrulz/sensey

7. This tutorial is also shown in the following video: shorturl.at/ptvM7

Additional information

Funding

This work was supported by the Smart Service and Interface for Ubiquitous Computing project Grant Agreement #C00L07ES220120120180002 under the University Training Research Program (PRFU) [C00L07ES220120120180002].

Notes on contributors

Zouhir Bellal

Zouhir Bellal received the B.Sc. and M.Sc. degree in computer science from Djillali Liabes University, Sidi Bel Abbes, Algeria in 2016, and 2014, respectively. He is currently pursuing the Ph.D. degree at the Ecole Superieure en Informatique, Sidi Bel Abbes, Algeria. His research interests include Human-Computer Interaction and End-User Programming.

Nadia Elouali

Nadia Elouali is a senior lecturer at ESI-SBA (Ecole Superieure en Informatique - Sidi Bel Abbes, Algeria). She is member of the SOC (Service Oriented Computing) team at the LabRI-SBA laboratory. Her research interests include, Human-computer interaction, Multimodal interaction, Persuasive technology and Deep Learning.

Sidi Mohamed Benslimane

Sidi Mohamed Benslimane is a Full Professor at ESI-SBA, Algeria. He received his PhD degree in computer science from Sidi Bel Abbes University in 2007. He is currently Head of Research Team 'Service Oriented Computing’ at the LabRI Laboratory. His research interests include, semantic web and Internet of Things.

Cengiz Acarturk

Cengiz Acarturk is a faculty member at the Cognitive Science Graduate Program at the Informatics Institute, Middle East Technical University, Turkey. His current research interests cover eye tracking applications, eye movements in reading, human-computer interaction, and human factors in cybersecurity.