A Toolkit for Prototyping Tabletop-Centric Cross-Device Interaction

ABSTRACT

Different types of devices are used in our daily lives, and each type has its own advantages and disadvantages. Cross-device interaction that involves multiple devices can potentially overcome each device’s inherent disadvantages. Without a toolkit, it is time consuming to develop cross-device interaction. Focusing on a horizontal large display (called a tabletop), this paper proposes a generic toolkit for prototyping tabletop-centric cross-device applications that involve a large display and multiple smartphones. In our toolkit, a user uses a smartphone as a look-through lens for browsing and selecting objects on a tabletop, and remotely manipulates the selected object with multimodal feedback. Based on the above interaction style, our toolkit formalizes the development of cross-device interaction as defining a mobile interface on each tabletop interface object. Since our toolkit encapsulates the interface distribution and synchronization through message passing, interface developers can focus on developing cross-device interaction by designing a mobile interface in response to a selected tabletop interface object. In order to demonstrate the versatility and design space of our toolkit, six design issues, i.e., data transfer, personalization, user interface composition, authentication, localized & private feedback, and input expressiveness, were discussed through a set of sample applications.

Notes

1. Without losing generality, we implemented the framework on the Microsoft SUR 40 tabletop and Android smartphones.

Additional information

Funding

This work is in part supported by NSF under grant #1722913.

Notes on contributors

Zheng Huang

Zheng Huang is a PhD candidate of Software Engineering at Department of Computer Science, North Dakota State University. His research interests include Human Computer Interaction and Mobile Computing. Especially, he aims at designing the intuitive cross-device interface to improve the user experience across different digital devices.

Jun Kong

Jun Kong received the B.S. degree from the Huazhong University of Science and Technology in 1998, the M.S. degree from Shanghai Jiao Tong University in 2001, and the Ph.D. degree from the University of Texas at Dallas in 2005, all in computer science. He is a full Professor of computer science with North Dakota State University, Fargo, USA. His research and teaching interests include human–computer interaction, software engineering and brain-computer interface.