Empirical Research in Mid-Air Interaction: A Systematic Review

ABSTRACT

Mid-air interaction is a distinct style of natural HCI (Human-Computer Interaction). In mid-air interaction, users make use of their whole body—with a strong focus on hands—and apply gestures, postures, and movements to interact with digital content on distant displays or remote devices. Although the idea of exploiting body movements and gestures in HCI is not new, mid-air interaction is now possible due to the availability of reasonably priced depth cameras and sensors. This article presents a systematic review of empirical research in mid-air interaction, based on a corpus of 104 publications from 2011 to 2018. The review includes: (a) a retrospective on mid-air interaction, which outlines its historical development and clarifications on important concepts; (b) current application domains of mid-air interaction; (c) user requirements methods, focusing on gesture elicitation studies; (d) dimensions of prototyping and design; (e) empirical evaluation methods and issues; (f) a discussion on several trends and challenges for further research and development.

Notes

1. Vincent John Vincent, personal web site. http://www.vjvincent.com/ .

2. Pattie Maes’ presentation in TED. Meet the Sixth Sense Interaction: https://www.ted.com/talks/pattie_maes_demos_the_sixth_sense .

3. BMW’s 7-Series ‘gesture controls’ work pretty well, USA TODAY: https://www.usatoday.com/story/money/cars/2016/05/16/bmws-7-series-gesture-controls-work-pretty-well/32613369/ .

4. https://www.polygon.com/2017/10/25/16543192/kinect-discontinued-microsoft-announcement .

5. https://orbbec3d.com/product-astra/ .

6. https://software.intel.com/en-us/realsense .

7. https://www.mendeley.com/.

8. https://developer.microsoft.com/en-us/windows/kinect.

9. https://www.leapmotion.com.

10. https://optitrack.com/.

11. https://www.vicon.com/.

12. http://www.intersense.com/.

13. https://www.myo.com/.

14. Non-critical body parts: i.e. non-tiring movements of the whole body without extensions of body ends.

15. Public display (wall sized) = 4; Media = 3; Drone = 3; Desktop = 3; Maps (2D) = 2; In-vehicle = 2; Smart watch = 2; Haptic feedback = 2; Mobile phone = 1; Web = 1; Gaming = 1; Factory automation = 1; AR = 1; Drawing = 1.

16. One study also involved children.

Additional information

Notes on contributors

Panayiotis Koutsabasis

Panayiotis Koutsabasis is assistant professor at the University of the Aegean, Greece. His research interests include: HCI and design, natural user interfaces, virtual reality, HCI education, user experience, usability and accessibility, in cultural heritage, cooperation and design support, with interactive technologies like the web, mobile devices and digital 3D environments.

Panagiotis Vogiatzidakis

Panagiotis Vogiatzidakis is a PhD candidate at the University of the Aegean, Greece. His research is on empirical methods for design and evaluation of mid-air interactions.