Touching Products Virtually: Facilitating Consumer Mental Imagery with Gesture Control and Visual Presentation

Abstract

Gesture-based interaction has greatly changed the way in which we interact with online products by allowing users to control digital systems with hand movements. This study investigates how gesture-based interaction modes, namely, mid-air gesture and touchscreen gesture, compared with mouse-based interaction, affect consumers’ virtual product experiences (VPE) by eliciting mental imagery (i.e., haptic imagery and spatial imagery). Furthermore, we explore how visual product presentation can be designed to facilitate different types of interaction modes. Through a lab experiment, we find that touchscreen gesture outper- forms mid-air gesture and mouse-based interaction in terms of eliciting haptic imagery, and this effect is mitigated when 3D presentation is used. We also find that mid-air gesture outperforms touchscreen gesture and mouse-based interaction in terms of eliciting spatial imagery when 3D presentation is used. Both haptic imagery and spatial imagery can further reduce consumers’ product uncertainty. Our results extend prior work on interactivity design of VPE and further contribute to the emerging literature on gesture-based interaction.

Key words and phrases:

• virtual product experience
• gesture-based interaction
• grounded cognition
• mental imagery
• product presentation

Notes

¹ Leap Motion is a device that facilitates user interaction with the computer via mid-air gestures. It can create a virtual “touch” surface in the 3D space and recognize users’ gestures around the virtual surface. An illustration of Leap Motion usage is presented in Figure 2. More information about the device can be accessed via www.leapmotion.com.

Additional information

Funding

The authors thank the Ministry of Education of Singapore (Academic Research Fund Tier 1 and Tier 2) and the University of Hong Kong (Seed Fund) for their financial support.

Notes on contributors

Yang (Alison) Liu

Yang (ALISON) Liu ([email protected]) is a Research Fellow in the Department of Information Systems & Analytics at National University of Singapore. Her research interests include online consumer behavior, human computer interaction, algorithmic decision making, and digital innovation. Her work has appeared and awarded the best paper at the International Conference on Information Systems (ICIS).

Zhenhui (Jack) Jiang

Zhenhui (JACK) Jiang ([email protected]; corresponding author) is a professor of Innovation and Information Management at Faculty of Business and Economics, the University of Hong Kong. He is also a professor of Information Systems and Analytics at School of Computing, National University of Singapore. His research interests include social media, visual analytics, e-commerce, information privacy, and human-computer interaction. His work has been published in top Information Systems journals, such as Management Information Systems Quarterly (MISQ), Information Systems Research (ISR), Journal of Management Information Systems, and Journal of the Association for Information Systems (JAIS), and at top Computer Science conferences, such as the ACM Conference on Human Factors in Computing Systems (CHI). He currently serves as an senior editor of JAIS and associate editor of ISR, and has served on the editorial board of MISQ and IEEE Transactions on Engineering Management (IEEE TEM).

Hock Chuan Chan

Hock Chuan Chan ([email protected]) is an Associate Professor in the Department of Information Systems and Analytics at National University of Singapore. He works on information systems, human computer interaction, and healthcare IT. His work has been published in top Information Systems journals, such as MISQ and ISR. He has served as an associate editor for MISQ and was on the editorial board of Journal of Databased Management and Association for Information Systems (AIS) Transactions on Human-Computer Interaction.