Computational Modeling of Touchscreen Drag Gestures Using a Cognitive Architecture and Motion Tracking

ABSTRACT

This article presents a computational model that predicts finger-drag gesture performance on touchscreen devices, by integrating the queueing network (QN) cognitive architecture and motion tracking. Specifically, the QN-based model was developed to predict two execution times: the finger movement time of drag-gesture (i.e., only the motion time of the finger touched and dragged on the surface of touchscreen) and the comprehensive process time of drag-gesture (i.e., the entire process time to complete the finger-drag task, including visual attention shift, memory storage and retrieval, and hand-finger movements). To develop predictive models for the finger movement time of drag-gesture, 11 participants’ motion data were collected and a regression analysis with parameters of hand-finger anthropometric data and eight angular directions was conducted. Human subject data from our previous study (Jeong & Liu, 2017a) were used to evaluate the QN-based model, generating similar outputs (R² was more than 80% and root-mean square was less than 300 msec) for both execution times.

Acknowledgments

The authors are grateful to Justin Haney for providing initial insights to collect the data, Sang Won Lee for developing the interface, and Aravindh Sabarish Ramakrishnan for helping the data analysis. Our thanks also go to Dr. Fred Feng for his help, particularly in the development and usage of the simulation model and software. We would also like to thank the anonymous reviewers of the article for their valuable feedback.

Additional information

Funding

This study was supported by funding from a Rackham Graduate Student Research Grant at the University of Michigan [U051990].

Notes on contributors

Heejin Jeong

Heejin Jeong received his Ph.D. degree in industrial and operations engineering from the University of Michigan, Ann Arbor. His research interests include cognitive ergonomics, human factors, human-machine interaction, human systems engineering, and computational cognitive modeling.

Yili Liu

Yili Liu is Arthur F Thurnau Professor in the Department of Industrial and Operations Engineering at the University of Michigan, Ann Arbor. His areas of research and teaching include computational cognitive modeling, cognitive engineering, human factors, and aesthetic and cultural ergonomics.