Exploring Relationships Between Eye Tracking and Traditional Usability Testing Data

ABSTRACT

This study explored the relationships between eye tracking and traditional usability testing data in the context of analyzing the usability of Algebra Nation™, an online system for learning mathematics used by hundreds of thousands of students. Thirty-five undergraduate students (20 females) completed seven usability tasks in the Algebra Nation™ online learning environment. The participants were asked to log in, select an instructor for the instructional video, post a question on the collaborative wall, search for an explanation of a mathematics concept on the wall, find information relating to Karma Points (an incentive for engagement and learning), and watch two instructional videos of varied content difficulty. Participants’ eye movements (fixations and saccades) were simultaneously recorded by an eye tracker. Usability testing software was used to capture all participants’ interactions with the system, task completion time, and task difficulty ratings. Upon finishing the usability tasks, participants completed the System Usability Scale. Important relationships were identified between the eye movement metrics and traditional usability testing metrics such as task difficulty rating and completion time. Eye tracking data were investigated quantitatively using aggregated fixation maps, and qualitative examination was performed on video replay of participants’ fixation behavior. Augmenting the traditional usability testing methods, eye movement analysis provided additional insights regarding revisions to the interface elements associated with these usability tasks.

Acknowledgment

We also would like to thank the Study Edge Corporation for their help with participant recruitment.

Additional information

Funding

This article is based on work supported by the National Science Foundation under Grant No. 1540888.

Notes on contributors

Jiahui Wang

Jiahui Wang is a PhD candidate and research fellow in Educational Technology program at the University of Florida. Her research focuses on individual differences affect learning with technology and how technology-based learning environments can be designed to accommodate individual differences.

Pavlo Antonenko

Pavlo “Pasha” Antonenko is an Associate Professor of Educational Technology and Director of NeurAL Lab at the University of Florida. His scholarship focuses on a) frameworks and technologies to encourage and scaffold learning and twenty-firstcentury skills and b) psychophysiological assessment of cognition to optimize design of technology-enhanced learning environments.

Mehmet Celepkolu

Mehmet Celepkolu is currently a PhD student and in the Computer Science program at the University of Florida. His research focuses on how computational models can reveal the hidden phenomena during dialogue and learning, which can create effective strategies for supporting human learning with intelligent systems.

Yerika Jimenez

Yerika Jimenez is currently a PhD student in Human-Centered Computing and an NSF graduate research fellow at the University of Florida. Her research focuses on computer science education. She is interested in understanding how much cognitive effort do students use to interact and learn computer science with block-based programming environments.

Ethan Fieldman

Ethan Fieldman is the President of Study Edge. Study Edge provides various services including education technology that uses social media, mobile devices, online communities, gamification, personalized learning, and some of the best, most energetic instructors in the world to help students from middle school through college.

Ashley Fieldman

Ashley Fieldman is the Vice President of Study Edge. Study Edge provides various services including education technology that uses social media, mobile devices, online communities, gamification, personalized learning, and some of the best, most energetic instructors in the world to help students from middle school through college.