Optical Character Recognition (OCR)-Based and Gaussian Mixture Modeling-OCR-Based Slide-Level “With-Me-Ness”: Automated Measurement and Feedback of Learners’ Attention State during Video Lectures

Abstract

As video lectures are gaining more popularity, determining their effectiveness and obtaining valuable feedback have become necessary. To measure the learners’ attention state during video lectures, we specified the conceptual “with-me-ness” (WMN) as slide-level WMN (SL-WMN). The content domain on each slide was automatically extracted via an optical character recognition (OCR)-based method, while the eye gazing behaviors were analyzed through a Gaussian mixture modeling (GMM) fixation clustering method. Both domain-specific WMN and behavior-enriched WMN were then computed via OCR- and GMM-OCR-based methods to measure the learners’ attention levels. We conducted an experiment to collect in-lecture eye-tracking data, video recordings, and post-lecture test scores from 50 Grade 8 students. The results demonstrated that both OCR- and GMM-OCR-based SL-WMNs are reliable and compatible automatic measurements of learners’ attention states during video lectures. A survey from participating learners and lecturers also revealed highly favorable feedback for the developed SL-WMNs.

Keywords:

• Video lecture
• eye tracking
• with-me-ness
• AOI
• online learning environments

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Key R&D Program of China (2022ZD0117900), the Open Research Fund of Guangxi Key Lab of Human-machine Interaction and Intelligent Decision (GXHIID2201).

Notes on contributors

Chengchen Lyu

Chengchen Lyu received her B.S. from East China Normal University, P.R. China, and received her M.S. degree from the University of Auckland, New Zealand. She is now a Ph.D. candidate with the Institute of Software Chinese Academy of Sciences, P.R. China. Her research interests include human-computer interaction and multimodal learning.

Hui Chen

Hui Chen received her Ph.D. degree in computer science from the Chinese University of Hong Kong, P.R. China. She is now a professor with the Institute of Software Chinese Academy of Sciences, P.R. China. Her research interests include human–computer interaction, affective interaction, haptics, and virtual reality.

Xiaolan Peng

Xiaolan Peng received her B.S. and M.S. degrees in Human Robotic Interaction Lab from the University of Science and Technology Beijing and received her Ph.D. degree from the Institute of Software Chinese Academy of Sciences, P.R. China. Her research interests include human–computer interaction, affective interaction, and multimodal learning.

Juntao Ye

Juntao Ye a PhD degree in computer science from the University of Western Ontario. He had been with the Institute of Automation, Chinese Academy of Sciences. He is currently a Senior Principal Researcher with Huawei Human-Machine Interaction Lab at Toronto. His research interests include computer graphics and affective interaction.

Hongan Wang

Hongan Wang received his Ph.D. degree from the Institute of Software Chinese Academy of Sciences, P.R. China. He has been a full professor and the director of Beijing Key Lab of Human-Computer Interaction, Institute of Software Chinese Academy of Sciences, P.R. China. His research interests include human-computer interaction and real-time intelligence.