Real-Time Workload Estimation Using Eye Tracking: A Bayesian Inference Approach

References

• Ahern, S., & Beatty, J. (1979). Pupillary responses during information processing vary with scholastic aptitude test scores. Science, 205(4412), 1289–1292. https://doi.org/10.1126/science.472746
   PubMed Web of Science ®Google Scholar
• Ahlstrom, U., & Friedman-Berg, F. J. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36(7), 623–636. https://doi.org/10.1016/j.ergon.2006.04.002
   Web of Science ®Google Scholar
• Backs, R. W., & Walrath, L. C. (1992). Eye movement and pupillary response indices of mental workload during visual search of symbolic displays. Applied Ergonomics, 23(4), 243–254. https://doi.org/10.1016/0003-6870(92)90152-l
   PubMed Web of Science ®Google Scholar
• Benedetto, S., Pedrotti, M., Minin, L., Baccino, T., Re, A., & Montanari, R. (2011). Driver workload and eye blink duration. Transportation Research Part F, 14(3), 199–208. https://doi.org/10.1016/j.trf.2010.12.001
   Google Scholar
• Bodala, I. P., Ke, Y., Mir, H., Thakor, N. V., & Al-Nashash, H. (2014). Cognitive workload estimation due to vague visual stimuli using saccadic eye movements. In 2014 36th annual international conference of the IEEE engineering in medicine and biology society (pp. 2993–2996). https://doi.org/10.1109/EMBC.2014.6944252
   Google Scholar
• Calinon, S. (2016). A tutorial on task-parameterized movement learning and retrieval. Intelligent Service Robotics, 9(1), 1–29. https://doi.org/10.1007/s11370-015-0187-9
   Web of Science ®Google Scholar
• Calinon, S., & Billard, A. (2005). Recognition and reproduction of gestures using a probabilistic framework combining PCA, ICA and HMM. In Proceedings of The 22nd international conference on machine learning (pp. 105–112). https://doi.org/10.1145/1102351.1102365
   Google Scholar
• Carmody, M. A. (1994). Current issues in the measurement of military aircrew performance: A consideration of the relationship between available metrics and operational concerns. (Tech. Rep.). Naval Air Warfare Center.
   Google Scholar
• Chang, C.-C., Boyle, L. N., Lee, J. D., & Jenness, J. (2017). Using tactile detection response tasks to assess in-vehicle voice control interactions. Transportation Research Part F, 51, 38–46. https://doi.org/10.1016/j.trf.2017.06.008
   Google Scholar
• Chen, S., & Epps, J. (2013). Automatic classification of eye activity for cognitive load measurement with emotion interference. Computer Methods and Programs in Biomedicine, 110(2), 111–124. https://doi.org/10.1016/j.cmpb.2012.10.021
   PubMed Web of Science ®Google Scholar
• De Waard, D. (1996). The measurement of drivers. Mental workload [Unpublished doctoral dissertation]. University of Groningen.
   Google Scholar
• Demberg, V. (2013). Pupillometry: The index of cognitive activity in a dual-task study. In Proceedings of the annual meeting of the cognitive science society (Vol. 35).
   Google Scholar
• Di Nocera, F., Camilli, M., & Terenzi, M. (2007). A random glance at the flight deck: Pilots’ scanning strategies and the real-time assessment of mental workload. Journal of Cognitive Engineering and Decision Making, 1(3), 271–285. https://doi.org/10.1518/155534307X255627
   Google Scholar
• Du, N., Huang, K. Y., & Yang, X. J. (2020). Not all information is equal: Effects of disclosing different types of likelihood information on trust, compliance and reliance, and task performance in human-automation teaming. Human Factors, 62(6), 987–1001. https://doi.org/10.1177/0018720819862916
   PubMed Web of Science ®Google Scholar
• Eggemeier, F., Biers, D., Wickens, C., Andre, A., Vreuls, D., Billman, E., & Schueren, J. (1990). Performance assessment and workload evaluation systems: Analysis of candidate measures (Tech. Rep.). Human Systems Division, Air Force Systems Command.
   Google Scholar
• Esterwood, C., Yang, X. J., & Robert, L. P. (2021). Barriers to AV bus acceptance: A national survey and research Agenda. International Journal of Human–Computer Interaction, 37(15), 1391–1403. https://doi.org/10.1080/10447318.2021.1886485
   Web of Science ®Google Scholar
• Fridman, L., Reimer, B., Mehler, B., & Freeman, W. T. (2018). Cognitive load estimation in the wild. In Proceedings of The 2018 Chi conference on human factors in computing systems (p. 652). https://doi.org/10.1145/3173574.3174226
   Google Scholar
• Gao, X., Yan, H., & Sun, H.-J. (2015). Modulation of microsaccade rate by task difficulty revealed through between-and within-trial comparisons. Journal of Vision, 15(3), 3–3. https://doi.org/10.1167/15.3.3
   PubMed Web of Science ®Google Scholar
• Goldberg, J. H., & Kotval, X. P. (1999). Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics, 24(6), 631–645. https://doi.org/10.1016/S0169-8141(98)00068-7
   Web of Science ®Google Scholar
• Guo, Y., & Yang, X. J. (2021). Modeling and predicting trust dynamics in human–robot teaming: A Bayesian inference approach. International Journal of Social Robotics, 13(8), 1899–1909. https://doi.org/10.1007/s12369-020-00703-3
   Web of Science ®Google Scholar
• Halverson, T., Estepp, J., Christensen, J., & Monnin, J. (2012). Classifying workload with eye movements in a complex task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 168–172. https://doi.org/10.1177/1071181312561012
   Google Scholar
• Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). Elsevier.
   Google Scholar
• He, X., Wang, L., Gao, X., Chen, Y. (2012). The eye activity measurement of mental workload based on basic flight task. In IEEE 10th international conference on industrial informatics (pp. 502–507).
   Google Scholar
• Heard, J., Harriott, C. E., & Adams, J. A. (2018). A survey of workload assessment algorithms. IEEE Transactions on Human-Machine Systems, 48(5), 434–451. https://doi.org/10.1109/THMS.2017.2782483
   Web of Science ®Google Scholar
• Hicks, T. G., & Wierwille, W. W. (1979). Comparison of five mental workload assessment procedures in a moving-base driving simulator. Human Factors, 21(2), 129–143. https://doi.org/10.1177/001872087902100201
   Web of Science ®Google Scholar
• Hogervorst, M. A., Brouwer, A.-M., & Van Erp, J. B. (2014). Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Frontiers in Neuroscience, 8, 322. https://doi.org/10.3389/fnins.2014.00322
   PubMed Web of Science ®Google Scholar
• Jacob, R. J. K. (1995). Eye tracking in advanced interface design. In Virtual Environments and Advanced Interface Design (pp. 258–288). Oxford University Press.
   Google Scholar
• Jacob, R. J., & Karn, K. S. (2003). Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The mind’s eye (pp. 573–605). Elsevier.
   Google Scholar
• Klingner, J., Kumar, R., & Hanrahan, P. (2008). Measuring the task-evoked pupillary response with a remote eye tracker. In Proceedings of the 2008 symposium on eye tracking research & applications (pp. 69–72). https://doi.org/10.1145/1344471.1344489
   Google Scholar
• Kosch, T., Hassib, M., Buschek, D., & Schmidt, A. (2018). Look into my eyes: Using pupil dilation to estimate mental workload for task complexity adaptation. In Extended abstracts of the 2018 chi conference on human factors in computing systems (p. 1–6). Association for Computing Machinery.
   Google Scholar
• Kosch, T., Hassib, M., Woźniak, P. W., Buschek, D., Alt, F. (2018). Your eyes tell: Leveraging smooth pursuit for assessing cognitive workload. In Proceedings of the 2018 Chi conference on human factors in computing systems (pp. 1–13).
   Google Scholar
• Kun, A. L., Palinko, O., Medenica, Z., & Heeman, P. A. (2013). On the feasibility of using pupil diameter to estimate cognitive load changes for in-vehicle spoken dialogues. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH (pp. 3766–3770). https://doi.org/10.21437/Interspeech.2013-595
   Google Scholar
• Lansdown, T. C., Brook-Carter, N., & Kersloot, T. (2004). Distraction from multiple in-vehicle secondary tasks: Vehicle performance and mental workload implications. Ergonomics, 47(1), 91–104. https://doi.org/10.1080/00140130310001629775
   PubMed Web of Science ®Google Scholar
• Leonard, J. J., Mindell, D. A., & Stayton, E. L. (2020). Autonomous vehicles, mobility, and employment policy: The roads ahead. MIT Task Force on Work of the Future Research Brief. Retrieved from https://workofthefuture.mit.edu/research-post/autonomous-vehicles-mobility-and-employment-policy-the-roads-ahead/
   Google Scholar
• Li, W.-C., Chiu, F.-C., Wu, K.-J. (2012). The evaluation of pilots performance and mental workload by eye movement. In Proceeding of the 30th European association for aviation psychology conference.
   Google Scholar
• Liang, Y., Reyes, M. L., & Lee, J. D. (2007). Real-time detection of driver cognitive distraction using support vector machines. IEEE Transactions on Intelligent Transportation Systems, 8(2), 340–350. https://doi.org/10.1109/TITS.2007.895298
   Web of Science ®Google Scholar
• Liu, K. (2019). Measuring and quantifying driver workload on limited access roads [Unpublished doctoral dissertation]. University of Michigan.
   Google Scholar
• Lu, S., Zhang, M. Y., Ersal, T., & Yang, X. J. (2019). Workload management in teleoperation of unmanned ground vehicles: Effects of a delay compensation aid on human operators’ workload and teleoperation performance. International Journal of Human–Computer Interaction, 35(19), 1820–1830. https://doi.org/10.1080/10447318.2019.1574059
   Google Scholar
• Luo, R., Hayne, R., & Berenson, D. (2018). Unsupervised early prediction of human reaching for human–robot collaboration in shared workspaces. Autonomous Robots, 42(3), 631–648. https://doi.org/10.1007/s10514-017-9655-8
   Web of Science ®Google Scholar
• Luo, R., Wang, Y., Weng, Y., Paul, V., Brudnak, M. J., Jayakumar, P., Reed, M., Stein, J. L., Ersal, T., & Yang, X. J. (2019). Toward real-time assessment of workload: A Bayesian inference approach. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 196–200. https://doi.org/10.1177/1071181319631293
   Google Scholar
• Luo, R., Weng, Y., Wang, Y., Jayakumar, P., Brudnak, M. J., Paul, V., Desaraju, V. R., Stein, J. L., Ersal, T., & Yang, X. J. (2021). A workload adaptive haptic shared control scheme for semi-autonomous driving. Accident; Analysis and Prevention, 152, 105968. https://doi.org/10.1016/j.aap.2020.105968
   PubMed Web of Science ®Google Scholar
• Mallick, R., Slayback, D., Touryan, J., Ries, A. J., & Lance, B. J. (2016). The use of eye metrics to index cognitive workload in video games. In 2016 IEEE second workshop on eye tracking and visualization (etvis) (pp. 60–64). https://doi.org/10.1109/ETVIS.2016.7851168
   Google Scholar
• Marquart, G., Cabrall, C., & de Winter, J. (2015). Review of eye-related measures of drivers’ mental workload. Procedia Manufacturing, 3, 2854–2861. https://doi.org/10.1016/j.promfg.2015.07.783
   Google Scholar
• Marshall, S. P. (2000, July). Method and apparatus for eye tracking and monitoring pupil dilation to evaluate cognitive activity. EyeTracking, Inc.
   Google Scholar
• Marshall, S. P. (2002). The index of cognitive activity: Measuring cognitive workload. In Proceedings of the IEEE 7th conference on human factors and power plants (pp. 7–7).
   Google Scholar
• May, J. G., Kennedy, R. S., Williams, M. C., Dunlap, W. P., & Brannan, J. R. (1990). Eye movement indices of mental workload. Acta Psychologica, 75(1), 75–89. https://doi.org/10.1016/0001-6918(90)90067-p
   PubMed Web of Science ®Google Scholar
• Menekse Dalveren, G. G., & Cagiltay, N. E. (2018). Insights from surgeons’ eye-movement data in a virtual simulation surgical training environment: Effect of experience level and hand conditions. Behaviour & Information Technology, 37(5), 517–537. https://doi.org/10.1080/0144929X.2018.1460399
   Web of Science ®Google Scholar
• Miller, E. E., Boyle, L. N., Jenness, J. W., & Lee, J. D. (2018). Voice control tasks on cognitive workload and driving performance: Implications of modality, difficulty, and duration. Transportation Research Record, 2672(37), 84–93. https://doi.org/10.1177/0361198118797483
   Web of Science ®Google Scholar
• Moacdieh, N. M., Devlin, S. P., Jundi, H., & Riggs, S. L. (2020). Effects of workload and workload transitions on attention allocation in a dual-task environment: Evidence from eye tracking metrics. Journal of Cognitive Engineering and Decision Making, 14(2), 132–151. https://doi.org/10.1177/1555343419892184
   Web of Science ®Google Scholar
• Owen, A. M., McMillan, K. M., Laird, A. R., & Bullmore, E. (2005). N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies. Human Brain Mapping, 25(1), 46–59. https://doi.org/10.1002/hbm.20131
   PubMed Web of Science ®Google Scholar
• Palinko, O., & Kun, A. (2011). Exploring the influence of light and cognitive load on pupil diameter in driving simulator studies.
   Google Scholar
• Palinko, O., Kun, A. L., Shyrokov, A., & Heeman, P. (2010). Estimating cognitive load using remote eye tracking in a driving simulator. In Proceedings of the 2010 symposium on eye-tracking research & applications (pp. 141–144). https://doi.org/10.1145/1743666.1743701
   Google Scholar
• Rayner, K. (1995). Eye movements and cognitive processes in reading, visual search, and scene perception. In Studies in visual information processing (Vol. 6, pp. 3–22). Elsevier.
   Google Scholar
• Rayner, K. (2009). The 35th sir frederick bartlett lecture: Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8), 1457–1506. https://doi.org/10.1080/17470210902816461
   PubMed Web of Science ®Google Scholar
• Rayner, K., & Morris, R. K. (1990). Do eye movements reflect higher order processes in reading? In From eye to mind: Information acquisition in perception, search, and reading (pp. 179–190). North-Holland.
   Google Scholar
• Recarte, M. A., & Nunes, L. M. (2000). Effects of verbal and spatial-imagery tasks on eye fixations while driving. Journal of Experimental Psychology. Applied, 6(1), 31–43. https://doi.org/10.1037//1076-898x.6.1.31
   PubMed Web of Science ®Google Scholar
• Recarte, M. A., & Nunes, L. M. (2003). Mental workload while driving: Effects on visual search, discrimination, and decision making. Journal of Experimental Psychology. Applied, 9(2), 119–137. https://doi.org/10.1037/1076-898x.9.2.119
   PubMed Web of Science ®Google Scholar
• Recarte, M. Á., Pérez, E., Conchillo, Á., & Nunes, L. M. (2008). Mental workload and visual impairment: Differences between pupil, blink, and subjective rating. The Spanish Journal of Psychology, 11(2), 374–385. https://doi.org/10.1017/S1138741600004406
   PubMed Web of Science ®Google Scholar
• Reimer, B. (2009). Impact of cognitive task complexity on drivers’ visual tunneling. Transportation Research Record, 2138(1), 13–19. https://doi.org/10.3141/2138-03
   Google Scholar
• Rerhaye, L., Blaser, T., & Alexander, T. (2018). Evaluation of the index of cognitive activity (ICA) as an instrument to measure cognitive workload under differing light conditions. In Congress of the international ergonomics association (pp. 350–359).
   Google Scholar
• Robinson, G. H. (1979). Dynamics of the eye and head during movement between displays: A qualitative and quantitative guide for designers. Human Factors, 21(3), 343–352. https://doi.org/10.1177/001872087902100309
   Web of Science ®Google Scholar
• Rozo, L., Silverio, J., Calinon, S., & Caldwell, D. G. (2016). Learning controllers for reactive and proactive behaviors in human–robot collaboration. Frontiers in Robotics and AI, 3, 30. https://doi.org/10.3389/frobt.2016.00030
   Google Scholar
• Salvucci, D. D., & Goldberg, J. H. (2000). Identifying fixations and saccades in eye-tracking protocols. In Proceedings of The 2000 symposium on eye tracking research & applications (pp. 71–78). https://doi.org/10.1145/355017.355028
   Google Scholar
• Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 136. https://doi.org/10.1214/aos/1176344136
   Web of Science ®Google Scholar
• Skaramagkas, V., Giannakakis, G., Ktistakis, E., Manousos, D., Karatzanis, I., Tachos, N. S., Tripoliti, E., Marias, K., Fotiadis, D. I., & Tsiknakis, M. (2023). Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Reviews in Biomedical Engineering, 16, 260–277. https://doi.org/10.1109/RBME.2021.3066072
   Google Scholar
• Skaramagkas, V., Ktistakis, E., Manousos, D., Tachos, N. S., Kazantzaki, E., Tripoliti, E. E., Fotiadis, D. I. Tsiknakis, M. (2021). Cognitive workload level estimation based on eye tracking: A machine learning approach. In 2021 IEEE 21st International Conference on Bioinformatics and Bioengineering (BIBE) (pp. 1–5). https://doi.org/10.1109/BIBE52308.2021.9635166
   Google Scholar
• Sun, Z., Li, B., Duan, F., Jia, H., Wang, S., Liu, Y., Cichocki, A., Caiafa, C. F., & Sole-Casals, J. (2021). Wlnet: Towards an approach for robust workload estimation based on shallow neural networks. IEEE Access. 9, 3165–3173. https://doi.org/10.1109/ACCESS.2020.3044732
   Web of Science ®Google Scholar
• Tobii Pro, A. B. (2014). Tobii pro lab. Computer software. Danderyd. Retrieved from http://www.tobiipro.com/
   Google Scholar
• Tsai, Y.-F., Viirre, E., Strychacz, C., Chase, B., & Jung, T.-P. (2007). Task performance and eye activity: Predicting behavior relating to cognitive workload. Aviation, Space, and Environmental Medicine, 78(5 Suppl), B176–B185.
   PubMedGoogle Scholar
• van der Wel, P., & van Steenbergen, H. (2018). Pupil dilation as an index of effort in cognitive control tasks: A review. Psychonomic Bulletin & Review, 25(6), 2005–2015. https://doi.org/10.3758/s13423-018-1432-y
   PubMed Web of Science ®Google Scholar
• Van Orden, K. F., Limbert, W., Makeig, S., & Jung, T.-P. (2001). Eye activity correlates of workload during a visuospatial memory task. Human Factors, 43(1), 111–121. https://doi.org/10.1518/001872001775992570
   PubMed Web of Science ®Google Scholar
• Vogels, J., Demberg, V., & Kray, J. (2018). The index of cognitive activity as a measure of cognitive processing load in dual task settings. Frontiers in Psychology, 9, 2276. https://doi.org/10.3389/fpsyg.2018.02276
   PubMed Web of Science ®Google Scholar
• Wang, J., Olson, E. (2016, October). AprilTag 2: Efficient and robust fiducial detection. In Proceedings of the IEEE/RSJ international conference on Intelligent Robots and Systems (IROS).
   Google Scholar
• Weng, Y., Luo, R., Jayakumar, P., Brudnak, M. J., Paul, V., Desaraju, V. R., Stein, J. L., Yang, X. J., & Ersal, T. (2020). Design and evaluation of a workload-adaptive haptic shared control framework for semi-autonomous driving. In 2020 American Control Conference (ACC) (pp. 4369–4374). https://doi.org/10.23919/ACC45564.2020.9147482
   Google Scholar
• Wu, T.-F., Lin, C.-J., & Weng, R. C. (2004). Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Research, 5(Aug), 975–1005.
   Google Scholar
• Yang, X. J., Unhelkar, V. V., Li, K., & Shah, J. A. (2017). Evaluating effects of user experience and system transparency on trust in automation. In 2017 12th ACM/IEEE international conference on human-robot interaction (Hri) (pp. 408–416). https://doi.org/10.1145/2909824.3020230
   Google Scholar
• Yang, Y., McDonald, M., & Zheng, P. (2012). Can drivers’ eye movements be used to monitor their performance? a case study. IET Intelligent Transport Systems, 6(4), 444–452. https://doi.org/10.1049/iet-its.2012.0008
   Web of Science ®Google Scholar
• Yokoyama, H., Eihata, K., Muramatsu, J., & Fujiwara, Y. (2018). Prediction of driver’s workload from slow fluctuations of pupil diameter. In 2018 21st international conference on intelligent transportation systems (ITSC) (pp. 1775–1780). https://doi.org/10.1109/ITSC.2018.8569279
   Google Scholar
• Zhang, Q., Yang, X. J., & Robert, L. P. (2022). Individual differences and expectations of automated vehicles. International Journal of Human–Computer Interaction, 38(9), 825–836. https://doi.org/10.1080/10447318.2021.1970431
   Web of Science ®Google Scholar
• Zhang, Y., Owechko, Y., & Zhang, J. (2008). Learning-based driver workload estimation. In Computational intelligence in automotive applications (pp. 1–17). Springer.
   Google Scholar
• Zhou, F., Yang, X. J., & de Winter, J. C. F. (2022). Using eye-tracking data to predict situation awareness in real time during takeover transitions in conditionally automated driving. IEEE Transactions on Intelligent Transportation Systems, 23(3), 2284–2295. https://doi.org/10.1109/TITS.2021.3069776
   Web of Science ®Google Scholar