Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning

References

• Baldwin, C. L., and B. N. Penaranda. 2012. “Adaptive Training Using an Artificial Neural Network and EEG Metrics for within- and Cross-Task Workload Classification[J].” Neuroimage 59 (1): 48–56. doi:10.1016/j.neuroimage.2011.07.047.
   PubMed Web of Science ®Google Scholar
• Brookings, J. B., G. F. Wilson, and C. R. Swain. 1996. “Psychophysiological Responses to Changes in Workload during Simulated Air Traffic Control[J].” Biological Psychology 42 (3): 361–377. doi:10.1016/0301-0511(95)05167-8.
   PubMed Web of Science ®Google Scholar
• Cao, Y., Q. Qu, V. G. Duffy, and Y. Ding. 2019. “Attention for Web Directory Advertisements: A Top-Down or Bottom-Up Process.” International Journal of Human–Computer Interaction 35 (1): 89–98. doi:10.1080/10447318.2018.1432162.
   Web of Science ®Google Scholar
• Casali, J. G., and W. W. Wierwille. 1984. “On the Measurement of Pilot Perceptual Workload: A Comparison of Assessment Techniques Addressing Sensitivity and Intrusion Issues.” Ergonomics 27 (10): 1033–1050. doi:10.1080/00140138408963584.
   PubMed Web of Science ®Google Scholar
• Chanel, G., C. Rebetez, M. Bétrancourt, and T. Pun. 2011. “Emotion Assessment from Physiological Signals for Adaptation of Game Difficulty.” IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 41 (6): 1052–1063. doi:10.1109/TSMCA.2011.2116000.
   Web of Science ®Google Scholar
• Charles, R. L., and J. Nixon. 2019. “Measuring Mental Workload Using Physiological Measures: A Systematic Review[J].” Applied Ergonomics 74: 221–232. doi:10.1016/j.apergo.2018.08.028.
   PubMed Web of Science ®Google Scholar
• Cinaz, B., B. Arnrich, R. La Marca, and G. Tröster. 2013. “Monitoring of Mental Workload Levels during an Everyday Life Office-Work Scenario[J].” Personal and Ubiquitous Computing 17 (2): 229–239. doi:10.1007/s00779-011-0466-1.
   Web of Science ®Google Scholar
• Collet, C., E. Salvia, and C. Petit-Boulanger. 2014. “Measuring Workload with Electrodermal Activity during Common Braking Actions[J].” Ergonomics 57 (6): 886–896. doi:10.1080/00140139.2014.899627.
   PubMed Web of Science ®Google Scholar
• De Rivecourt, M., M. N. Kuperus, W. J. Post, and L. J. M. Mulder. 2008. “Cardiovascular and Eye Activity Measures as Indices for Momentary Changes in Mental Effort during Simulated Flight[J].” Ergonomics 51 (9): 1295–1319. doi:10.1080/00140130802120267.
   PubMed Web of Science ®Google Scholar
• De Waard, D., and B. Lewis-Evans. 2014. “Self-Report Scales Alone Cannot Capture Mental Workload[J.” Cognition, Technology and Work 16 (3): 303–305. doi:10.1007/s10111-014-0277-z.
   Web of Science ®Google Scholar
• Ding, Y., Y. Cao, and Y. Wang. 2019. Physiological Indicators of Mental Workload in Visual Display Terminal Work[C]//International Conference on Applied Human Factors and Ergonomics, , 86–94. Cham: Springer.
   Google Scholar
• Engström, J., E. Johansson, and J. Östlund. 2005. “Effects of Visual and Cognitive Load in Real and Simulated Motorway Driving.[J.” Transportation Research Part F: Traffic Psychology and Behaviour 8 (2): 97–120. doi:10.1016/j.trf.2005.04.012.
   Web of Science ®Google Scholar
• Fairclough, S. H., L. Venables, and A. Tattersall. 2005. “The Influence of Task Demand and Learning on the Psychophysiological Response[J].” International Journal of Psychophysiology 56 (2): 171–184. doi:10.1016/j.ijpsycho.2004.11.003.
   PubMed Web of Science ®Google Scholar
• Fairclough, S. H., and L. Venables. 2006. “Prediction of Subjective States from Psychophysiology: A Multivariate Approach[J].” Biological Psychology 71 (1): 100–110. doi:10.1016/j.biopsycho.2005.03.007.
   PubMed Web of Science ®Google Scholar
• Fawcett, T. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8): 861–874. doi:10.1016/j.patrec.2005.10.010.
   Web of Science ®Google Scholar
• Finsen, L., K. Søgaard, C. Jensen, V. Borg, and H. Christensen. 2001. “Muscle Activity and Cardiovascular Response during Computer-Mouse Work with and without Memory Demands[J].” Ergonomics 44 (14): 1312–1329. doi:10.1080/00140130110099065.
   PubMed Web of Science ®Google Scholar
• Fong, A., C. Sibley, A. Cole, C. Baldwin, and J. Coyne. 2010. “A Comparison of Artificial Neural Networks, Logistic Regressions, and Classification Trees for Modeling Mental Workload in Real-Time[J].” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 54 (19): 1709–1712. doi:10.1177/154193121005401973.
   Google Scholar
• Fournier, L. R., G. F. Wilson, and C. R. Swain. 1999. “Electrophysiological, Behavioral, and Subjective Indexes of Workload When Performing Multiple Tasks: manipulations of Task Difficulty and Training[J.” International Journal of Psychophysiology 31 (2): 129–145. doi:10.1016/S0167-8760(98)00049-X.
   PubMed Web of Science ®Google Scholar
• Gopher, D., and E. Donchin. 1986. “Workload: An Examination of the Concept.” In Handbook of Perception and Human Performance, Vol. 2: Cognitive Processes and Performance, edited by K. R. Boff, L. Kaufman, and P. Thomas, 1–49. Oxford, U.K.: Wiley.
   Google Scholar
• Grassmann, M., E. Vlemincx, A. von Leupoldt, and O. Van den Bergh. 2016. “The Role of Respiratory Measures to Assess Mental Load in Pilot Selection[J].” Ergonomics 59 (6): 745–749. doi:10.1080/00140139.2015.1090019.
   PubMed Web of Science ®Google Scholar
• Grassmann, M., E. Vlemincx, A. von Leupoldt, J. M. Mittelstädt, and O. Van den Bergh. 2016. “Respiratory Changes in Response to Cognitive Load: A Systematic Review[J].” Neural Plasticity 2016: 1–16. doi:10.1155/2016/8146809.
   Web of Science ®Google Scholar
• Guyon, I., and A. Elisseeff. 2006. “An Introduction to Feature Extraction.” In Guyon, I., Gunn, S., Nikravesh, M., Zadeh, L.A. (Eds.). Feature Extraction: Foundations and applications studies in fuzziness and soft computing, 1–25. Berlin/Heidelberg, Germany: Springer.
   Google Scholar
• Halder, A., C. Gao, M. Miller, and K. Kuklane. 2018. “Oxygen Uptake and Muscle Activity Limitations during Stepping on a Stair Machine at Three Different Climbing Speeds[J].” Ergonomics 61 (10): 1382–1394. doi:10.1080/00140139.2018.1473644.
   PubMed Web of Science ®Google Scholar
• Hancock, P A., and J L. Szalma. 2006. “Stress and Neuroergonomics [M].” In The Brain at Work, edited by R. Parasuraman, and M. Rizzo, 195–206. Oxford, UK: Oxford University Press.
   Google Scholar
• Hancock, P A. 1989. “A Dynamic Model of Stress and Sustained Attention.” Human Factors: The Journal of the Human Factors and Ergonomics Society 31 (5): 519–537. doi:10.1177/001872088903100503.
   PubMed Web of Science ®Google Scholar
• Hancock, P. A., and G. Matthews. 2019. “Workload and Performance: Associations, Insensitivities, and Dissociations[J].” Human Factors: The Journal of the Human Factors and Ergonomics Society 61 (3): 374–392. doi:10.1177/0018720818809590.
   PubMed Web of Science ®Google Scholar
• Hart, S. G., and L. E. Staveland. 1998. “Development of NASA-TLX: Results of Empirical and Theoretical Research [M].” In Human Mental Workload, edited by P. A. Hancock and N. Meshkati, 139–183. Amsterdam: Elsevier Science.
   Google Scholar
• Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning: data Mining, Inference, and Prediction. Berlin: Springer Science & Business Media.
   Google Scholar
• Healey, J. A., and R. W. Picard. 2005. “Detecting Stress during Real-World Driving Tasks Using Physiological Sensors.” IEEE Transactions on Intelligent Transportation Systems 6 (2): 156–166. doi:10.1109/TITS.2005.848368.
   Web of Science ®Google Scholar
• Hockey, G. R. J., P. Nickel, A. C. Roberts, and M. H. Roberts. 2009. “Sensitivity of Candidate Markers of Psychophysiological Strain to Cyclical Changes in Manual Control Load during Simulated Process Control[J].” Applied Ergonomics 40 (6): 1011–1018. doi:10.1016/j.apergo.2009.04.008.
   PubMed Web of Science ®Google Scholar
• Hogervorst, M. A., A. M. Brouwer, and J. B. Van Erp. 2014. “Combining and Comparing EEG, Peripheral Physiology and Eye-Related Measures for the Assessment of Mental Workload[J].” Frontiers in Neuroscience 8: 322. doi:10.3389/fnins.2014.00322.
   PubMed Web of Science ®Google Scholar
• Jafari, M.-J., F. Zaeri, A. H. Jafari, A. T. Payandeh Najafabadi, S. Al-Qaisi, and N. Hassanzadeh-Rangi. 2020. “Assessment and Monitoring of Mental Workload in Subway Train Operations Using Physiological, Subjective, and Performance Measures[J].” Human Factors and Ergonomics in Manufacturing and Service Industries 30 (3): 165–175. doi:10.1002/hfm.20831.
   Web of Science ®Google Scholar
• Jang, E.-H., B.-J. Park, M.-S. Park, S.-H. Kim, and J.-H. Sohn. 2015. “Analysis of Physiological Signals for Recognition of Boredom, Pain, and Surprise Emotions[J].” Journal of Physiological Anthropology 34 (1): 25. doi:10.1186/s40101-015-0063-5.
   PubMed Web of Science ®Google Scholar
• Jimenez-Molina, A., C. Retamal, and H. Lira. 2018. “Using Psychophysiological Sensors to Assess Mental Workload during Web Browsing[J].” Sensors 18 (2): 458. doi:10.3390/s18020458.
   PubMed Web of Science ®Google Scholar
• Jorna, P. G. 1992. “Spectral Analysis of Heart Rate and Psychological State: A Review of Its Validity as a Workload Index.” Biological Psychology 34 (2–3): 237–257. doi:10.1016/0301-0511(92)90017-O.
   PubMed Web of Science ®Google Scholar
• Kolodyazhniy, V., S. D. Kreibig, J. J. Gross, W. T. Roth, and F. H. Wilhelm. 2011. “An Affective Computing Approach to Physiological Emotion Specificity: Toward Subject Independent and Stimulus Independent Classification of Film Induced Emotions.” Psychophysiology 48 (7): 908–922. doi:10.1111/j.1469-8986.2010.01170.x.
   PubMed Web of Science ®Google Scholar
• Kramer, A. F. 1991. “Physiological Metrics of Mental Workload: A Review of Recent Progress [M].” In Multiple-Task Performance, edited by D. L. Damos, 279–327. London: Taylor and Francis.
   Google Scholar
• Mallat, C., J. Cegarra, C. Calmettes, and R. L. Capa. 2019. “A Curvilinear Effect of Mental Workload on Mental Effort and Behavioral Adaptability: An Approach with the Pre-Ejection Period[J].” Human Factors. doi:10.1177/0018720819855919.
   PubMed Web of Science ®Google Scholar
• Matthews, G., L. Joyner, and K. Gilliland., et al. 1999. “Validation of a Comprehensive Stress State Questionnaire: Towards a State Big Three.” Personality Psychology in Europe [J] 7: 335–350.
   Google Scholar
• Matthews, G., L. E. Reinerman-Jones, D. J. Barber, and J. Abich. 2015. “The Psychometrics of Mental Workload: Multiple Measures Are Sensitive but Divergent[J].” Human Factors: The Journal of the Human Factors and Ergonomics Society 57 (1): 125–143. doi:10.1177/0018720814539505.
   PubMed Web of Science ®Google Scholar
• Matthews, G., L. Reinerman-Jones, R. Wohleber., et al. 2015. “Workload is Multidimensional, Not Unitary: what Now?[C].” In Foundations of Augmented Cognition: 9th International Conference, AC 2015, Held as Part of HCI International 2015, edited by D. D. Schmorrow, and C. M. Fidopiastis, 44–55. Los Angeles, CA, USA; Cham: Springer International Publishing.
   Google Scholar
• Miyake, S., S. Yamada, T. Shoji, Y. Takae, N. Kuge, and T. Yamamura. 2009. “Physiological Responses to Workload Change. A Test/Retest Examination[J].” Applied Ergonomics 40 (6): 987–996. doi:10.1016/j.apergo.2009.02.005.
   PubMed Web of Science ®Google Scholar
• Mulder, L. J. M., D. de Waard, and K. A. Brookhuis. 2004. “Estimating Mental Effort Using Heart Rate and Heart Rate Variability [M].” In Handbook of Human Factors and Ergonomics Methods, edited by N. A. Stanton, A. Hedge, and K. Brookhuis, 201–210. London: Taylor and Francis.
   Google Scholar
• Nixon, J., and R. Charles. 2017. “Understanding the Human Performance Envelope Using Electrophysiological Measures from Wearable Technology[J].” Cognition Technology and Work 19 (5): 1–12.
   Google Scholar
• Orlandi, L., and B. Brooks. 2018. “Measuring Mental Workload and Physiological Reactions in Marine Pilots: Building Bridges towards Redlines of Performance[J].” Applied Ergonomics 69: 74–92. doi:10.1016/j.apergo.2018.01.005.
   PubMed Web of Science ®Google Scholar
• Parasuraman, R., and M. Rizzo (Eds.). 2007. Neuroergonomics: The Brain at Work. New York: Oxford University Press.
   Google Scholar
• Roscoe, A. H. 1992. “Assessing Pilot Workload: Why Measure Heart Rate, HRV and Respiration?[J].” Biological Psychology 34 (2–3): 259–287. doi:10.1016/0301-0511(92)90018-P.
   PubMed Web of Science ®Google Scholar
• Ryu, K., and R. Myung. 2005. “Evaluation of Mental Workload with a Combined Measure Based on Physiological Indices during a Dual Task of Tracking and Mental Arithmetic[J].” International Journal of Industrial Ergonomics 35 (11): 991–1009. doi:10.1016/j.ergon.2005.04.005.
   Web of Science ®Google Scholar
• Shuggi, I. M., H. Oh, P. A. Shewokis, and R. J. Gentili. 2017. “Mental Workload and Motor Performance Dynamics during Practice of Reaching Movements under Various Levels of Task Difficulty.” Neuroscience 360: 166–179. doi:10.1016/j.neuroscience.2017.07.048.
   PubMed Web of Science ®Google Scholar
• Stephenson, M. L., A. G. Ostrander, H. Norasi, and M. C. Dorneich. 2019. “Shoulder Muscular Fatigue from Static Posture Concurrently Reduces Cognitive Attentional Resources[J].” Human Factors first published online, doi:10.1177/0018720819852509.
   PubMed Web of Science ®Google Scholar
• Sun, F T., C. Kuo, H T. Cheng, S. Buthpitiya, P. Collins, and M. Griss. 2010. “Activity-Aware Mental Stress Detection Using Physiological Sensors.”. In International Conference on Mobile Computing, Applications, and Services, 282–301. Berlin, Heidelberg: Springer.
   Google Scholar
• Tao, D., H. Tan, H. Wang, X. Zhang, X. Qu, and T. Zhang. 2019. “A Systematic Review of Physiological Measures of Mental Workload.” International Journal of Environmental Research and Public Health 16 (15): 2716. doi:10.3390/ijerph16152716.
   PubMed Web of Science ®Google Scholar
• Tjolleng, A., K. Jung, W. Hong, W. Lee, B. Lee, H. You, J. Son, and S. Park. 2017. “Classification of a Driver’s Cognitive Workload Levels Using Artificial Neural Network on ECG Signals.” Applied Ergonomics 59: 326–332. doi:10.1016/j.apergo.2016.09.013.
   PubMed Web of Science ®Google Scholar
• Van Acker, B. B., D. D. Parmentier, P. Vlerick, and J. Saldien. 2018. “Understanding Mental Workload: From a Clarifying Concept Analysis toward an Implementable Framework.” Cognition, Technology and Work 20 (3): 351–365. doi:10.1007/s10111-018-0481-3.
   Web of Science ®Google Scholar
• van Amelsvoort, L. G. P. M., E. G. Schouten, A. C. Maan, C. A. Swenne, and F. J. Kok. 2000. “Occupational Determinants of Heart Rate Variability[J].” International Archives of Occupational and Environmental Health 73 (4): 255–262. doi:10.1007/s004200050425.
   PubMed Web of Science ®Google Scholar
• Veltman, J. A., and A. W. K. Gaillard. 1998. “Physiological Workload Reactions to Increasing Levels of Task Difficulty.[J.” Ergonomics 41 (5): 656–669. doi:10.1080/001401398186829.
   PubMed Web of Science ®Google Scholar
• Viitasalo, J. H. T., and P. V. Komi. 1977. “Signal Characteristics of EMG during Fatigue[J].” European Journal of Applied Physiology and Occupational Physiology 37 (2): 111–121. doi:10.1007/BF00421697.
   PubMed Web of Science ®Google Scholar
• Warm, J. S., R. Parasuraman, and G. Matthews. 2008. “Vigilance requires hard mental work and is stressful[J]”. Human Factors 50 (3): 433–441. doi:10.1518/001872008X312152.
   Google Scholar
• Wickens, C. D. 2017. Mental Workload: assessment, Prediction and Consequences[C]//International Symposium on Human Mental Workload: Models and Applications, 18–29. Cham: Springer.
   Google Scholar
• Wilson, G. F., and C. A. Russell. 1999. “Operator Functional State Classification Using Neural Networks with Combined Physiological and Performance Features[J].” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 43 (20): 1099–1102. doi:10.1177/154193129904302009.
   Google Scholar
• Wilson, G. F., and C. A. Russell. 2003. “Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks. [J].” Human Factors 45 (4): 635–643. doi:10.1518/hfes.45.4.635.27088.
   PubMed Web of Science ®Google Scholar
• Young, M. S., K. A. Brookhuis, C. D. Wickens, and P. A. Hancock. 2015. “State of Science: mental Workload in Ergonomics[J].” Ergonomics 58 (1): 1–17. doi:10.1080/00140139.2014.956151.
   PubMed Web of Science ®Google Scholar
• Zhao, G., Y. J. Liu, and Y. Shi. 2018. “Real-Time Assessment of the Cross-Task Mental Workload Using Physiological Measures during Anomaly Detection.” IEEE Transactions on Human-Machine Systems 48 (2): 149–160. doi:10.1109/THMS.2018.2803025.
   Web of Science ®Google Scholar