Using Multimodal Methods and Machine Learning to Recognize Mental Workload: Distinguishing Between Underload, Moderate Load, and Overload

Abstract

Mental workload recognition is of great significance in preventing human errors and accidents. This study constructed a multimodal recognition scheme to recognize three mental workload states: underload, moderate load, and overload. Based on driving scenarios, these three states were induced in this study by changing the driving modes and situations. Multimodal recognition of underload, moderate load, and overload was performed using electroencephalography (EEG), electrocardiography (ECG), and pupillometry. In addition, various machine learning methods were used to evaluate the recognition performance of different feature combinations. The results showed that the random forest method, trained using spectral power, pupil diameter, and heart rate variability, achieved the highest recognition accuracy of 83.13% for the three mental workload states. This study provides valuable reference information for multimodal recognition of mental workload states.

Keywords:

• Mental workload
• multimodal measures
• machine learning
• workload recognition
• driving simulation

Acknowledgments

We extend our sincere appreciation for their invaluable contributions, which have significantly enhanced the realization of this study. We would like to thank Editage (www.editage.cn) for English language editing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was generously funded by the 74th Postdoctoral General Project of the China Postdoctoral Science Foundation (Grant No. 2023M743062). Additionally, support was received from the Humanity and Social Science Youth Foundation of the Ministry of Education of China (Grant No. 23YJC190038).

Notes on contributors

Zebin Jiang

Zebin Jiang is a PhD candidate at the Center for Psychological Sciences at Zhejiang University. He obtained his bachelor’s degree in Computer Science and Technology from Zhengzhou University in 2021. His research interests include status recognition and human-machine interaction.

Xinyan Li

Xinyan Li is a researcher fellow at the China North Vehicle Research Institute. She has long been engaged in the research of intelligent human-machine interaction-related technologies for special vehicles. She is committed to exploring methods of human-machine interaction and application modes suitable for special vehicles.

Liezhong Ge

Liezhong Ge is a professor at the Center for Psychological Sciences at Zhejiang University. He has been engaged in teaching and research in engineering psychology and cognitive psychology for more than three decades. His research interests include human-machine interaction, user experience and product usability, and facial recognition research.

Jie Xu

Jie Xu is an Assistant Professor at the Center for Psychological Sciences at Zhejiang University. He obtained his PhD in industrial and systems engineering from the University of Wisconsin–Madison in 2016. His research interests include human-agent interaction, physiological computing, and system safety.

Yandi Lu

Yandi Lu is a PhD candidate at the Center for Psychological Sciences at Zhejiang University. He has an interdisciplinary background in design and psychology. His research interests include human–computer interaction, user experience, and human-centered AI.

Yijing Zhang

Yijing Zhang is currently a postdoctoral fellow at the Center for Psychological Sciences at Zhejiang University. He employs cognitive neuroscience approaches to delve into various aspects of research. His areas of focus include the investigation and regulation of mental workload, human-machine interaction, and special vehicle driver state recognition.

Ming Mao

Ming Mao is an academician of the Chinese Academy of Sciences. He has long been engaged in theoretical research and engineering practice on the overall design of armored weapons, drive systems, action systems, and other technologies.