Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning

Abstract

This study attempted to multimodally measure mental workload and validate indicators for estimating mental workload. A simulated computer work composed of mental arithmetic tasks with different levels of difficulty was designed and used in the experiment to measure physiological signals (heart rate, heart rate variability, electromyography, electrodermal activity, and respiration), subjective ratings of mental workload (the NASA Task Load Index), and task performance. The indices from electrodermal activity and respiration had a significant increment as task difficulty increased. There were no significant differences between the average heart rate and the low-frequency/high-frequency ratio among tasks. The classification of mental workload using combined indices as inputs showed that classification models combining physiological signals and task performance can reach satisfying accuracy at 96.4% and an accuracy of 78.3% when only using physiological indices as inputs. The present study also showed that ECG and EDA signals have good discriminating power for mental workload detection.

Practitioner summary: The methods used in this study could be applied to office workers, and the findings provide preliminary support and theoretical exploration for follow-up early mental workload detection systems, whose implementation in the real world could beneficially impact worker health and company efficiency.

Abbreviations: NASA-TLX: the national aeronautics and space administration-task load index; ECG: electrocardiographic; EDA: electrodermal activity; EEG: electroencephalogram; LDA: linear discriminant analysis; SVM: support vector machine; KNN: k-nearest neighbor; ANNs: artificial neural networks; EMG: electromyography; PPG: photoplethysmography; SD: standard deviation; BMI: body mass index; DSSQ: dundee stress state questionnaire; ANOVA: analysis of variance; SC: skin conductance; RMS: root mean square; AVHR: the average heart rate; HR: heart rate; LF/HF: the ratio between the low frequencies band and the high frequency band; PSD: power spectral density; MF: median frequency; HRV: heart rate variability; BPNN: backpropagation neural network

Keywords:

• Mental workload
• multi-modal measures
• psychophysiology
• machine learning
• workload classification

Acknowledgments

We thank Kingfar International Inc. (China) for providing related equipment and scientific and technological support. We thank the editor and anonymous reviewers for their valuable comments and advice, which help further to improve the quality of this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work is supported by the National Natural Science Foundation of China [71801002, 71701003, 71802002]; the Humanities and Social Science Fund of Ministry of Education of China [18YJC630023]; the Anhui Natural Science Foundation Project [1808085QG228]; and Ministry of Education Industry-University Cooperation Collaborative Education Project [201901024006].