Predicting changes in performance due to cognitive fatigue: A multimodal approach based on speech motor coordination and electrodermal activity

Abstract

Objective

Military job and training activities place significant demands on service members’ (SMs’) cognitive resources, increasing risk of injury and degrading performance. Early detection of cognitive fatigue is essential to reduce risk and support optimal function. This paper describes a multimodal approach, based on changes in measures of speech motor coordination and electrodermal activity (EDA), for predicting changes in performance following sustained cognitive effort.

Methods

Twenty-nine active duty SMs completed computer-based cognitive tasks for 2 h (load period). Measures of speech derived from audio were acquired, along with concurrent measures of EDA, before and after the load period. Cognitive performance was assessed before and during the load period using the Automated Neuropsychological Assessment Metrics Military Battery (ANAM MIL). Subjective assessments of cognitive effort and alertness were obtained intermittently.

Results

Across the load period, participants’ ratings of cognitive workload increased, while alertness ratings declined. Cognitive performance declined significantly during the first half of the load period. Three speech and arousal features predicted cognitive performance changes during this period with statistically significant accuracy: EDA (r = 0.43, p = 0.01), articulator velocity coordination (r = 0.50, p = 0.00), and vocal creak (r = 0.35, p = 0.03). Fusing predictions from these features predicted performance changes with r = 0.68 (p = 0.00).

Conclusions

Results suggest that speech and arousal measures may be used to predict changes in performance associated with cognitive fatigue. This work supports ongoing efforts to develop reliable, unobtrusive measures for cognitive state assessment aimed at reducing injury risk, informing return to work decisions, and supporting diverse mobile healthcare applications in civilian and military settings.

Keywords:

• Military
• cognition
• workload
• speech
• neuromotor coordination

Acknowledgments

We thank the Army Soldiers who were participants in the study and the past and present USARIEM Military Performance Division members for their assistance and support in data collection efforts. We would also like to thank Drs. Katheryn Taylor and Daryush Mehta for their assistance with data analyses, Drs. Joseph Seay, William Tharion, and Kenneth Pitts for their assistance with initial protocol design and data collection, and Ms. Nicole Ekon, Ms. Caitlin Ridgewell, Ms. Audrey Hildebrandt, and Dr. Gregory Ciccarelli for their assistance with manuscript preparation.

Note

Author AL completed a portion of this work while employed at MIT Lincoln Laboratory. Author KF completed a portion of this work as an Oak Ridge Institute for Science and Education (ORISE) Research Fellow at USARIEM.

Disclosure statement

The opinions or assertions contained herein are the private views of the author(s) and are not to be construed as official or as reflecting the views of the Army, Air Force, or Department of Defense. Use of trade names is for identification only and does not imply endorsement by the Department of the Army, Department of the Air Force, or the U.S. Department of Health and Human Services.

Additional information

Funding

This work was supported in part by the U.S. Army Medical Research and Development Command (USAMRDC), Department of the Army under Air Force Contract No. FA8702-15-D-0001, and by an appointment to the Department of Defense (DOD) Research Participation Program administered by the Oak Ridge Institute for Science and Education (ORISE) through an interagency agreement between the U.S. Department of Energy (DOE) and the DOD. ORISE is managed by ORAU under DOE contract number DE-SC0014664.