How nervous am I? How computer vision succeeds and humans fail in interpreting state anxiety from dynamic facial behaviour

ABSTRACT

For human interaction, it is important to understand what emotional state others are in. Especially the observation of faces aids us in putting behaviours into context and gives insight into emotions and mental states of others. Detecting whether someone is nervous, a form of state anxiety, is such an example as it reveals a person’s familiarity and contentment with the circumstances. With recent developments in computer vision we developed behavioural nervousness models to show which time-varying facial cues reveal whether someone is nervous in an interview setting. The facial changes, reflecting a state of anxiety, led to more visual exposure and less chemosensory (taste and olfaction) exposure. However, experienced observers had difficulty picking up these changes and failed to detect nervousness levels accurately therewith. This study highlights humans’ limited capacity in determining complex emotional states but at the same time provides an automated model that can assist us in achieving fair assessments of so far unexplored emotional states.

KEYWORDS:

• Nervousness
• state anxiety
• facial behaviour
• computer vision
• emotion

Acknowledgements

We thank the students Roxana Alexandru, Piet Jonker, Sjors van de Ven, Rosemarijn Damen, and Neurolytics employees Juan Rivas, Lianne Hamhuis, Felix Hermsen and Belen Hein for their help during data collection and their contributions to the development of the assessment. This study was financially supported by the NWO take-off valorisation grant (number 17777).

Disclosure statement

Authors MKa and MKu declare no competing interests. Author MN declares to have received equity in a start-up company in exchange for the valorisation, supported by the Dutch Scientific Institute for Research (NWO), of the here reported models.

Author contributions

Authors MKa and MN designed the experiment. Author MKa collected the data. MN analysed the data. Authors MN and MKu wrote the first versions of the paper and author MKa edited the final paper. All authors approved the final version of the manuscript for submission.

Data availability

The data that support the findings of this study can be requested at Neurolytics BV but restrictions apply to the availability of these data, which were provided under licence to the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Neurolytics BV. The code that is used to generate the results of this study can be requested at Neurolytics BV but restrictions apply to the availability of this code, which is developed in collaboration with, and is part of the intellectual property of Neurolytics BV, and so is not publicly available. Enquiries can be sent to [email protected].

Additional information

Funding

This work was supported by NWO take-off [grant number 17777].