A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges

Abstract

Affective states, moods and emotions, are an integral part of human nature: they shape our thoughts, govern the behavior of the individual, and influence our interpersonal relationships. The last decades have seen a growing interest in the automatic detection of such states from voice, facial expression, and physiological signals, primarily with the goal of enhancing human-computer interaction with an affective component. With the advent of brain-computer interface research, the idea of affective brain-computer interfaces (aBCI), enabling affect detection from brain signals, arose. In this article, we set out to survey the field of neurophysiology-based affect detection. We outline possible applications of aBCI in a general taxonomy of brain-computer interface approaches and introduce the core concepts of affect and their neurophysiological fundamentals. We show that there is a growing body of literature that evidences the capabilities, but also the limitations and challenges of affect detection from neurophysiological activity.

Keywords:

• brain computer interfaces
• affect
• emotions
• moods
• EEG
• fNIRS

Acknowledgements

The first and the last author contributed to this survey article in equal parts. We gratefully acknowledge the help of Alan Pope, Thierry Pun, and Boris Reuderink, and thank them for their valuable comments, insightful additions, and critical requests for clarifications. It should be noted that parts of Sections 2 and 5 have been revised from [9]. The research of the last author was supported by the National Center of Competence in Research (NCCR) Affective sciences financed by the Swiss National Science Foundation (n° 51NF40-104,897) and hosted by the University of Geneva.

Notes

1. The extension enables a distinction with regard to the use of exogenous stimulation by the system and user volition in a given approach. This seems relevant to cover reactive approaches that are stimulus-dependent, but do not require user volition (e.g., engagement scoring, affective media tagging, preference communication).

2. Since lie-detection is rather used for monitoring than for the adaptation of interaction, lacking thus a direct feedback element, it is rather a marginal instance of affective brain-computer interface systems.

3. Appraisal models and notions of basic or dimensional emotion models are not necessarily incompatible. They can be viewed as treating different aspects of the same object, namely affect, in more detail, specifically the mechanisms which lead to certain affective states. Consequently, appraisal theories have also incorporated the notion of valence (or positive/negative appraisal) and arousal dimensions as underlying structure to the emotional responses.[31,34]

4. BCI systems, by definition, rely at least in part on brain signals, and researchers should strive for the identification of the sources used by the affect classifier. This is especially important for the use of such systems in contexts that preclude the use of non-neural signals, as is the case in communication systems for ALS or locked-in patients.