Continuous affective states recognition using functional near infrared spectroscopy

Abstract

Monitoring the affective states of a person can be highly relevant for numerous disciplines, including adaptive user interfaces, entertainment, ergonomics, medicine and therapy. In many situations, the affective state of a user is not easily observable from outside by audio or video, but may be identified by a brain-computer interface (BCI). Functional near-infrared spectroscopy (fNIRS) is a brain imaging modality gaining rising attention in the BCI community. However, fNIRS emotion recognition studies have only analyzed stimulus-locked effects. For realistic human-machine interaction scenarios, the point of time of an emotion-triggering event and the time span of an affective state are unknown. In this paper, we investigate a BCI that monitors the affective states of the user continuously over time (i.e. asynchronous BCI). In our study, fNRIS signals from eight subjects have been recorded at eight prefrontal locations in response to three different classes of affect induction by emotional audio-visual stimuli plus a neutral class. Our system evaluates short windows of 5 s length to continuously recognize affective states. We analyze hemodynamic responses, present a careful evaluation of binary classification tasks, compare time-domain and wavelet-based signal features, and investigate classification accuracies over time.

Keywords:

• functional near-infrared spectroscopy
• affect recognition
• emotion recognition
• continuous
• asynchronous
• brain-computer interfaces

Notes

1. The following IAPS picture items have been used:

VA: 4001, 4002, 4003, 4006, 4141, 4142, 4150, 4180, 4220, 4225, 4232, 4235, 4250, 4255, 4274, 4275, 4279, 4290, 4300, 4310, 4311, 4607, 4608, 4651, 4652, 4659, 4660, 4664, 4670, 4683, 4694, 4695, 4800, 8030, 8080, 8185, 8400, 8501; Va: 1440, 1441, 1460, 1560, 1590, 1600, 1601, 1602, 1603, 1604, 1610, 1620, 1640, 1670, 1900, 2000, 2050, 2070, 2170, 2260, 2304, 2341, 2360, 2370, 2501, 5000, 5010, 5020, 5030, 5200, 5551, 5720, 5760, 5800, 5891, 7080, 7325, 7545, 7900; vA: 3000, 3010, 3015, 3030, 3051, 3053, 3060, 3062, 3063, 3068, 3069, 3071, 3080, 3101, 3102, 3110, 3120, 3160, 3168, 3170, 3181, 3250, 3261, 3266, 3301, 3400, 3500, 3530, 6230, 6260, 6313, 6510, 6540, 9400, 9405, 9410, 9433, 9570, 9635.1, 9810

The following IADS sound items have been used:

VA: 201, 202, 311, 215, 815, 716, 360, 200, 366, 717, 817, 205, 352, 367, 415, 204, 353, 216, 110, 355; Va: 112, 150, 151, 171, 172, 230, 262, 270, 370, 374, 377, 602, 705, 721, 725, 726, 809, 810, 811, 812; vA: 115, 260, 275, 276, 277, 278, 279, 284, 285, 286, 290, 292, 420, 422, 424, 600, 624, 709, 711, 712