Augmenting the Instant Messaging Experience Through the Use of Brain–Computer Interface and Gestural Technologies

Abstract

Interpersonal communication benefits greatly from the emotional information encoded by facial expression, body language, and tone of voice. However, these cues are often lost when sending text-based instant messages. This article describes a novel approach utilizing gestural and brain–computer interface (BCI) input to replace the missing emotional cues, with the aim of augmenting the instant messaging process. A set of exploratory studies were conducted using a commercially available BCI headset. An initial study validated the emotional data automatically captured by the device. Subsequently, an instant messaging application was developed, which detected emotions and facial gestures that are presented to the user's chat partner via progress bars and an avatar. Findings from an evaluation revealed that the novel approach facilitates communication containing a greater percentage of affective terms compared with traditional, text-based instant messaging environments. Strong levels of confidence were expressed when using the system to both convey and infer affective states, contributing to a rich subjective user experience.

ACKNOWLEDGMENTS

We thank Dr. Andrew Sears (RIT) and Dr. Wayne G. Lutters (UMBC) for their input into this work.

Notes

¹Channel names based on the International 10–20 locations include AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4.