RSVP IconMessenger: icon-based brain-interfaced alternative and augmentative communication

Abstract

One of the principal application areas for brain-computer interface (BCI) technology is augmentative and alternative communication (AAC), typically used by people with severe speech and physical disabilities (SSPI). Existing word- and phrase-based AAC solutions that employ BCIs that utilize electroencephalography (EEG) are sometimes supplemented by icons. Icon-based BCI systems that use binary signaling methods, such as P300 detection, combine hierarchical layouts with some form of scanning. The rapid serial visual presentation (RSVP) IconMessenger combines P300 signal detection with the icon-based semantic message construction system of iconCHAT. Language models are incorporated in the inference engine and some modifications that facilitate the use of RSVP were performed such as icon semantic role order selection and the tight fusion of language evidence and EEG evidence. The results of a study conducted with 10 healthy participants suggest that the system has potential as an AAC system in real-time typing applications. Ability to construct messages with reduced physical movement demands due to RSVP and increased message construction speed and accuracy due to the incorporation of an icon-based language model in the inference process are the significant findings of this study.

Keywords:

• rapid serial visual presentation
• augmentative and alternative communication
• brain computer interface
• P300 detection

Acknowledgements

This work was supported by the National Science Foundation [IIS-0914808, CNS-1136027, IIS-1149570, SMA-0835976], the National Institutes of Health [R01DC009834], and the National Institute on Disability and Rehabilitation [H133E140026].