Brain-computer interface for the communication of acute patients: a feasibility study and a randomized controlled trial comparing performance with healthy participants and a traditional assistive device

Abstract

This study presents the outcome of the 5-year-long French national project aiming at the development and evaluation of an effective brain-computer interface (BCI) prototype for the communication of patients with acute motor disabilities. It presents results from two clinical studies: a clinical feasibility study carried out partly in the intensive care unit (ICU) and the clinical evaluation of an innovative BCI prototype. In this second study the BCI performance of patients was compared to that of healthy volunteers and benchmarked against a traditional assistive technology (scanning device). Altogether, 15 of 22 patients could control the BCI system with an accuracy significantly above the chance level. The bit-rate of the traditional assistive technology proved superior, even though an equivalent bit-rate could be achieved using personalized parameters for the BCI. Fatigue was found to be the primary limitation factor, which was particularly true for patients and during the use of the BCI. A classifier based on Riemannian geometry was found to contribute significantly to the accuracy of the BCI system. This study demonstrates that the communication of patients with severe motor impairments can be effectively restored using an adequately designed BCI system. All electrophysiological data are freely available at the Physionet.org platform.

Keywords:

• Assistive technology
• communication aids
• event-related potentials, P300 speller
• electroencephalography
• Riemannian geometry

Introduction

Lack of communication in the hospital may be a great source of distress for both patients and caregivers, which is particularly exacerbated in the acute context of intensive care units (ICUs) [1]. Several neurological disorders impair communication abilities while leaving cognitive capacities almost untouched. This has long been reported for chronic conditions such as myopathies, spinal cord injuries, amyotrophic lateral sclerosis (ALS), and multiple sclerosis (MS), but is also true for acute neuropathies such as the Guillain-Barré syndrome (GBS). Moreover, quadriplegic or locked-in syndrome (LIS) patients undergoing invasive mechanical ventilation (via endotracheal tube or tracheotomy) see their communication abilities impaired by the sudden loss of speech [2].

The use of computers in this context can facilitate communication and many technical solutions to facilitate computer access are available [3]. In particular, occupational therapists equip patients with ‘scanning’ systems [4–6] enabling interaction with computers or other devices such as speech synthesizers. The assistive technology (AT) in this case consists of a ‘click interface’ (or switch) connected to an application where a limited number of options are visually ‘scanned’ (highlighted) for the patient. There are many types of available switches to control a scanning system [7]: muscular switches, tactile switches, puff switches, and mechanical switches, the most common type in France, which consist of a press-down button whose activation force, size, location (feet, near the head, thumb), and type of activation (validation on press-down or release) are set according to the patient’s specific condition. Whenever a reliable switch command can be obtained, the system is connected to a software interface. Typically, this consists of a grid of symbols displayed on a screen, which are flashed at a regular pace (i.e. the ‘scanning’) until the patient selects one. The configuration of both the physical and the software interface requires a tremendous amount of time and expertise, which is one reason why, as a matter of fact, patients with severe motor disabilities in acute contexts such as the ICU are often left unequipped.

The brain-computer interface (BCI) is a longstanding technology [8] that translates the brain electrical activity into a command for a device [9,10]. BCIs are usually characterized by: (1) a brain activity recording modality (2), a paradigm that relates sensorial stimulations to some specific brain activity, and (3) an interface connecting to the front-end application by means of pattern-recognition techniques. Electroencephalography (EEG)-based BCIs [11,12] have been extensively studied since EEG is a non-invasive and portable neuroimaging modality. In this article, we focus on BCI technology based on the P300 event-related potential (ERP). A P300 is a positive ERP occurring 300 to 500 ms after the presentation of a rare meaningful stimulus presented in a flow of many irrelevant stimuli[13]. The repeated presentation of several stimuli among which there is a possible target for the user is referred to as the ‘oddball paradigm’[13–15]. The ‘P300 speller’ is currently the best-known and most widespread ERP-based BCI communication interface. In a typical P300 speller a regular grid of symbols is displayed on the screen while lines and columns are flashed at random. The participant is asked to concentrate on a target symbol. A P300 is elicited in response to the flash of the line and the column containing the target symbol. Since the amplitude of the P300 is much smaller as compared to the EEG background activity and EEG artifacts, the ERP is elicited several times and averaged in order to increase the signal-to-noise (SNR). The averaging procedure increases the discrimination ability (target ERPs vs. non-target ERPs), effectively resulting in a trade-off between accuracy and speed of selection. The P300 speller represents a great hope for severely disabled patients, especially for those who are not able to use traditional AT.

Despite the tremendous amount of research over the past 20 years, EEG-based BCIs still suffer from significant drawbacks such as setup time and sensitivity to noise [16]. For instance, the placement of electrodes connected to the scalp skin through a conductive gel is a time-consuming process, certainly prohibitive for everyday use in both hospitals and at the patient’s home. Moreover, caregivers are usually not familiar with EEG technology. Attempts to overcome this limitation include the use of caps with pre-positioned gel-based [16,17] or dry [18–20] electrodes, possibly interfaced with active recording systems [21,22] Recently, the EPOC headset [23,24] has opened the path to low-cost EEG technology for the general public, although the quality of the signal (hence the overall BCI performance) has proved inferior to research-grade EEG acquisition systems [25,26]. Thus, to date, a state-of-the-art EEG-based BCI requires the support of a technical and clinical team composed of highly qualified professionals [27–29]. A major concern for the present study is the possibility of generating and processing visual ERPs in an ICU environment overflowing with uncontrolled sources of noise, both acoustic (alarms, staff, mechanical ventilation) and electromagnetic (bedside monitors, automated syringes, mechanical ventilation). While EEG applications are known to be particularly sensitive to non-controlled environments [16,30], to the best of our knowledge no report exists about their use in an ICU.

P300-based BCI also have specific limitations. First of all, since several ERPs need to be averaged to obtain a sufficient SNR, the overall bit-rate is low in practice (up to 1 minute per letter in a P300 speller) [31,32]. The signal processing and machine learning research communities actively address this issue by improving ERP detection methods [33–35]. Also, the use of a P300 speller requires constant concentration and there is still a debate on whether some clinical populations possesses the necessary attention span to effectively control a P300 speller, especially if the session is prolonged [36,37]. This is particularly true for populations of patients admitted to the ICU where central nervous system (CNS) depressant drugs are usually administered as they are known to influence vigilance and attention [38]. It is worth noting that while P300 speller BCI systems have traditionally been clinically evaluated in populations of chronic patients such as those suffering from ALS [27,28,30,39], its potential benefit in acute nervous conditions, such as GBS, have not yet been reported.

To complete the technological transfer from ‘bench to bedside’, BCI must gain ease of use and robustness in terms of both algorithms and interface (signal-processing and applications). Overcoming the aforementioned technical challenges appears important, particularly in the context of a better assessment of pain and cognitive function (perhaps identification of delirium) of non-communicating patients. This would certainly result in an improved quality of ICU stay. The Robust Brain-computer Interface for virtual Keyboard (RoBIK) project [40] aimed at the development of a BCI system for the communication of patients that could be used on a daily basis in the hospital with limited external intervention. In order to achieve this goal a multidisciplinary approach was chosen and developments were carefully framed by clinical specifications and validation, according to a user-centered design. The project resulted in two clinical studies:^{Appendix 1, Appendix 2}

•

Study 1: Identification of the clinical and technical limitations affecting the use of BCI applications for the communication of acute patients in a clinical setting; a questionnaire was administered to patients and caregivers [41] in order to identify appropriate user specifications;

•

Study 2: the clinical validation of a BCI system developed with specifications derived from Study I, which consisted of ◦ An EEG headset with portable electronics designed for convenience of use in the hospital (Appendix 1), ◦ dedicated software comprising a user-friendly interface and a classification algorithm based on Riemannian geometry (Appendix 2); the performance of this BCI prototype on clinical samples was compared with a state-of-the-art assistive device for the communication of patients (scanning device), in order to assess the actual clinical value of the two technologies, and withhealthy volunteers, in order to quantify the drop in performance induced by the patients’ condition.

Finally, an offline analysis of the data collected during these two clinical trials was carried out in order to quantify the contribution of each system’s element (hardware and software) to the whole and to estimate the performance of a personalized BCI. The details of this analysis are available in Appendix 3.

Materials and methods

Table 1 summarizes the populations, hardware, software, algorithms, and cross-validation techniques used in each clinical trial.

Table 1. Summary of materials and methods for both studies showing the population, hardware, software, algorithms, and cross-validation procedures. Abbreviations: minimum distance to mean (MDM) is a Riemannian classifier; support vector machine (SVM) is a classifier. xDAWN is a spatial filter for event-related potentials.

	Population	Hardware	Software	Algorithms	Cross-validation
Study I	• Patients (P1)	• TMSi	• OpenViBE	• xDAWN • SVM	• 20-Fold (Training) • Cross-session
Study 2	• Patients (P2) • Volunteers (V2)	• RoBIK	• OpenViBE • Python 2.7	• Riemannian Potato • MDM	• 5-Fold (Training) • Cross-session

Study 1: clinical feasibility

The aim of this study was to evaluate the usability of a state-of-the-art P300 speller BCI for communication [15] in the challenging environment of intensive care and rehabilitation units.

Study design and protocol

The protocol was registered on ClinicalTrials.gov (NCT01005524) and received clearance from local ethics boards (‘Comité de protection des personnes’, CPP, Saint Germain-en-Laye, 2009/10/15). Quadriplegic patients admitted to ICU and rehabilitation unit at a tertiary-care hospital were enrolled after giving their informed consent. Pregnant women, illiterate patients, patients under judicial protection or without social security as well as patients with a history of epilepsy were excluded from the study.

The protocol consisted of three P300 speller sessions [15].

(1)	A training session was given during which patients were instructed to spell two words (‘OCEAN’ and ‘NUAGES’, meaning in French ‘Ocean’ and ‘Clouds’, respectively) and for which no feedback was provided. Signals collected were then used to train a state-of-the-art pattern-recognition algorithm based on a spatial filter and a classifier, detailed in the next section [33].
(2)	The classifier performance was evaluated during an ‘online’ session during which instructions were given for three words (‘OEUFS’, ‘AVION,’ and ‘OASIS’, meaning in French ‘eggs’, ‘airplane’, and ‘oasis’, respectively). The letters identified by the algorithm after each sequence of stimulations were displayed on the screen as a feedback.
(3)	Finally, an optional ‘free-spelling’ session was offered to the patient, during which no instruction was given and patients could spell whatever they wished.

At the end of the session patients were requested to rate their satisfaction with the BCI system as a communication tool by means of a visual analogue scale (VAS) ranking from 0 (completely unsatisfied) to 10 (completely satisfied).

Data acquisition and processing

Twenty-four silver chloride (AgCl) disk-electrodes were connected by active-shielded coaxial cables to a Porti32 EEG amplifier (TMSi, Twente, Netherlands) sampling at 512 Hz. Signal acquisition, processing, and storage were performed using the open-source platform OpenViBE [42]. Raw EEG signals were stored as GDF format [43] including the type and location of visual stimulations. This data-set is freely available on the Physionet platform ¹

As pre-processing, all EEG signals were band-pass-filtered between 0.01 and 30 Hz by means of a Butterworth fourth-order filter with linear phase response. The xDAWN spatial filter [33] was trained on the training session. xDAWN is a spatial filter specifically designed for ERP data [44]. The aim of a spatial filter in this context is to enhance the signal of interest (ERPs) while suppressing the background noise [45]. It also dramatically reduces the dimensionality of EEG data, since a few filters suffice to summarize the signal of interest (ERPs). The spatially filtered signal is fed to an ensemble of support vector machine (SVM) classifiers for identification of target and non-target stimulation [46].

Cross-validation scheme

For each participant the accuracy was assessed with a 20-fold cross-validation procedure on the training session. A K-fold validation [47] is a technique often used to estimate the performance of a classification technique (and its variability) on ‘unseen’ data. At each fold, 20% of the data was left aside for model validation (the ‘test’ set) while the remaining 80% (the ‘training set’) was used to design the model (the xDAWN spatial filter and the ensemble of SVMs). Online letter-recognition accuracy during the test session was simply obtained by applying models derived from the training session, live, to the online data.

Technical developments

Following this feasibility study and results from a survey conducted with patients and caregivers [41], we identified optimal hardware and software BCI specifications, which were implemented as described in this section and used for the second clinical study. Technical details can be found in Appendices 1 and 2.

The EEG headset

The RoBIK EEG headset, shown in Figure 1 (left), has been designed to be an effective means of recording EEG activity usable by people not familiar with EEG technology, while offering performance equivalent to that of a clinical-grade EEG system. The resulting headset has silver chloride (AgCl) electrodes mounted at the following 14 standard 10/20 locations: Po7, Fp1, Oz, Fz, C3, F4, Pz, C4, P7, Fp2, F3, P8, Po8, and Cz. Connection to the scalp skin is obtained by means of a cotton pad soaked with physiological saline solution. The signal is sampled at 580 Hz with 12-bit resolution.

Figure 1. The RoBIK prototype: the RoBIK headset (left) with 14 wet EEG channels and the virtual keyboard interface ‘Brainmium’ (right).

The user interface

The application and generation of visual stimulations were handled by a dedicated application called ‘Brainmium’ (Figure 1, right). It was designed to be user-friendly so that a caregiver with basic knowledge of informatics can operate it after a short training. Brainmium implements state-of-the-art P300 paradigms, using random inter-stimulus interval (ISI) and random group flashing (different from the line-column paradigm), which are meant to reduce visual fatigue and increase discriminatory power. Details can be found in Appendix 2.

Online artifact rejection

In this work we use a signal quality index (SQI) in order to reject trials contaminated by excessive artifacts during both the training and the online phase of the experiment. The SQI is an improvement of the ‘Riemannian potato’ methods [48] based on the suggestion from Congedo [49]. Technical details are given in Appendix 2.

Online classifier

The online classification was performed by means of the Riemannian minimum distance to means (MDM) classifier [50] as applied to P300 data [49]. The standard method was complemented with a new logistic decision function as detailed in Appendix 2.

Study 2: clinical evaluation

The primary objective of this study was to evaluate the performance of the RoBIK BCI prototype and to compare its performance to that of a traditional assistive technology. The second objective was to compare the performance of patients using the BCI prototype to that of healthy volunteers.

Study design and setting

The protocol was registered on ClinicalTrials.gov (NCT01707498) and received clearance from the local ethics board (CPP de Saint Germain-en-Laye, 2012/07/05) and the regulatory agency for the use of a non-CE-marked device (Agence Nationale de Sécurité des Médicaments, ANSM, 2012/09/14, 2012-A00613–40). The clinical protocol inclusion criterion for patients was the presence of functional quadriplegia. The inclusion criterion for healthy participants was age greater than 18 years. Patients were enrolled in the intensive care unit (ICU) and rehabilitation units at a tertiary care hospital after giving their informed consent. Healthy volunteers were enrolled at the Center for Clinical Investigation and Technological Innovation (CIC-IT) after giving their informed consent. Pregnant women, patients with hemodynamic instability, illiterate participants, patients under judicial protection or without social security, epileptic patients, and participants showing skin/scalp sensitivity or severe visual impairment or aged less than 18 years were excluded from the study.

The experimental design is shown in Figure 2; patients were randomly assigned to try the RoBIK prototype or the scanning device (described below) first. Because there are no well-documented learning effects in P300-based BCIs [51], the RoBiK prototype was evaluated only over a single half-a-day session. Instead, the scanning speller performance was estimated over three sessions in order to estimate a possible learning effect [52]. Each session was separated by at least a half-day wash-out period in order to avoid a possible bias due to fatigue.

Figure 2. Patients were randomly assigned to try first the RoBIK BCI or the scanning device. The RoBIK prototype was evaluated within half a day while the scanning device was evaluated over three sessions in order to account for a possible learning effect. Four different texts of equivalent difficulty were selected so that the RoBIK device was evaluated with text 1 or 4.

At the end of each session we assessed the fatigue and satisfaction of patients by means of a VAS ranging from 0 (very tired / very unsatisfied) to 10 (not tired at all / very satisfied). In both cases participants were asked to fill a questionnaire to express their opinion on both techniques (including comfort and fatigue).

Bedside procedure

The RoBIK prototype was setup by an occupational therapist who had no specific training in electrophysiology. The protocol consisted of the following steps:

(1)	The EEG headset was set up and time was counted from the beginning of the installation to the beginning of step 2; the operator was instructed to reduce the presence of power line contaminations (50 Hz) by adjusting electrode positioning until the presence of eye blinks, jaw muscular artifacts, and alpha waves could be observed when the patient was instructed to blink, clamp his jaw, and close his eyes, respectively. The impedance between each electrode and the reference was measured at the beginning and end of each session and maintained below 5 kΩ.
(2)	The interface was presented to the patient and directions for the training sessions were given. When possible, the participant was asked to re-formulate to make sure the task was correctly understood.
(3)	The training session consisted of the spelling of 10 consecutive characters. The participant was instructed to concentrate on the chosen letter and count the number of times it flashed. Each letter was flashed 20 times. The target letter was continuously indicated in the upper section of the screen and each new letter was notified by printing in blue the corresponding key on the virtual keyboard before each new sequence of random stimulation.
(4)	Data collected during this training session were automatically passed down the processing pipeline for training to the MDM classification algorithm. The data from the training sessions were split into five distinct training and test sets in order to estimate the ‘real accuracy’ (percent of correctly identified characters within 36 choices) on unseen data. The patient was allowed to continue the protocol and pass to the ‘online’ session if the estimated accuracy was above 70%, meaning that, on average, three spelling errors every 10 characters were tolerated, otherwise the participant was discharged from the study.
(5)	The ‘online’ session was equivalent to the previous one except that visual feedback was provided to the patients by the classification algorithm. The number of repetitions for the online session was chosen to maximize the estimated accuracy on the training data. The online session was timed to last exactly 10 minutes, during which participants were instructed to spell as many letters as possible without corrections.
(6)	Finally, patients could use the interface during a ‘free-spelling’ session where no instruction was given, but an output was still provided.

These data are freely available on the Physionet platform ².

The scanning system

An occupational therapist, as part of every day’s clinical activity, was in charge of equipping patients with a ‘switch interface’ (‘Buddy button’ switch, Ablenet, Roseville, MN, USA). The nominal activation force of the switch varied from 10 to 600 grams. Depending on the specific condition of the patient the following parameters were tuned by the occupational therapist: activation force, size, location (feet, near the head, thumb), type of activation (validation on press-down or release), and possible filtering of double-clicks. Once a reliable switch command could be obtained, the system was connected via the ‘Joycable’ USB interface (Sensory software, Malvern, Worcestershire, UK) to the KEYVIT virtual keyboard (Jabbla, Ghent, Belgium). On this interface, a regular grid of symbols flashes lines at a regular pace (i.e. the ‘scanning’) until the patient selects one line by activating the switch. Once a line is selected, each element of the line is then flashed until a second click selects the desired symbol or letter. After installation of the interface, patients were asked to spell as many characters as they could within 10 min. The instruction text was printed big enough so that it could be seen as displayed on an A4 sheet of paper next to the screen.

Statistical analysis

For all tests, samples were tested for normality by means of the Jarque-Bera test (significance level at 5%) and analyzed with non-parametric or parametric tests depending on whether the hypothesis of normal distribution of the data was rejected or not, respectively. Tests for comparing central location parameters (means) were chosen paired or unpaired according to the test at hand. Differences between the healthy volunteers and patients (unpaired) for the variables age, setup time, comfort, performance during the training and online sessions were tested using the Student t-test or the Mann Whitney U-test for normally and non-normally distributed samples, respectively. Comparison of fatigue, setup time, and spelling accuracy in patients (paired) using the RoBIK and the scanning system (paired statistics) was assessed with a paired Student t-test or a Wilcoxon test depending on whether data were found to be normally distributed or not. Comparison of fatigue in patients for each interface was estimated with a repeated-measures ANOVA that uses a multivariate framework (Hotelling T-square) to account for correlation between measures [53], which therefore does not need correction for sphericity [54]. Comparison of fatigue between healthy volunteers and patients using the RoBIK interface was estimated with the same technique considering two independent groups of participants. For all statistical tests the tolerance for type I error was set to 0.05. All analyses were performed using Matlab (Version 8.0.0.783 - R2012b) and toolboxes referenced.

Results

Study 1: clinical feasibility

Patients

Twelve quadriplegic patients admitted in adult medical ICU [7] and rehabilitation units [5] met the inclusion criteria and were consecutively included in the study after giving their informed consent. Table 1 summarizes the sample, composed of eight men (67%) and four women, aged from 22 to 63 years old. The tolerance was good in all patients. Four patients (33%) did not complete the protocol. Access to the occipital area was complicated by a central catheter in the first patient (#01). In one patient a technical issue interrupted the inclusion (#02). One patient fell asleep during the training session (#10). The last patient (#11) was suspected to have eyesight problems, although this could not fully be confirmed by medical files.

P300 speller for communication

The results of all patients included in Study 1 are presented in Table 2. Out of 10 patients who completed the training session, eight used the system during a test session with accuracy significantly above chance (2.6%) level (rank sum test, W = 100, p < .001); seven could use the system with more than half the symbols correctly identified and an average accuracy of 84.7%. One patient had an accuracy of 40% and the BCI technique did not work for two patients. Five out of eight patients (62.5%) who completed the test session asked for the optional free spelling session and spelled on average 10 characters with a median accuracy of 100.0% (72.2,100). Interestingly, the spelled words were all the first names of relatives. The overall satisfaction for the technique amongst participants who completed the training sessions was high (7.3/10).

Table 2. Description of the population included in Study 1 with results: AC, access to computer; HM, voluntary head mobility; OE, oral expression; MV, mechanical ventilation, ET, endotracheal intubation. ‘Treatments’ details drugs administered to patients during the 48 h preceding the inclusion in the study. Training session performance is evaluated with a 20-fold cross-validation procedure on the training data using a combination of xDAWN and SVM. It shows discrimination – area under the receiver operating curve (AUROC) – between targets (33% of stimuli, i.e. One line and one column) and non-targets (random classifier is 50%). Testing and online performance shows accuracy in letter selection (1 symbol in 36; chance is 2.7%). The last lines of the table summarize the distributions: categorical variables are represented by the proportion of each category (noted in parenthesis) and continuous variables are represented by the mean and standard deviation; whenever relevant, the sample size is indicated in parenthesis. For speller performance, brackets indicate the average number of letters per patient. P-values are computed with a Mann-Whiteney U-test (*).

ID	Type of Service	Age	Gender	Origin of handicap	HM	OE	AC	MV	ET	Treatments (up to 48 hours)	Training session AUROC (# letters)	Testing session Percent Accuracy (# letters)	Free spelling Session Percent Accuracy (%) (# letters)	Satisfaction EVA	Details
1	ICU	21	Female	Functional post-surgery Quadriplegia (severe idiopathic scoliosis)	Y	Y	N	Y	Y	Sufentanyl, Midazolam, Gabapentine, Hydroxyzine	N.A.			N.A.	EEG leads could not be setup
2	ICU	62	Female	Post-traumatic quadriplegia C5-C6	Y	N	N	Y	Y	Alprazolam, Hydroxyxine, Prégabaline, Escitalopram, Alimemazine	80.0% (20)			N.A.	Technical issue with acquisition software
3	ICU	30	Male	Post traumatic spinal cord hematoma C1-C5	N	N	Y	Y	Y	Baclofene, Clonazepam, Sodium Valproate, Hydroxyzine	73.0% (15)	60.00% (5)		N.A.
4	ICU	25	Male	Duchene’s muscular dystrophy	N	N	Y	Y	Y	Tramadol , Alprazolam, Mirtazapine, Hydroxyzine	85.0% (10)	100.00% (15)	100.00% (8)	9
5	ICU	22	male	Duchene’s muscular dystrophy	Y	N	Y	Y	Y	None	68.0% (10)	73.30% (11)	72.70% (11)	7.6
6	Rehab	32	male	Post-traumatic Quadriplegia C4	N	N	Y	N	Y	Zopiclone , Escitalopram, Alprazolam	89.0% (10)	100.00% (15)	100.00% (6)	9
7	Rehab	58	male	Locked-In Syndrome (LIS, Stroke)	Y	Y	Y	N	Y	Bromazepam, Baclofene	77.0% (10)	93.30% (14)	100.00% (15)	9.9
8	Rehab	36	male	LIS (Head injury)	Y	Y	Y	N	Y	Baclofene	67.0% (10)	86.70% (13)	53.80% (13)	8.5
9	Rehab	46	female	LIS (Stroke)	Y	Y	Y	N	Y	Baclofene	69.0% (10)	40.00% (6)		5.5	Problem of concentration related to swallowing issues
10	ICU	34	female	Guillain-Barré Syndrome (SGB)	N	Y	N	Y	N	Tramadol, Hydroxyzine, Gabapentine, Zopiclone, Clonazepam	N.A.			N.A.	Painful Guillain Barré Syndrome with sleep deprivation
11	Rehab	59	male	LIS (Stroke);	Y	Y	Y	N	Y	Zolpidem, Gabapentine	65.0% (10)			3.8	Possibly bad eyesight
12	ICU	22	male	Duchene’s muscular dystrophy.	N	N	N	Y	Y	None	75.0% (10)	80.00% (4)		4.8	Patient was bored and interrupted the protocol
	58% (ICU)	37.7±15.3	66% (male)	Neuromuscular (33%)	58% (yes)	50% (yes)	67% (yes)	58% (yes)	92% (yes)	Sedated (8%)	74.8±8.0 (n=10) [11]	79.2±20.9 (n=8) [12]	85.3±21.2 (n=5) [10]	7.3	Mean ± std
				Stroke (15%)						Use of CNS depressant (75%)	76.2±6.5 (n=5)	78.3±16.6 (n=4)	86.4±19.3 (n=2)		ICU
				Trauma (42%)							73.4±9.8 (n=5)	80.0±27.2 (n=4)	84.6±26.7 (n=3)		Rehabilitation
											0.46	0.62	-		p-value*

Study 2: clinical evaluation

Participants

Table 3 summarizes demographic and other variables of the participants.

Table 3. Participants enrolled in Study 2 (patients and healthy volunteers). Abbreviations: ABP, arterial blood pressure – systolic (s) and diastolic (d); bpm, beats per minute; F, female; GBS, Guillain-barré syndrome; HR, heart rate; LIS, locked-in syndrome, M, male; mmHg, millimetres of mercury; MS, mulitple sclerosis; Q, quadriplegic; S, scanning device.

Patients
Age	46	47		56	60	39	35	64	30	52	37
Gender	M	M		F	M	F	M	m	F	F	M
Pathology	LIS	GBS		MS	LIS	LIS	Q	Q	Q	Q	GBS
HR (bpm)	64	76		–	65	101	47	60	58	–	96
ABPs / ABPd (mmHg)	130 / 80	111 / 72		–	150 / 110	111 / 73	58 / 78	130 / 60	8646	–	85 / 67
Randomization	RoBIK	S		RoBIK	S	RoBIK	S	RoBIK	RoBIK	S	S
Healthy volunteers
Age	33	50	26		31	32	28	21	21	22
Gender	M	F	M		F	F	M	M	M	F
HR(bpm)	51	75	26		31	–	77	57	61	69
ABPs / ABPd(mmHg)	123 / 75	111 / 84	123 / 69		112 / 82	–	108 / 82	122 / 65	111 / 66	130 / 87

As seen in Table 3, the patients recruited in this study were on average 46.5 years old (37.0–56.0) and the healthy volunteers were on average 28.0 years old (21.8–32.2). The age difference was statically significant (rank sum test, W = 136, p = .002). One patient was discharged prematurely from the study on his request. Two additional patients did not meet the threshold of performance (cross-validated accuracy on training session above 70%) and were discharged from the study.

Comparison of patients and healthy volunteers populations

For the RoBIK system, the setup time was 11.0 min (8.5–15.5) for patients and 13.0 min (10.5–14.2) for healthy volunteers (Figure 3). The difference was not found statistically significant (rank sum test, W = 66, p = .58). Likewise, the experience was rated equally pleasant (rank sum test, W = 85, p = .23), with patients reporting an average rating of 2.5 (1.4–3.0) and healthy participants an average rating of 1.5 (1.2–1.9). The comparison of information transfer rate (bit-rate) between patients and healthy controls (Figure 4) did not reveal significant differences (F(1,18) = 3.90, p = .07) even though a significant time factor (F(2,18) = 24.46, p < .001) and time-group interaction effect was found (F(2,18) = 0.55, p < .001), indicating that the drop in EEG discriminatory power between the calibration and the test session was greater in patients as compared to healthy volunteers. This might indicate that the evoked potential amplitude (P300) of patients decreases faster over time, which in turn might relate to changes in attention. This interpretation is corroborated by the results on self-reported fatigue, showing that although no difference between the populations can be found (F(3,15) = 1.00, p = .33) and both samples get tired during sessions (time factor, F(3,15) = 9.85, p = .001), the effect is significantly stronger in patients (interaction factor time-population, F(3,15) = 3.44, p = .049).

Figure 3. Comparison of setup time (min), self-evaluated comfort, spelling accuracy (%), and bit-rates (bit/min) between the RoBIK system (online results) and the three sessions of the scanning device.

Figure 4. Comparison of the evolution of fatigue (left) and information rate (right) for patients (dark line) and healthy volunteers (bright line) over three times of the BCI protocol (train, online, free).

Comparison of RoBIK prototype with scanning device

The RoBIK system was associated in patients with a setup time of 11.0 min (8.5–15.5) against only 3.0 min (2.0–4.2) for the scanning device, a difference that was found statistically significant (Kruskal-Wallis, H(1) = 11.48, p = .001).

The performance of the RoBIK system was significantly lower as compared to the performance of the spelling device in the group of patients (the only group having evaluated both techniques). The accuracy was 0.8 (0.5–0.8) and 1.0 (1.0–1.0) for the RoBIK system and the scanning device, respectively, which was found statically significant (Kruskal-Wallis, H(1) = 4.90, p = .027). Likewise, the device speed for the RoBIK system was 0.5 characters per minutes (0.5–0.6) against 4.5 characters per minute (3.5–5.0) for the scanning device (Student t-test, t(4) = −14.78, p < .001). Naturally these translated into the significant superiority of the scanning system over the RoBIK system, with bit-rates of 28.3 bpm (19.0–31.1) and 1.7 bpm (1.1–2.1), respectively (Student t-test, t(4) = −12.10, p < .001).

In terms of self-reported fatigue in the patient group, it was found significantly greater while using the RoBIK system, 2.7 (2.6–3.0), as compared to using the scanning device, 2.0 (1.1–2.9) (F(1,29) = 49.83, p < .001). Interestingly, the repeated-measures ANOVA did not identify the time factor (measured before and after intervention) as significant (F(1,29) = 1.96, p = .172), instead suggesting a strongly significant time-technique interaction effect (F(1,29) = 16.33, p < .001), indicating that the BCI system was associated with greater fatigability. These results, however, did not translate into a decreased self-reported comfort, which was found equally good for the two techniques: 4.3 (2.2–6.8) for BCI and 5.0 (4.3–6.2) for the scanning device (Student t-test; t(7) = −0.60, p = .565).

Discussion

Study 1: clinical feasibility

Feasibility studies and evaluation of performance of P300 speller systems so far have concentrated on relatively small samples (n < 10) of late-stage ALS or acquired brain injury patients using the system at home or in a controlled environment [16,27,36,55,56]. A comparison of performance between normal and severely disabled participants showed significantly better performance in the healthy population [29], which is a strong rationale for the evaluation of this technology with a larger variety of patients and preferably in a natural environment. A recent study on a bigger cohort (n = 27) of ALS patients at home mentions mechanical ventilation, but it is unclear if patients where using the ventilators during the sessions [30]. The Study 1 aimed at the investigation of the use of BCI by acute patients in their real environment of use (here, the ICU).

As reported in Table 2, we found no statistically significant mean difference of performance between mechanically ventilated patients (in the ICU, n = 7) and non-ventilated patients (in the rehabilitation unit, n = 5) over the different sessions: training (p = .46) and test (p = .62). These results have to be handled with care, as the study was certainly not designed to study such an effect. However, the individual performance reported in this study show that neither mechanical ventilation, nor the origin of tetraplegia, nor the use of CNS depressants may be reliably related to failure at controlling the BCI, which certainly is encouraging.

For three patients who could not achieve adequate performance during the test session, several possible explanations were identified. A technical problem on the acquisition module of the software degraded the system performance for patients 2 and 3 (which was subsequently fixed on the open-source platform). This illustrates a well-known limitation of existing BCI systems (usability of hardware and software components) when it comes to their transfer to the patients’ bedside [57]; we believe that the joint efforts of the scientific and industrial communities is gaining momentum to address these limitations. Patient 9 reported difficulties in simultaneously swallowing and concentrating on the task, finally achieving low performance. Patient 10 fell asleep during the training session despite initially showing interest and motivation for the protocol; this was possibly explained by painful GBS accompanied by sleep deprivation. There was no obvious reason for the low performance found in patient 11 apart from a light visual impairment. This patient may be a ‘BCI illiterate’ [58].

To the best of our knowledge, this is the first comprehensive attempt to explore the use of a brain-computer interface in an adverse clinical context. Four key factors were explored: first, a non-controlled clinical environment and the particularly adverse setup of the ICU (58%); second, all ICU patients were evaluated during invasive mechanical ventilation; third, patients with myopathy or Guillain-Barré syndrome (42%) are populations that have not yet been reported to use BCI; finally, this study provides an initial insight into the impact of central nervous system (CNS) depressants on concomitant use of a P300 ERP-based BCI for communication. While this study was clearly not dimensioned to fully explore this phenomenon, we welcome the successful use of the BCI by some patients under high levels of CNS depressant, which could have primarily been thought to be prohibitive. Hence, further research is required to exactly understand the extent to which this kind of medication influences the performance of a BCI.

Study 2: clinical evaluation

The comparison of the BCI performance of healthy volunteers and patients showed no statistical difference in the primary performance outcome (bit-rate). Our finding adds to the small and contrasted available literature [57,59]. However, the progression of bit-rate over time within the same day, in particular for patients, was found significant and corroborated the evolution of self-reported fatigue, which increased in both groups, but was stronger in patients. The inclusion of fatigue and time in the analysis might in part explain the discrepancies within the literature. We have included self-reported fatigue in this study because we knew that the typical population of quadriplegic patients admitted to the ICU is prone to fatigue and because the outcome of Study 1 indicated that fatigue was a recurrent complainr about the system. As a consequence, we hypothesized it could constitute an important limitation for the use of BCIs in this population. This was despite the fact that some elements of the RoBIK prototype were specifically designed to reduce fatigue: for instance, visual fatigue was limited by the use variable ISI, which is illustrated by Figure 5. Since the patient population was significantly older it is not possible to disentangle the effect of age from that of fatigue. This certainly constitutes a significant limitation of our study, even though one can argue that the direction of this effect is nonetheless unfavorable to the patient population since fatigability is expected to increase with age. Such a hypothesis is, however, in contrast with evidence of a positive correlation between age and performance in a P300 speller, which was reported in healthy volunteers as well as in patients [59,60]. The retrospective analysis of existing cohorts could help investigate this correlation further.

Figure 5. Example of two event-related potentials (ERPs) elicited from the interface used in the pilot study (left) and the RoBIK prototype (right); the RoBIK prototype does not show the typical steady-state visual evoked potential (SSVEP) in response to non-target stimulations that can be seen on the non-target response (left).

The comparison of online performance for the use of BCI and the scanning system in the patient population showed that the BCI did not compare advantageously to the traditional assistive technology. All measured indicators (with the exception of self-reported comfort and satisfaction – non-significant) were found to be in favor of the scanning system: setup time was longer for the BCI; accuracy, speed, and bit-rate were all found better for the scanning device. Again, fatigue was strongly associated with the use of the BCI system, since both the technique and the time × technique interaction factors were significant.

Offline analysis of clinical data

The analysis of the EEG data collected during the two clinical trials described in this paper is reported in Appendix 3. The results we obtained stress the added value of the Riemannian approach over state-of-the art classification techniques, which was found particularly relevant for the development of calibration-free BCIs (also referred to as ‘cross-learning’). Conversely, the analysis did not identify the presence of a Riemannian SQI (artifact rejection) as a factor contributing positively to the BCI performance, which was surprising. On the hardware side, the offline analysis revealed that the gain in setup time offered by the hardware prototype developed was not associated with a noticeable drop in system performance.

Put together these innovative steps in EEG data collection and analysis have the potential to dramatically reduce the setup and calibration time. This would limit the accumulation of fatigue prior to the actual use of the communication interface and thereby improve overall system performance, which was shown to be greatly affected by fatigue. These results are encouraging for the future use of P300 spellers for the communication of patients. Apart from patients for whom a reliable neural interface could not be found (so called ‘BCI illiterates’), patients’ performance appeared not to differ significantly from that of healthy participants. Most importantly, adequately chosen parameters for the BCI interface lead to an estimated performance that was not found to differ significantly from that of a traditional assistive technology device (t = 0.56, permuted p-value = .33). Again, these results should be interpreted with extreme care because this study was not statistically powered to answer these specific questions and because we excluded three of ten patients (from Study 2) who could not use the BCI system.

Conclusions

Some neurological disorders leave patients with baseline cognitive status and no or little communication capacity. LIS patients – whatever their etiology – are a well-known illustration of such a condition. Quadriplegic patients with invasive mechanical ventilation are also temporarily deprived of communication, resulting in additional stress for all people involved with the management of their disease. Traditional assistive technologies for chronic patients, such as those suffering from neurodegenerative disorders, have long been used and most severe cases are equipped, whenever possible, with a ‘scanning’ spelling device interfaced with simple ‘click’ contactors. However, the use of these systems requires the intervention of an experienced occupational therapist who must find the optimal set of parameters for each patient. BCIs, on the other hand, have long been promised as a potential ‘universal’ assistive device for chronic patients such as those with ALS. So far BCI technology has consistently shown major limitations such as low information transfer rate (low spelling rate and accuracy) and the need for cumbersome and expensive setups for the recording of the EEG and its processing. However, recent advances in EEG hardware and software appear to open a new era where these limitations will be overcome.

A feasibility study was carried out in the ICU and in a rehabilitation unit in a population of quadriplegic patients using a traditional BCI system for communication (the ‘P300 speller’). Twelve patients were included in the study. Among them, eight could successfully control the system with above-chance accuracy. Performance could not directly be related to the presence of either mechanical ventilation or sedation. In contrast, eyesight and fatigue were identified as possible limitation factors.

 Based on this pilot phase, technical specifications and user requirements were drafted for the development of adequate bedside BCI prototypes for communication. The hardware was a 14-channel EEG system that could be set up in approximately 10 min. The software was developed with a user-friendly interface and state-of-the art processing/classification techniques (based on Riemannian geometry) in the back-end.

The prototype’s performance was evaluated in a clinical trial. Its performance was compared to the performance of traditional assistive technology (a scanning spelling device) in patients. The comparison was carried out using bit-rate, accuracy, setup time, comfort, and fatigue, confirming the superiority of the traditional AT technique in terms of simplicity of use (setup time), cost, performance, and – most importantly – patient fatigue. The BCI was also evaluated in a group of healthy volunteers to benchmark performance, showing that, if both groups performed equally, a stronger decrease in performance over time was observed in patients. This further strengthens the importance of fatigue in this population.

Finally, three offline analyses were carried out using the data collected during phases 1 and 3:

•	First, the ‘personalized’ estimated performance of the BCI rate was derived from the training data, pseudo-prospectively applied to the test set (online session) and the results were compared to those of the scanning device. The analysis confirmed that the bit-rate is greatly improved when an optimal and patient-specific number of repetitions is chosen. The resulting performance was found comparable to that of the scanning spelling device.
•	Second, the individual added value of each element of the design was quantified in an offline analysis. It showed that the gain in ease of use generated by the new headset came at no cost in performance. It also confirmed the superiority of Riemannian methods over state-of-the-art techniques for ERP detection. However, the rejection of artifactual segments of the data did not increase the performance above the threshold of statistical significance.
•	Last but not least, we investigated possible model initialization using an existing database. The rationale was to reduce fatigue by removing the tiring calibration session. The analysis revealed that the Riemannian methods enjoy superior cross-subject generalization, providing a good initialization that needs to be adapted online [44].

To conclude, this study demonstrates that caregivers who are unfamiliar with EEG and BCIs in general can restore some form of communication in severely disabled patients by means of an adequately designed BCI system. However, the population of patients who could immediately benefit from it was found to be much smaller than initially expected, since traditional assistive techniques are remarkably effective and compare very favorably to the use of P300-based spellers. Today, this is probably – more than ever – due to the lack of affordable, convenient, and reliable EEG systems, while other technological limitations are finally being overcome. Algorithms, for instance, are probably reaching a form of maturity and little further improvement can be foreseen for the coming few years; in our opinion, adaptive techniques based on cross-subject and cross-session initialization will play a dominant role. Fortunately, the ongoing transition of the EEG field from cottage industry to a more financially structured sector will soon translate into better and cheaper EEG systems. In turn, BCIs will cover the needs of an increasing proportion of severely disabled patients and ultimately find legitimate room next to other assistive technologies on the occupational therapist’s shelf.

Funding

We would like to thank the National Research Agency (ANR) and the General Directorate for Armament (DGA) for funding this project (Project ANR- 09-TECS-013–01-RoBIK). We would like to thank the French Association for Myopathies (AFM) for partly funding this project.

Acknowledgements

We would like to thank the medical staff who have been involved on this project, and in particular Marjorie Figère, Marjorie Dezeaux, and Sandra Potier from the CICIT, Jean-Marie, Gilles, Prof. Lofaso, and Prof. Herault from the Functional Exploration Unit, Justine Bouteille from the New Technologies Platform (PFNT), as well as many others involved in the project.

Notes

1. https://physionet.org/works/P300SpellerBCIICU/.

2. https://physionet.org/works/P300SpellerBCIICU/.

3. https://github.com/alexandrebarachant/covariancetoolbox/commit/bcccb4d750b2ad9ae6dc3dd76e7e6633f98bc1f3.

Appendix 1.

Details of hardware developments

The EEG headset was designed to minimize the need for EEG recording preparation: as a primary goal in the design phase, skin preparation with abrasive paste, electrode positioning with gel, and retention with tape had to be avoided. The resulting headset is a 3D-printed polyamide structure holding 14 Ag-Cl electrodes (Neuroservices, Evry, France) located at standard 10/20 locations: Po7, Fp1, Oz, Fz, C3, F4, Pz, C4, P7, Fp2, F3, P8, Po8, and Cz. These locations were selected in order to maximize the chance of capturing P300 ERPs using as reference and ground the right and left mastoids, respectively, according to previous studies by the RoBIK consortium [61]. Each electrode was mounted on a polyethurane wheel connected to a spring in order to control the pressure applied to the scalp with the electrode. The headset was designed to meet the need of patients and safety requirements, as well as to be compliant with the ICU environment. In particular, the headset could be used in a lying position or in the presence of a headrest.

The headset contains an electronic component that was designed to record EEG activity with high fidelity while maintaining minimal volume and weight. An MSP430 ultra-low-power microcontroller (Texas Instrument, Dallas, USA) was used to control a dedicated application-specific integrated component (ASIC) developed for this purpose: the CIrcuit for NEuronal SIgnal Conversion (CINESIC32). Each input channel is combined with an external capacitor (1.5 nF) in order to suppress the risk of leaking current in a first default condition, which is essential for medical applications. The analogue channel is composed of a fully differential low-noise amplifier, followed by a voltage gain amplifier and a programmable low-pass filter. Each channel consumes about 34 μA, summing up to an averaged consumption of 13 mA at full data streaming including baseline consumption at 3.3 V. With these settings 24-hour continuous operation can be achieved with one high-energy-density 3.6 V lithium battery. Only 14 of the 32 available channels were used in the final prototype and each of them was configured with a (0.5–30 Hz) band-pass filter and a 60 dB voltage gain; analogue signals were digitized through a 12-bit analogue-to-digital converter (ADC) with nominal sampling frequency of 580 Hz per channel.

Raw EEG were transmitted to a field-trip buffer [62] that was subsequently read by an OpenViBE acquisition server [42]. Signals were then band-pass-filtered between 1 and 20 Hz with a fourth-order Butterworth with linear phase response and decimated to 145 Hz for further analysis.

Appendix 2.

Details of software development

Graphical user interface

The Brainmium speller interface was designed to automate the calibration of machine learning algorithms implemented in OpenViBE and to provide a user-friendly interface for patients and operators. Rather than leaving the application synchronizing the received EEG data with the precise temporal occurrence of visual stimulations, which would be prone to delays and jitter induced by the operating system (Windows 7 professional edition, Microsoft Corporation, Redmond, WA, USA), we decided to send these stimulations through the USB directly to the EEG electronic acquisition unit to obtain accurate timestamps for all stimulation events. Flashes had a duration of 75 ms [55] and random inter-stimulus intervals (ISI) were drawn from an exponential distribution with mean . In previous research by our consortium, an exponential ISI was found to reduce the cortical fatigue associated with stimulation at fixed frequency while enhancing the ERP single-trial estimation [63]. In the original P300 speller paradigm symbols flash by rows and columns. Often detection errors arise because of the ‘adjacency-distraction’ phenomenon [64,65], according to which non-target symbols in rows or columns adjacent to the target attract the user’s attention when they flash, producing evoked activity similar to the P300, making the detection of the target P300 more difficult. To mitigate this effect we flash the symbols by random groups [63]. Not only is the ‘adjacency-distraction’ effect mitigated, we also find that the pattern of flashing becomes totally unpredictable, which is expected to sustain the attention of the user and to enhance the P300. More importantly, random-group flashing allows arbitrary positioning of the symbols on the screen (no more need to arrange symbols on a grid), which greatly expands the usability of the P300 paradigm.

Online artifact rejection

The online artifact detection was initialized at the beginning of each session in order to interrupt the sequence of stimulations in the presence of excessive noise. This technique allows a simple definition of a rejection region for incoming data segments. The participant was instructed to stay still for 10 seconds during which a series of n_sqi clean overlapping EEG time-windows of dimension E × S were extracted, with S the number of samples in the time-window and E = 14 the number of electrodes. For each time-window , a covariance matrix of dimension E × E was computed. Covariance matrices belong to the Riemannian manifold of symmetric positive-definite matrices wherein a Riemannian metric can be used to define a distance δ between any two covariance matrices such as [50]:(1)

where are the E eigenvalues of or of . Using this distance, a geometric center of mass (or Fréchet mean) of a set of covariance matrices can be estimated with a gradient descend algorithm solving the following optimization problem:(2)

In other words, just as the mean in Euclidean space is the value minimizing the variance, the Riemannian center of mass M of a set of points is the point minimizing their dispersion (variance) in the manifold. For every covariance matrix , we can compute a distance δ_i to the center of mass of the data set as . Finally, given all distances we can compute a scalar geometric mean μ and and a scalar geometric standard deviation σ of them using the following formula [4(4) 9]:(3)

which for Riemannian distances will approximate a symmetric distribution [49]. Finally, we can then derive z-scores of the distances as(4)

Once μ and σ are estimated on the training data, covariance matrices computed on new online EEG epochs are discarded whenever their distance to the center of mass exceeds a z-score of 2.5. The set of points on the manifold having a distance to a center of mass less than 2.5 standard deviations forms a closed region; in three dimensions, such a region would look like a potato rather than a sphere, because of the non-linear nature of the Riemannian manifold, which is why this technique was named the ‘Riemannian Potato’ [48]. During online experiments, visual feedback was provided to patients so that they could relate artifacts to specific behaviors (blinks, coughing, head movements). This ensures that they can associate any artefactual behavior with its impact on the data quality and take corrective actions: move less and blink at appropriate times as much as possible. The Riemannian potato was implemented in OpenViBE for online experiments (Study 2) and in Matlab³ for offline experiments (Study 1).

Online classifier

Using the Riemannian distance in Equation (1) and the definition of center of mass, in Equation (2) the detection of the P300 evoked potential can be achieved by a deceptively simple classification algorithm named the minimum distance to means (MDM) [50]. Denoting by + the target class of flashes, each trial is concatenated with a prototypical P300 evoked response (for instance, the ensemble average estimation obtained on the training data) to build a ‘super’ trial:(5)

A special covariance matrix is then estimated using this super trial such as(6)

If the data have been previously band-pass filtered in an appropriate band-pass region, such a super covariance matrix contains all the spatial and temporal information needed to achieve the detection of ERPs,[70] therefore it can be used directly as a single feature for the classification algorithm as a point on the Riemannian manifold. For each of the two classes, Target (+) and Non-Target (−), a Riemannian center of mass is estimated using the data from the calibration session. The center of mass can be understood simply as the expected covariance matrix of a trial belonging to the corresponding class, wherein the use of the Riemannian metric ensures that this expectation is a much better representative as compared to the arithmetic mean. In particular, extensive testing presented in [70] has established that this expectation is more robust to noise and outliers. The classification of an unseen trial is achieved by comparing the distance of the trial to the center of mass of the Target and Non-Target class. A classification score is given to unseen trials according to the score function(7)

where are probabilities found by fitting a logistic regression curve with parameter α and β [66] to the set of distances and and and are the center of mass of the Target and Non-Target classes, respectively. These scores are averaged across repetitions, and the symbol is assigned to the symbol at the intersection of the row and column with highest score.

Appendix 3.

Offline analysis

This offline analysis has the following objectives:

•	Experiment 1: comparison of the personalized BCI with the traditional assistive technology;
•	Experiment 2: assessment of the individual contribution of each component (RoBIK headset, artifact-detection algorithm, classifier algorithm) to the overall system performance;
•	Experiment 3: assessment of the performance of a calibration-free BCI system.

Materials and methods

Elements of the materials and methods are summarized in Table 4.

Table 4. Summary of materials and methods for both studies showing the population, hardware, software, algorithms, and cross-validation procedures used in each phase. Abbreviations: minimum distance to mean (MDM) is a Riemannian classifier; support vector machine (SVM) is a classifier; stepwise linear discriminant analysis (swLDA) is a linear classifier; xDAWN is a spatial filter for event-related potentials; P1, P2, and V2 refer to patients and volunteers included in clinical studies 1 and 2, respectively.

	Population	Hardware	Software	Algorithms	Cross-validation
Exp. 1	• Patients (P2) • Volunteers (V2)	• RoBIK	• OpenViBE • Matlab 14	• Riemannian Potato • MDM	• 5-Fold (Training) • Cross-session
Exp. 2–3	• Patients (P1) • Patients (P2) • Volunteers (V2)	• TMSi • RoBIK	• OpenViBE • Matlab 14	• Riemannian Potato • MDM • xDAWN+swLDA	• 5-Fold (Training) • Cross-session • Cross-participant

Offline analysis 1: optimal BCI compared to scanning device

The comparison of performance for communication interfaces is usually assessed using the bit-rate [67,68]. The advantage of bit-rate over accuracy as a performance measure is that it also takes into account the speed of the interface and the amount of choices it offers. Bit-rates thereby gives a more representative metric of ‘information flow’ from the participant’s brain to the machine. The bit-rate br is defined as(1)

where M is the number of symbols and p the spelling accuracy (the percentage of correctly identified symbols). In this study, M equals 36 and 73 for the RoBIK BCI and the scanning device, respectively.

The evaluation of the RoBIK prototype was designed to demonstrate the feasibility of a well-designed BCI prototype for communication in a clinical context. Therefore, the stimulation parameters (flash duration, average time between two stimulations, and number of repetitions) were chosen to promote optimal accuracy rather than speed. The system evaluated at the bedside in this study was not optimized for bit-rate, but for optimal data collection and robustness in the context of a clinical feasibility investigation. This approach ensured the collection of a large amount of data for the offline analysis. For the BCI performance, we consider the bit-rate achieved with the optimal number of repetitions. In order to assess the optimal number of repetitions, the MDM classifier described above was fitted to the training data after artifact identification using the aforementioned SQI index. The test set was a bootstrapped dataset generated from the ‘online’ session of each participant as follows: for each number of repetitions of the stimulation k = 1…50 (that is the number of times a specific letter is flashed before a decision is made), B = 1000 groups of k target and five times k (5000) non-target responses were randomly selected (with replacement) and averaged, so that the target to non-target ratio was preserved. Then, for each group of six responses, the predictions were obtained by assigning 1 to the element featuring the minimum distance to the center of mass of the Target class and 0 to the five other elements. The seed for the random sampling was preserved in order to evaluate the performance of the different techniques on the same random bootstraps. At the end of this procedure, 6000 predictions were used to derive a performance index for each participant and for each number of repetitions of the visual stimulation (the flash). In particular, the ‘real accuracy’ was defined as the square of the raw accuracy, since each letter is found at the intersection between two groups of symbols (commonly referred as the ‘lines’ and ‘columns’ in the traditional P300 speller experiments). If predictions were to be made at random, the ‘real accuracy’ would be 1 in 36, meaning that there is a probability of 2.8% that the correct letter is selected by chance. With such a definition it was arbitrarily decided that an interface should provide a minimum ‘real accuracy’ of 70%, meaning that on the average 7 out of 10 characters should correctly be identified. The optimal number of repetitions was chosen accordingly, corresponding bit-rates were derived as described in Equation 1(1) , and subsequently compared to that of the scanning device for the same patients.

Offline analysis 2: added value of the prototype design

A second offline analysis was run to quantify the added value of the proposed BCI prototype design. More precisely, we used an offline study to ensure that the hardware design, which was meant to increase ease of use, came at no cost in performance and we similarly quantified the added value of the Riemannian approach for artifact trial rejection and classification. To do so, offline analysis 1 was repeated with and without online artifact trial rejection using the SQI index described in Appendix 2. Likewise, the performance of the Riemannian MDM classifier was compared to a state-of-the-art technique composed of the xDAWN spatial filter combined with a stepwise linear discriminant analysis (SWLDA) [69]. The comparison was carried out with and without the artifact trial rejection and on both clinical datasets (Studies 1 and 2). Because the RoBIK interface natively interrupts visual stimulations in the presence of noise, it is expected that most target and non-target epochs from Study 2 are clean. For this reason, only data from Study 1 were considered to assess the benefit of artifact rejection, while both datasets were used to compare the classifiers.

Offline analysis 3: cross-participant-analysis

A calibration-free ERP-based BCI has been proposed by Congedo [70] and Barachant et al. [48]; the center of mass of the available classes is initialized using a database of previous users and then continuously updated using the incoming data from the online sessions of the user. In order to assess the potential of such a approach, offline analysis 2 was run a second time replacing the training set by all data available from other participants (cross-subject learning). While offline analyses 1 and 2 used little data from the same participant, more data are available in this offline analysis; however, they belong to different participants. To allow for cross-subject comparison, all covariance matrices for the MDM models were normalized so as to have a unit determinant as:(2)

which derives directly from the following property of the determinant: , where denotes the determinant of matrix , c is a scalar, and E the size of the matrix.

Statistical analysis

The benefit of the proposed BCI design was evaluated by means of a Kruskal-Wallis test, using the technique methods as an independent factor and the bit-rate as the dependent variable. Using the best-performing of each method, optimal bit-rate was identified (at 70% real accuracy) and compared to that of the scanning system on the same patients using a paired Student t-test. For the repeated-measures framework above, in this manuscript we will refer to group factor, time factor, and time-group interaction factor and related effects.

Results

Added value of signal processing and classification algorithms

Figure 6 shows the influence of several factors with respect to real spelling accuracy: the presence of an artifact-removal technique (the SQI) and the use of a traditional algorithm (xDAWN+SWLDA) versus a Riemannian algorithm (MDM). It also compares the performance of healthy volunteers and patients and benchmarks the performance of the RoBIK EEG headset versus a traditional disc electrode system (TMSi amplifier). As detailed in Figure 6, these comparisons were carried out at different levels: with a five-fold cross-validation on the training set, prospectively on the online dataset, and using cross-participant fitting. For the latter, only data from other participants wee used to fit the model. The comparison of optimal (maximum) bit-rates over all repetitions for these conditions reveals a significant benefit of the MDM over the xDAWN+SWLDA approach (paired t-test, t = 32.4, permuted p < .001). This was particularly true for the cross-participant condition where only the MDM model could generalize well (paired t-test, t = 42.5, permuted p < .001). Other factors were not found significant. This seems to indicate that patients and healthy volunteers have equivalent performance. Likewise, the performance of the RoBIK headset did not significantly differ from that of the traditional EEG system, which strengthened the rationale for the improvement in setup time (from 30 minutes to 10 minutes). Finally, these results also indicated that the use of an artifact-rejection technique does not necessarily improve the accuracy, although it results in a reduction of variability, which can be noticed only with the use of a traditional EEG recording system.

Figure 6. Evolution of ‘real accuracy’ (% of correctly identified characters) in relation to number of flashes of each symbol under different conditions; the three rows of plots indicate the performance in (1) cross-validated training data, (2) test-set (online) data, and (3) cross-subject performance. The first column of plots shows the effect of artifact removal using the SQI on the data from Study 1 only. The second column shows the impact of the classifier comparing xDAWN + stepwise linear discriminant analysis (SWLDA) and minimum distance to mean (MDM) Riemannian classifier. The third column compares the performance of healthy volunteers and patients. The fourth column compares the data collected with a traditional EEG system (Study 1) to that collected with the RoBIK headset (Study 2). For all factors considered, data are collapsed across the other factors.

Comparison of optimal BCI to scanning device

As seen in Table 5, seven out of nine patients (77%) who completed the Online session during Study 2 could successfully use the P300 speller. For those (we excluded two additional patients P05 and P09 who could not use the system at all), the comparison of bit-rates did not show statistical significance between the first scanning session and the personalized RoBIK performance (T = 0.56, permuted p-value = .33).

Table 5. Bit-rates (bits per minutes) for the clinical sample included in the clinical study of Study 2 for scanning device (session 1 only), the RoBIK system during the online session, and the performance computed offline using the same data and algorithms but with the use of an optimal number of repetitions (personalized). Patients 5, 8, and 9, for whom no BCI signal could be exploited, were removed from this analysis and subsequent statistical testing.

	P01	P02	P03	P04	P05	P06	P07	P08	P09	P10
Scanning 1	24.5	22.6	30.8	20.3	18.9	33.9	20.7	NA	25.1	14.5
Online RoBIK	3.1	1.7	1.7	0.2	0.0	1.4	0.1	NA	0.0	0.1
Personalized RoBIK	27.8	27.8	23.0	27.8	0.0	1.7	2.9	NA	0.0	31.5

Discussion

First, the results confirm the superiority of the Riemannian MDM approach over traditional classification based on spatial filtering (for Online mode, line 2 on Figure 6). This finding replicates previous reports on healthy participants [49,50,70]. In some patients, the Riemannian MDM offers real accuracy above 80% with as little as two repetitions, which favorably compares with state-of-the-art techniques [71]. This superiority was even more evident for the cross-subject transfer learning (Cross-subject, line 3 on Figure 6), which is also in line with our previous investigations [50,70]. Overall these results suggest that the Riemannian MDM classifier offers better generalization properties as compared to spatial filters. Interestingly, the advantages of the MDM algorithm in terms of accuracy and generalization are accompanied by a dramatic reduction of the algorithmic complexity; the Riemannian MDM requires only the computation of centers of mass and distances between two points and has no free parameters to be tuned by cross-validation or heuristics, therefore it reduces the risk of over-fitting. In addition to this, Riemannian methods generalize well across sessions (see Figure 6, line 2 column 2) because the Riemannian distance function is invariant by any linear transformation of the data and covariance shifts observed across sessions due to movement in headset positions or changes in electrode impedance are of this type. As expected, the performance of the cross-participant model was found to be significantly lower than that of the ‘online’ mode (trained on data from a training session from the same participant), which implies that under these conditions online adaptation may be required to keep the performance of the calibration-free BCI optimal.

The use of the SQI to reject trials contaminated by artifacts did not show clear benefit in terms of bit-rate. However, Figure 5 (column 1) shows that monitoring artifacts and stopping the operation in the presence of high noise reduce the variability of the system, which is an important benefit for real-word operation. It should be noted that our study has been carried out in a realistic environment. Even though the data were acquired under the strict supervision of an EEG technician, they were collected in an environment prone to artifacts. We have made the data de-identified, open, and accessible to peers in the digital supplement of this article so that the community of researchers can investigate this and other questions further.

No difference in performance was found between the RoBIK and the TMSi headset, even though lower variability can be observed for the TMSi headset. This supports the well-known fact that a gel-based cap collects EEG data more consistently. Nonetheless, the RoBIK headset has lower technical specifications (no true DC, 12 bits instead of 24), is therefore cheaper, and allows a dramatic reduction in setup time (approximately three-fold). While the headset prototype tested had numerous drawbacks (design, weight, corrosion with saline water, mechanical electrical contact to name a few) it shows that cheap and easy-to-use EEG systems for bedside applications are becoming a reality. The trending developments in dry electrode systems [18,20,71] may be expected to lower the tolerance threshold further and broaden the use of BCIs.