Closed loop BCI system for Cybathlon 2020

ABSTRACT

We developed a Brain-Computer Interface (BCI) System for the BCI discipline of Cybathlon 2020 competition, where participants with tetraplegia (pilots) control a computer game with mental commands. To extract features from one-second-long electroencephalographic (EEG) signals, we calculated the absolute of the Fast-Fourier Transformation amplitude (FFTabs) and introduced two methods: Feature Average and Feature Range. The former calculates the average of the FFTabs for a specific frequency band, while the later generates multiple Feature Averages for non-overlapping 2 Hz wide frequency bins. The resulting features were fed to a Support Vector Machine classifier and tested on the PhysioNet database and our dataset containing 16 offline experiments recorded with the help of 2 pilots. 27 gameplay trials (out of 59) with our pilots reached the 240-second qualification time limit, which demonstrates the usability of our system in real-time circumstances. We critically compared the Feature Average of canonical frequency bands (alpha, beta, gamma, and theta) with our suggested range30 and range40 methods. On the PhysioNet dataset, the range40 method combined with an ensemble SVM classifier significantly reached the highest accuracy level (0.4607), with a 4-class classification; moreover, it outperformed the state-of-the-art EEGNet.

KEYWORDS:

• — BCI
• EEG
• SVM
• FFT
• Normalization

1. Introduction

Cybathlon Competition was first introduced on October 5^th, 2014 [1]. This event, also known as the ‘Bionic Olympics’, provides a platform for research groups, industrial companies, and technology providers to showcase their products, applications, and technologies across six disciplines with the assistance of physically disabled participants, referred to as pilots in Cybathlon terminology. The six disciplines include Brain-Computer Interfaces (BCI), Functional Electrical Stimulation (FES) Bike Race, Leg Prosthesis, Powered Arm Prosthesis, Powered Exoskeleton, and Powered Wheelchair.

Brain-Computer Interfaces are integrated systems comprised of both software and hardware components. According to Wolpaw et al. [2] these systems capture bioelectrical signals from the brain and translate them into computer commands. In the BCI discipline of Cybathlon, pilots with quadriplegia compete in a car-racing-like computer game by controlling their avatar using well-timed imagined mental commands recorded by Electroencephalography (EEG). The raw EEG data recorded is often subject to internal or external noise interference such as eye blinking, swallowing, electric powerline noise or motion artifacts. The use of any artifact for control is strictly prohibited and the implementation of a filtering and artifact rejection algorithm is mandatory. The computer game can be controlled using three active commands plus the absence of any commands. Pilots are required to reach the finish line within 240 seconds.

In the FES discipline, pilots with paraplegia compete in a tricycle race where muscle activity is generated through functional electrical stimulation (FES). The Leg and Arm Prosthesis disciplines involve completing obstacle courses designed to demonstrate the capabilities and usability of the prosthesis in various real-life situations. The Powered Exoskeleton assists paraplegic pilots in standing up, walking and climbing stairs while the Powered Wheelchair discipline challenges pilots to navigate through stairs, different types of roads and crowds. This article focuses on the BCI discipline.

Perdikis et al. [3] participated in the first Cybathlon competition with two pilots forming a team called Brain Tweakers. They utilized electrooculographic (EOG) electrodes and the FORCe algorithm to detect artifacts. Laplacian derivation was performed on pure EEG signals as a special filtering technique, followed by Power Spectral Density calculation on 2 Hz wide frequency intervals. The resulting features were classified using the Gaussian mixture model. Two class motor imaginary signals were used, namely movements of Both Hands and Both Feet. To meet the requirements for controlling the game, they implemented a strategy where if two different types of commands were generated within a configurable time window, the third active control signal was sent to the game. Instead of further developing and fine-tuning the control algorithm, they focused on training their pilots, asserting that learning to purposefully modulate brain waves significantly impacts the usability of the BCI system. As evidence of their hypothesis, one of their pilots won the Cybathlon 2016 competition and they successfully employed their algorithm in both the Cybathlon BCI Series 2019 and Cybathlon Global Edition 2020 competitions [4] as WHI Team, achieving first place.

Team MIRAGE91 from Graz [5] developed an online artifact detection system that included a blinking detector by thresholding on the band power of the AFz electrode and autoregressive modeling to detect high deviations. For feature extraction, they used Common Spatial Pattern with shrinkage regularized Linear Discriminant Analysis for classification at Cybathlon 2016. They implemented a 3-class paradigm with a thresholding strategy using the following motor imagery (MI) tasks: Left Hand, Right Hand, and Both Feet. The control output was only sent to the game if the classification probability met the threshold. Their pilots demonstrated a smooth learning curve; however, they encountered unexpected issues during the competition and performed below their training results. For the Cybathlon BCI Series 2019 and Cybathlon Global Edition 2020 competitions, they improved their algorithm by incorporating a novel adaptive thresholding algorithm [6] for controlling the output of the BCI.

In parallel with the Cybathlon, numerous efforts have been focused on developing suitable offline BCI systems capable of accurately classifying EEG signals originating from MI tasks [7–10]. Many of these systems employ artificial neural networks as a classifier [9,11–13]. The EEGNet [8], developed by Lawhern et al., is one of the state-of-the-art networks. One advantage of this algorithm over simpler classification methods is that it does not require any feature-extracted signal; instead, it only requires raw EEG data in matrix form and learns features similar to Filter Bank Common Spatial Pattern [7]. However, simple classifier methods, such as K-Nearest-Neighbor [14,15], Linear Discriminant Analysis [14,16] or Support Vector Machine (SVM) [17–19] are also preferred in BCI systems, where computational requirements are planned as modest, as in our approach, presented in section 2.2 below. Therefore, we selected SVM for our algorithm and compared it with the state-of-the-art EEGNet. The majority of classifier comparison studies utilize one of the BCI Competition datasets [20–23]; however, these datasets contain only a limited number of subjects $(\le 10)$. In [13,15,24] the MI dataset on PhysioNet [25] is utilized. This database was created with the participation of 109 subjects; therefore, we selected it for comparison to ensure statistical significance.

Our study aims to present the comprehensive development trajectory of a Brain-Computer Interface (BCI) system, designed for the Cybathlon Global Edition 2020 by the Hungarian research team, Ebrainers. During the design we aimed to generate a subject-specific BCI pipeline instead of a general one, as it was reported in [8,26], to be superior in classification results. The development process commenced with the creation of signal processing and classification algorithms, which were subsequently validated offline using the PhysioNet dataset. Concurrently, we engaged our pilots in recording experiments. Following data evaluation and algorithm optimization, a real-time BCI system was implemented to interface with the video game provided by the Cybathlon 2020 organizers. Regular experimentation continued until March 5th, 2020, when the advent of the pandemic resulted in the cancellation of further experimentation and our participation in Cybathlon 2020.

2. Materials and Methods

To create a BCI system, which can be used to control a computer game, the first step is to acquire a reliable database, which include not even a large amount of EEG signals but also the correct event markers of the experimental tasks with appropriate labels as annotations. Accurate labeling is essential for appropriately testing the precision the later developed feature extraction and classification algorithms.

2.1. Physionet Database

The EEG Motor Movement/Imagery Dataset, accessible via PhysioNet (Physionet) [25], represents one of the biggest repositories of MI task-based data, acquired using the BCI2000 system [27]. The Physionet dataset contains EEG recordings from 109 subjects, obtained using a 64-channel 10–20 EEG system.

In summary, the experimental paradigm employed by Physionet entailed the following: At the onset of the experiment, subjects underwent two one-minute baseline sessions, during which they were instructed to remain calm with eyes open and subsequently closed. This was followed by a movement execution and imagination period, during which subjects were required to perform overt Left Hand and Right Hand movements, succeeded by an imaginary session involving covert movement. Subsequently, executed and imagined sessions involving Both Hands and Both Feet were conducted. These tasks were repeated three times sequentially, resulting in a total of 14 experimental sessions in addition to the two baseline sessions. Each executed and imagined movement lasted for 4 seconds and was followed by a 4-second resting period.

Due to minor data acquisition issues with the Physionet database, we elected to exclude subjects 88, 89, 92, and 100 for the following reasons: In the case of subject 89, we discovered that the labels were incorrect. The records of subjects 88, 92, and 100 deviated from the primary database description. The duration of the task execution and resting phases were altered to 5.125 and 1.375 seconds respectively. Furthermore, a sampling frequency of 128 Hz was employed instead of the original 160 Hz. These issues have been reported in [13,24].

2.2. Signal Processing and Classification

This section delineates the central component of the BCI system, encompassing artifact rejection, feature extraction, and classification methodologies. The comprehensive signal processing architecture is presented in Figure 1. The System’s source code, developed and implemented in Python, is accessible at: https://github.com/kolcs/bionic_apps and utilize state-of-the-art EEG signal processing and machine learning packages such as MNE [28] and TensorFlow [29].

Figure 1. BCI pipeline – The FASTER algorithm was employed to eliminate EOG and EMG artifacts from raw EEG signals. The absolute value of the Fast Fourier Transformation (FFTabs) was computed as a feature for each EEG channel using a 1-second window. With the feature Average method, the mean of the FFTabs was calculated for each channel within a specified frequency range, which could correspond to one of the canonical EEG bands (theta, alpha, beta, gamma). These features were utilized to train a Support Vector Machine (SVM). The feature range method builds upon and extends the feature Average method. The feature Average was computed for an odd number of distinct frequency bins, each 2 Hz wide. Each feature, corresponding to a frequency bin, was used to train a separate SVM, with the final classification result determined by taking the maximum vote of all SVM units.

2.2.1. Artifact Rejection Algorithm

The Fully Automated Statistical Thresholding algorithm (FASTER), as published by Nolan et al. [30], was employed for the purpose of artifact rejection, with minor modifications as detailed below. Our implementation of the algorithm in Python was derived from the work of Vliet [31]. The implemented algorithm comprises four steps, designed to eliminate channels, epochs, and components where the Z-score of specified parameters exceeds 3. In the first step, EEG channels exhibiting a Z-score exceeding the threshold for variance, correlation, or Hurst exponent are removed. Subsequently, epochs are discarded based on amplitude range, deviation from channel average, and variance parameters. The third step seeks to eliminate artifact-related components of the signal using independent component analysis (ICA), which separates time-dependent data into statistically independent waveforms. The algorithm produces a mixing matrix that transforms EEG data into Independent Components (ICs) through multiplication. This computation was performed using the FastICA implementation of the Scikit-learn package [32]. ICs are omitted if their correlation is high with designated electrodes closest to the eyes, thereby filtering out blinking artifacts. Additionally, ICs exhibiting a Z-score greater than 3 for kurtosis, power gradient, Hurst exponent, or median gradient are discarded. To transform the ICs back into the time domain, multiplication by the inverse of the mixing matrix is performed. In the fourth step, bad channels within individual epochs are identified based on variance, median gradient, amplitude range, and channel deviation parameters. All channels designated as bad, including those marked in previous steps, are interpolated using spherical spline interpolation. While the original FASTER algorithm comprises five steps, the final step – detecting artifacts across subjects – was omitted to create a subject-specific process.

Although originally designed for offline brain signal processing, our aim was to utilize the algorithm in real-time. Consequently, minor modifications were made to implement an online version of FASTER. The most time-consuming aspect of the original code is the generation of the ICA matrix; as this cannot be performed online, the matrix is computed using a prerecorded training dataset from the Online Paradigm. Globally bad channels are also determined using this training data. The second step of the FASTER algorithm is omitted in our implementation, as all incoming data must be processed during the real-time gameplay. In the third step, the precalculated ICA matrix is used for transformation and online filtering of components. The fourth step remains unchanged in our online version of the algorithm.

2.2.2. Feature Extraction

During feature extraction, epochs were generated from the artifact filtered EEG signals with reference to event markers. Each epoch commenced at the event marker and terminated 4 seconds later, corresponding to the duration of MI task execution by experimental subjects. A majority of EEG processing and classification methodologies reported in BCI competition datasets [20–23] generate 2-second-long windows from epochs [7–9,33–35]. However, a review of complete BCI systems employed in Cybathlon competitions reveals a preference for 1-second-long windows for real-time signal processing, feedback, and control mechanisms [3–5, 36–38] to avoid time lag emphasized in [39]. Therefore, shorter 1-second-long EEG windows were extracted from the generated epochs using a sliding window approach with a 0.1-second shift.

Subsequently, the absolute value of the complex Fast Fourier Transformation [40] (FFTabs) was computed for each EEG channel as a frequency domain feature. The FFTabs function was utilized in subsequent methods.

a) Feature Average: In our Feature Average method, the numerical average of FFTabs values was calculated in a specified frequency range for each EEG channel, as represented by the following equation:

(1) $\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}{\mathrm{e}}_{\mathrm{c}{\mathrm{h}}_{\mathrm{i}}}=\frac{1}{N}\sum _{f=f_{min}}^{f_{max}}\mathrm{F}\mathrm{F}\mathrm{T}\mathrm{a}\mathrm{b}{\mathrm{s}}_{c{h}_i}\left(f\right)$(1)

where $c{h}_i$ denotes the $i^{th}$ EEG channel and $N$ represents the number of FFTabs samples in the defined $\left[f_{min},f_{max}\right]$ frequency range.

This process can be interpreted as the truncation of FFTabs to the selected frequency interval and subsequent compression of this matrix into a channel number x 1 feature vector. The boundaries of the frequency range constitute parameters that can be selected to correspond to one of the canonical EEG bands: theta (4–7 Hz), alpha (7–14 Hz), beta (14–30 Hz), and gamma (30–40 Hz).

b) Feature Range: The Feature Range method builds upon and extends the Feature Average. Our objective was to augment the information content of the calculation relative to the Feature Average, analogous to the manner in which the Filter Bank Common Spatial Pattern [7] sought to improve upon the performance of the original Common Spatial Pattern algorithm. This method generated multiple Feature Averages for non-overlapping, 2 Hz wide frequency bins. This method has two parameters defining the lowest and highest frequency edges. The first frequency bin ranges from $f_{low}$ to $f_{low}+2Hz$ and the last from $f_{high}-2Hz$ and $f_{high}$. We created two feature sets from Feature Range, called range30 and range40, where the numbers correspond to $f_{high}$. In the case of range30 $f_{low}=4Hz$ and $f_{high}=30Hz$, which resulting in a total of 13 frequency bins. Consequently, the size of the feature matrix is channel number x 13. In the case of range40 $f_{low}=2Hz$ and $f_{high}=40Hz$, resulting in a total of 19 frequency bins.

2.2.3. Feature Normalization

After calculating the features, L2 normalization was performed to enhance classification outcomes, as reported in [41]. The L2 normalization is defined as follows:

(2) $X_{l2}=X/\sqrt{\sum _{k=1}^n{\left|x_k\right|}^2}$(2)

2.2.4. Support-Vector Machine based Classifiers

As a classification tool, we employed the Support Vector Machine (SVM) methodology, owing to its modest computational demands and frequent utilization in BCI applications [17–19,37].

The original SVM problem was formulated by Vapnik [42]. Given a training set $\left({\mathbf{x}}_i,y_i\right),i=1,\dots ,k$ where ${\mathbf{x}}_i\in R^n$ represents a training sample with label $y_i\in \{-1,1\}$, the SVM solves the following optimization problem:

(3) $\underset{\mathbf{w},b,\xi }{min}\frac{1}{2}{\mathbf{w}}^T\mathbf{w}+C\sum _{i=1}^k{\xi }_i$(3)

subject to

(4) $y_i\left({\mathbf{w}}^T\varphi \left({\mathbf{x}}_i\right)+b\right)\ge 1-{\xi }_i,{\xi }_i\ge 0$(4)

where $\varphi$ denotes a nonlinear function that can maps $x_i$ to a higher dimensional feature space and $C>0$ represents a penalty hyperparameter of error term. The term $K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)\equiv \varphi {\left({\mathbf{x}}_i\right)}^T\varphi \left({\mathbf{x}}_j\right)$is referred to as the kernel function and can be an arbitrary mathematical equation. The most used kernel functions are as follows:

• linear: $K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)={\mathbf{x}}_i^T{\mathbf{x}}_j$

• polynomial: $K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)={\left(\gamma {{\mathbf{x}}_i}^T{\mathbf{x}}_j+r\right)}^d,\gamma >0$

• Radial-basis-function (RBF):

$K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=exp\left(-\gamma \parallel {\mathbf{x}}_i-{\mathbf{x}}_j{\parallel }^2\right),\gamma >0$

where $\gamma$, $r$ and $d$ denote the kernel parameters that can also be considered hyperparameters.

To solve the SVM problem, we utilized the Scikit-learn package [32], which encompasses numerous efficient implementations and other useful machine learning tools. From the available SVM classifiers in Scikit-learn, we selected the SVC class, which defines an RBF-kerneled Support Vector Classifier. The default hyperparameters were employed in all experiments for all classifications. In the case of the Feature Average method, channel number x 1 size feature vectors were used to train and classify data.

In the case of the Feature Range method, feature vectors of frequency bins were used to train separate SVMs in parallel. Consequently, each SVM unit was trained on different EEG bands (e.g. 4–6 Hz and 6–8 Hz) and learned distinct characteristics of brain signals. Each SVM unit made its own classification decision and individual results were aggregated using the majority vote method to compute the final classification result. To avoid draws, an odd number of SVM units were selected. We refer to this classifier methodology as Voting SVM. A similar approach was presented in [43]; however, their algorithm was applied to Common Spatial Pattern features and utilized Bagging to generate random sub-datasets for each SVM unit. Additionally, they omitted the use of any artifact rejection algorithm.

2.2.5. Epoch based Cross-validation

N-fold cross-validation was employed to evaluate the validity of feature extraction algorithms in conjunction with SVM classifiers.

Initially, data was partitioned into N distinct subsets. In each iteration, N-1 subsets were designated as the training set and one subset as the testing set. New instances of classifiers were created, trained using the training set, and their classification performance was evaluated on the testing set. This process was repeated N times. The results of each iteration were recorded and the final accuracy was computed as the arithmetic mean of individual classification outcomes. We set N to 5 in all experiments.

Partitioning the data at the window level, in instances where windows overlap, rather than at the epoch level, may result in invalid accuracy and evoke the issue of overfitting. This is due to the fact that windows derived from the same epoch may be allocated to both the training and testing sets. This can constitute a significant error when dealing with highly overlapping windows, as it implies that nearly identical features are present in both sets. Consequently, we opted to partition the data at the epoch level rather than at the window level. This ensures that windows derived from a single epoch are used exclusively in either the training or testing set.

2.3. Our Pilots and Experimental Setup

For the real-time working BCI System, we required a participant with tetraplegia to serve as a pilot. To this end, we collaborated with MEREK, the Rehabilitation Centre for Physically Disabled People in Hungary.

2.3.1. Pilots

Two subjects were applied (B. and C., both male) having C5 or higher spinal cord lesions. The injury of each pilot was confirmed and classified by a neurologist concerning the International Standards for Neurological Classification of Spinal Cord Injury. This pre-competition Medical Check was mandated by the organizers for all Cybathlon participants.

Pilot B was 44 years old and had an incomplete C5 Neurological level of injury (NLI). His Asia Impairment Scale (AIS) was B. The additional comments of the neurologist were: ‘Dysesthesia in palms. No muscle function in the non-key muscles either (on neither scale)’.

Pilot C was 38 years old and had a complete C4 NLI, with AIS A, by the time of the experiments. The additional comments of the neurologist were: ‘Paresthesia in palms and foot. No muscle function in the non-key muscles either (on neither scale)’.

Both offline and online experiments were conducted with the pilots to record our dataset, test the BCI system on them, and enable them to control the online game.

2.3.2. Ethical Permit

This study was carried out following the Declaration of Helsinki and national guidelines, with written informed consent obtained from all participants. The study received approval from the United Ethical Review Committee for Research in Psychology (EPKEB reference number: 2018–54).

Prior to each experiment, participants were informed about the experimental procedures and provided written consent.

2.3.3. Data Acquisition

Electroencephalography (EEG) data were recorded using a 64-channel ActiChamp amplifier system (Brain Products GmbH, Gilching, Germany) and an actiCAP EEG cap in accordance with the international 10–20 system. The POz electrode was used as the reference, and data were acquired from 63 electrodes. During experimental preparation, the impedance of the EEG electrodes was measured and maintained below $30k\mathrm{\Omega }$ Impedance values were recorded and saved along with the EEG data.

Participants were seated approximately 110–130 cm from an LG Flatron L204WT-SF 20” wide LCD monitor. Experiments were conducted in rooms equipped with Faraday cage shielding as well as in common rooms without electrical shielding. The use of unshielded rooms was intended to simulate the environment that would be present during the Cybathlon 2020 competition.

Raw EEG signals were recorded using the BrainVision Recorder program (version: 1.22.0001) without additional software or hardware filters.

2.4. Two Choice Paradigm

This section presents the offline paradigm used for EEG recording with our pilots. The so-called Two Choice Paradigm was designed to simplify the execution of the Physionet task as our pilots reported difficulty in performing four-limb imagination during some experimental trials.

Prior to each experiment, pilots were instructed to avoid blinking, swallowing, clenching, or making any movements or facial expressions unrelated to the task during task periods. They were asked to repeatedly perform only the required motor imagery (MI) tasks while the fixation cross was displayed on the screen. During rest periods, the paradigm control program presented the next task on the screen in written form. During these periods, pilots were permitted to blink, swallow, and make any necessary movements to prepare for the next task. Pilots were instructed to perform motor tasks for 4 seconds and rest tasks for 3 seconds.

The Two Choice Paradigm, illustrated in Figure 2, began with a one-minute period during which participants were required to open their eyes and focus on the cross displayed on the screen. This was followed by a one-minute period during which participants were instructed to close their eyes. In both cases, participants were required to sit as calmly as possible, both physically and mentally, without engaging in any thoughts. This introductory session served as a baseline for the experiments and aimed to capture the pilots’ full attention in preparation for the MI sessions.

Figure 2. Two Choice Paradigm – It started with a one-minute open-eye and a one-minute closed-eye task, which served as a baseline and aimed to get the pilots’ full attention, preparing them for the MI sessions. Under one MI session, 8 active and 8 calm mental tasks were required from the pilots. The order of the tasks was randomized. The MI session was repeated 5 times under one experiment.

Following the introductory session, the experiment continued with 5 MI sessions. Each MI task was presented 8 times per session in a randomized order. After each completed session, participants were allowed to take a self-defined break without leaving the experimental setup.

For the active MI tasks, pilots were permitted to select and combine any hand and foot motor movements. However, these movements had to be decided upon and fixed prior to the start of the experiment. Pilot B selected Left Hand movements for the active task, while pilot C selected Both Feet movements. The calm task required participants to sit with their eyes open and refrain from making any movements or engaging in any thoughts or other potential sources of artifacts.

2.5. Offline Analyses

This section presents the methodology employed to compare features corresponding to EEG bands and to evaluate our BCI pipeline against the state-of-the-art EEGNet. These comparative analyses were conducted to determine the optimal configuration of the BCI system for real-time game control.

2.5.1. Investigating the Effects of EEG Bands on Classification Accuracy

To determine the optimal EEG bands for BCI control, an experiment was conducted in which several distinct signal processing and classification steps were performed. Each classifier received one of the investigated EEG bands: alpha, beta, gamma, theta, range30, or range40.

Experiments were conducted on both the Physionet dataset and our Two Choice Paradigm dataset. For the Physionet dataset, a 4-class classification was performed using the active MI tasks: Left Hand, Right Hand, Both Hands, and Both Feet. For the Two Choice Paradigm dataset, a 2-class classification was performed in which active MI imagination was classified against the calm phase.

Classification results for different EEG bands were collected and compared statistically. Repeated-measures ANOVA (rm-ANOVA) was performed for normally distributed data, followed by two-sided t-tests as post hoc tests. For non-normally distributed data, the Friedman test was used in place of rm-ANOVA and two-sided Wilcoxon signed-rank tests were used as post hoc tests. P-values were corrected using Bonferroni correction and the significance level was set at 0.05.

2.5.2. Comparison with EEGNet

EEGNet [8] was used as a benchmark to evaluate the accuracy and precision of our feature extraction and classification algorithms. The network was trained using 1-second-long EEG windows generated following FASTER artifact detection. The number of training epochs was set to 500, and an early stopping strategy was employed to prevent overfitting, with the patience parameter set to 20. Additionally, a custom strategy was used to save and restore the best network weights during testing: network weights were saved if the validation accuracy was greater than or equal to the previous value and the corresponding validation loss was lower.

Comparisons between our method and the state-of-the-art EEGNet were conducted using both the Two Choice Paradigm and Physionet datasets. The normality of the accuracy results was assessed, and either a t-test or Wilcoxon test was used to determine significant differences between the methods based on the results of the normality test.

2.6. Online Paradigm and Experiments

Following the offline comparison of signal processing and classification algorithms, the most suitable configuration was selected for use in the real-time BCI system. An online paradigm was developed to record data for tuning the BCI system’s classifier and to enable control of the game.

2.6.1. Online Paradigm

The online paradigm was designed to meet the requirements of the BCI race in the Cybathlon 2020 competition for subjects with tetraplegia (https://cybathlon.ethz.ch/en/event/disciplines/bci). The paradigm began with an offline training period used to calibrate the online game-playing phase. During this training period, the Two Choice Paradigm was conducted and the recorded data were used to train the BCI system’s classifier. Following calibration, the BCI system was able to control the BrainDriver program provided by the organizers of the Cybathlon 2020 competition. This program managed the virtual environment and race conditions for the BCI discipline. A computer monitor displayed the game for the pilot, providing immediate visual feedback on the results of their mental commands.

The BrainDriver program required four input commands from the user (three active commands plus the absence of any commands), but the Two Choice Paradigm was designed to elicit only two. To bridge this gap, a unique mechanism called the Toggle Switch was introduced, inspired by the Brain Tweaker team [3]. When an active MI task was performed by the user, game control commands were cycled through in sequence at a predefined frequency. When the desired control command was reached, the user had to initiate a calm mental task to maintain that command and send no further commands to the game. This mechanism is illustrated in Figure 3.

Figure 3. Components of our real-time BCI system and the Toggle Switch control mechanism.

Utilizing the Online Paradigm, a total of 16 experiments were executed with the participation of our two pilots.

2.6.2. BrainDriver Game

The BrainDriver software is a car racing-style video game developed for the BCI Race of the Cybathlon 2020 in collaboration with ETH Zurich and Insert Coin, Switzerland (http://www.insert-coin.ch/). Up to four players (pilots) can compete simultaneously in the game. Each player controls an avatar that moves forward by default and must reach the finish line. The objective is to guide the avatar through the racetrack by providing properly timed mental commands in designated zones. If an incorrect command is given or no input is provided when required, the pilot’s avatar slows down. Conversely, providing the correct control command restores the avatar’s default speed.

The game features four types of track elements: Left Turn, Right Turn, Light On, and Straight Zone. In the Light On zone, pilots must turn on their vehicle’s front light when the surrounding lights go off. In the Straight Zone, any command results in a slowdown of the avatar. The length of the game track was fixed at a virtual 500 meters by the Cybathlon organizers and includes four instances of each type of track element.

The BrainDriver game can be controlled using the UDP network communication protocol. Each player can send their control command as an unsigned byte code to the server’s IP and port address.

A track generator program was developed for the game to randomize the order of different track elements such that a straight track element always followed a turn. This program was used prior to each experiment in the Online Paradigm to generate a new game track and prevent pilots from memorizing the path.

2.6.3. Real-time BCI System

Our real-time BCI system required prerecorded training data obtained immediately prior to pilots playing the BrainDriver game. To acquire this data, the Online Paradigm was used to create the necessary dataset.

An RBF kerneled Voting SVM was used as the classifier for the BCI system. The classifier was trained offline using the range40 method with L2 normalization applied to 1-second-long windows with 0.1-second shifts.

Real-time EEG data were acquired from the amplifier using the Lab Streaming Layer (LSL) protocol [44]. Based on feedback from our pilots, one signal processing and decision-making step was performed every 1.6 seconds, corresponding to the periodic time for sending a game control command. During each decision-making step, the most recent 1-second of EEG data were treated as an EEG window and subjected to the same signal processing and classification steps as the prerecorded dataset: the range40 method was calculated with L2 normalization and input to the trained RBF kerneled Voting SVM for classification. The classification result was directly associated with a game control command (Figure 3) which was immediately sent to the game server’s IP address and port number via UDP protocol. The implemented signal processing and classification methods were sufficiently fast for use in a real-time environment.

Experiments with the real-time BCI system were conducted in a common room at the pilots’ institution without electrical shielding to simulate a Cybathlon-like environment.

3. Results

In this chapter, we present our findings regarding the investigation of EEG bands on both the Physionet and our own Two Choice Paradigm datasets, recorded with the assistance of our pilots. We also report the results of comparing our Voting SVM classifier using the range40 feature with the state-of-the-art EEGNet. Additionally, we present results obtained using the real-time BCI application, in which we measured the time required for pilots to reach the finish line in the BrainDriver game.

3.1. Investigating the Effect of EEG Bands on Classification Accuracy

For the Physionet dataset, a 4-class classification was performed, while for the Two Choice Paradigm dataset, a 2-class classification was conducted. The accuracy of each classification was measured for each experiment using 5-fold cross-validation. The final accuracy for each dataset was determined as the average of all 5-fold cross-validated experimental accuracies for all participants (Avg. Acc). These results are presented in Figure 4. Statistical tests were used to determine significant differences between EEG bands. Normality tests were first performed on the 5-fold cross-validated accuracies to determine the appropriate statistical tests to use.

Figure 4. EEG band effect investigation – On both Physionet and the Two Choice Paradigm database, the impact of different frequency range based features were investigated and compared with each other statistically. The significant differences between the canonical EEG bands and the range30 and range40 methods are marked with stars. The p-value annotation legend is the following: *: ${10}^{-2}<p\le 5\times {10}^{-2}$; **: ${10}^{-3}<p\le {10}^{-2}$; ***: ${10}^{-4}<p\le {10}^{-3}$; ****: $p\le {10}^{-4}$. The mean of the data is presented with the ’+’ symbol.

Table I presents the results of normality tests, the type of main effect statistical tests used, and their corresponding p-values. The 5-fold cross-validated results were not normally distributed; therefore, Friedman tests were used on both the Physionet and Two Choice Paradigm datasets. The main effect was significant for both datasets, so Wilcoxon tests were used to determine which EEG band could produce significantly higher classification accuracies.

Table I. Statistical test results of main effect on EEG band investigation.

Database	Normal distribution	Stat. test	p-value
Physionet	False	Friedman	1.612 x 10^−45
Two Choice Par.	False	Friedman	1.533 x 10^−9

As shown in Figure 4a for the Physionet dataset, the beta band achieved the highest accuracy among canonical EEG bands at 0.4285. It significantly outperformed all other canonical EEG bands except for gamma, which achieved an accuracy of 0.4097. However, when including range methods, range40 achieved significantly higher accuracy at 0.4607.

For the Two Choice Paradigm dataset, shown in Figure 4b, we obtained similar but less significant results compared to those for the Physionet dataset. The accuracies for beta, range30, and range40 were 0.7314, 0.7494, and 0.75 respectively. There were no significant differences between these EEG bands; however, this dataset contained only 16 experiments compared to 105 for the Physionet dataset after exclusion.

3.2. Comparision with EEGNet

We compared our Voting SVM classifier using the range40 method with the state-of-the-art EEGNet. For the Two Choice Paradigm dataset, the Voting SVM and EEGNet achieved accuracy levels of 0.7151 and 0.6857 respectively, while for the Physionet dataset, they achieved accuracy levels of 0.4866 and 0.4126 respectively. The distribution of results for EEGNet on the Physionet dataset was non-normal, so a Wilcoxon significance test was used with a preset significance level of 0.05. On the Physionet dataset, our Voting SVM with the range40 feature significantly outperformed EEGNet (p-value < ${10}^{-4}$). However, there was no significant difference between the two methods on our Two Choice Paradigm dataset. The results of these comparisons are presented in Figure 5.

Figure 5. 5-fold cross-validated accuracy level comparison of range40 + Voting SVM with EEGNet. The p-value annotation legend is the following: non-significant (ns): $5\times {10}^{-2}<p$; ****: $p\le {10}^{-4}$. The mean of the data is presented with the ’+’ symbol.

3.3. Real-time Working BCI Experiment

Using the online paradigm, we conducted a total of 59 gameplay trials over 9 experimental days with the assistance of pilots B and C. The experiments were conducted in a common room at the pilots’ institution without electrical shielding.

Each pilot played the BrainDriver game four times per experimental day. For each gameplay session, we measured the time required for the pilot to reach the finish line, as shown in Figure 6. The time limit of 240 seconds set by the organizers of the Cybathlon 2020 is indicated in red. A total of 27 gameplay sessions were completed within this time limit. We also present the learning curves for each pilot in Figure 6. Pilot B showed significant improvement (p-value 0.01 < 0.05), although it should be noted that he was absent for three experiments. Pilot C showed an insignificant increasing learning curve (p-value 0.092 > 0.05).

Figure 6. Gameplay performance of pilots per experimental day. 240 seconds were marked with a red line, which is the time limit defined by the organizers. The gray lines present the learning curves.

4. Discussion

In this article, we have presented the development of our BCI system for the Cybathlon 2020 competition. We chose to test our algorithms using the Physionet [25,27] dataset rather than BCI competition datasets because it contains experimental data from 109 subjects, while others have fewer than 10. This allowed us to obtain reliable information about the performance of our algorithms and conduct significance comparisons.

In addition to the Physionet dataset, we created our own dataset for the real-time BCI system with the assistance of participants with tetraplegia (referred to as pilots) who had spinal cord lesions at or above the C5 level. We designed a Two Choice Paradigm instead of a standard four-choice paradigm, as used in the Physionet dataset, because our pilots reported difficulty in performing four-limb imagination.

With regard to the BCI system, we implemented the FASTER algorithm [30] to meet the high requirements for artifact removal set by the Cybathlon organizers.

Our BCI system operates in the frequency domain. The absolute value of the FFT spectrum was calculated for 1-second-long EEG windows as a feature. From this FFTabs data, we either calculated the average between two frequencies or multiple averages from 2 Hz wide, non-overlapping frequency bins (referred to as the Feature Range method). In the case of the Feature Range method, multiple SVMs were trained, each receiving only one frequency bin. The final decision was determined by taking the maximum vote of all SVM units. We refer to this ensemble classifier as Voting SVM. To the best of our knowledge, Voting SVM combined with the range40 method based on FFTabs has not been previously investigated and compared statistically on MI datasets or used to control a computer game as part of a BCI application. A similar approach was reported in [43], but they used their algorithm on Common Spatial Pattern features and employed Bagging to generate random sub-datasets for each SVM unit. Their MI tasks were left little finger and tongue movements, and they did not use any artifact rejection algorithm prior to signal processing. Their proposed algorithm was not compared with any other published signal processing or classification methods.

We conducted multiple comparison analyses on both the Physionet and our own datasets to determine the most suitable configuration for our BCI system. First, we compared different EEG bands and range30 and range40 methods statistically. On the Physionet dataset, among canonical EEG bands (alpha, beta, gamma, theta), beta achieved the highest accuracy level and significantly outperformed all other bands except for gamma.

However, when including range methods in the comparison, range40 significantly outperformed all other methods. As a result, we selected range40 with Voting SVM classifier for comparison with state-of-the-art EEGNet [8] algorithm to provide a broader perspective on our findings within the BCI community. According to Wilcoxon statistical tests, our method significantly outperformed EEGNet on the Physionet dataset. Repeating these tests on our Two Choice Paradigm dataset yielded less significant results.

The performance of EEGNet was initially reported on the BCI Competition IV 2a dataset [23], where it achieved an accuracy of 0.6547 for 4-class classification on 9 subjects. On the Physionet dataset, we obtained an accuracy of 0.4126 for 4-class classification on 105 subjects. These two analyses may not be directly comparable due to the larger number of subjects in the Physionet dataset and our use of the FASTER algorithm to filter artifacts from the source signals. To obtain statistically significant results about differences between classifiers, we recommend using datasets with large number of subjects. Our Two Choice Paradigm dataset comprises data from 16 experimental sessions, collected with the participation of our two pilots. This may account for the lack of statistically significant difference between EEGNet and Voting SVM on this dataset.

After conducting these comparisons, we developed a real-time BCI system that includes a unique control protocol called the Toggle Switch. This algorithm allowed our pilots to control the BrainDriver computer game using only two mental commands instead of four. Our approach was inspired by Perdikis et al. [3], who developed an algorithm that classified two MI signals using a thresholding technique. When a third active game control command was required, their pilot initiated two different active MI tasks within a given time window. In contrast, our method cycles through active control commands one after another when our pilots initiate an active MI task, allowing for easy extension with additional commands.

Using the Online Paradigm and our BCI system, we conducted real-time BCI experiments with our pilots using the BrainDriver game developed for the BCI discipline of the Cybathlon 2020 competition. During these gameplay sessions, pilots received immediate feedback from the computer about the correctness of their mental commands. Our pilots completed the game with varying runtimes between 200 and 280 seconds, as shown in Figure 6. Pilot B showed a significant learning curve, while Pilot C faced difficulties. Nevertheless, the latter was statistically insignificant. Due to pandemic-related restrictions, we were only able to conduct 9 experimental days resulting in a total of 59 gameplay trials for both pilots. To further investigate the learning effect, additional experiments with the Online Paradigm would be required, which is considered as a limitation of our research.

To provide context for our work, we present results from other Cybathlon teams that participated in either the Cybathlon BCI Series 2019 or the Cybathlon 2020 event. The Nitro1 team [36] focused on minimizing within- and between-session variability and shifts using the Riemann framework. They projected generated features onto a common reference before performing classification using a Minimum Distance to Mean classifier. Their 4-class BCI system included two MI classes (Left Hand and Right Hand), mental subtraction, and an idle state. A blinking detector and thresholding technique applied to absolute EEG signals were used to reject features containing artifacts. While their approach showed increasing classification accuracies, this was not reflected in their game performance as measured by time required to reach the finish line. Most of their runs exceeded 250 seconds, whereas most of ours were below this threshold.

Team SEC FHT [37] implemented an artifact removal algorithm that detected EOG artifacts using Pearson’s correlation and interpolated affected channels. Filter Bank Common Spatial Pattern was calculated on the purified EEG data and a Gaussian-kerneled Support Vector Machine (SVM) was used for classification. Four MI tasks (Left Hand, Right Hand, Both Feet, Rest) were used to elicit control signals. The team investigated the precision of their control system with respect to the training protocol, comparing offline arrow-based, offline game-based, and online gameplay training. They achieved their best performance using training data derived from online gameplay, where the pilot received immediate feedback about the correctness of their commands. However, even when using their best method (training the classifier on data from previous gameplays), their gameplay results showed similar fluctuations to ours, with finish times ranging from 210 to 310 seconds.

The most significant improvements in BCI performance were reported by teams MIRAGE91 [6] and WHI [4], both of which achieved regression p-values below 0.001 for their learning curves. The performance range of team MIRAGE91 (160–300 seconds) is comparable to our results. However, team WHI outperformed all other teams, with competition times starting at 280 seconds and decreasing to 160 seconds.

Additional Cybathlon BCI topics can be found in [38,45–47].

5. Conclusion

In this paper, we presented a novel ensemble SVM classifier, termed Voting SVM, incorporating the range40 feature. To our knowledge, this configuration has not been previously utilized in MI-based BCI applications. Our signal processing and classification algorithm were rigorously evaluated using the Physionet dataset and demonstrated superior performance compared to the state-of-the-art EEGNet classifier.

We introduced the Two Choice Paradigm with a unique Toggle Switch control mechanism in our real-time BCI system for controlling the BrainDriver computer game. A total of 59 gameplay trials were conducted with two pilots, both diagnosed with C5 or higher spinal cord lesions. Our results, in terms of online gameplay, were comparable to those of other teams participating in Cybathlon 2020.

Future work will involve continued experimentation and data collection to expand our existing dataset. This will provide an opportunity to further develop our BCI system by incorporating additional features and normalization techniques and exploring the use of neural networks alongside EEGNet. Additionally, we plan to focus on subject learning, as it has been shown to significantly impact the robustness of BCI systems [3].

Despite challenges, our efforts have yielded promising results. As such, we intend to participate in the next Cybathlon event in 2024.

Credit authorship contribution statement

Csaba Köllőd: Conceptualization, Methodology, Experiments, Software, Statistics, Writing – original draft, Visualization

András Adolf: FASTER algorithm implementation, Experiments, Writing – FASTER algorithm

Gergely Márton: Experiments, Writing – review

Moutz Wahdow: Experiments, Writing – review

Ward Fadel: Experiments, Writing – review

István Ulbert: Supervision, Project administration, Writing – review & editing

Acknowledgements

We thank our Pilots for their weekly availability and participation in the experiments. We are also grateful for their regular feedback about the system.

Prepared with the professional support of the Doctoral Student Scholarship Program of the Co-operative Doctoral Program of the Ministry of Innovation and Technology financed from the National Research, Development and Innovation Fund.

Project no. FK132823 has been implemented with the support provided by the Ministry of Innovation and Technology of Hungary from the National Research, Development and Innovation Fund, financed under the FK 19 funding scheme.

This research was also funded by the Hungarian Brain Research Program (2017 1.2.1-NKP-2017- 00002) and TUDFO/51757-1/2019-ITM grant by the Hungarian National Research, Development and Innovation Office.

Disclosure statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability statement

The dataset analyzed during the current study is available from the corresponding author upon reasonable request.

The paradigm leader code can be found at https://github.com/kolcs/GoPar. For further details about the code, please read Supplementary Materials I.

Our BCI application’s source code is published at https://github.com/kolcs/bionic_apps.

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/2326263X.2023.2254463.

Additional information

Funding

This work was supported by the Ministry of Innovation and Technology of Hungary under Grant [KDP-2021-12 1020483].