Two sides of the same coin: adaptation of BCIs to internal states with user-centered design and electrophysiological features

ABSTRACT

The ideal brain–computer interface (BCI) adapts to the user’s state to enable optimal BCI performance. Two methods of BCI adaptation are commonly applied: User-centered design (UCD) responds to individual user needs and requirements. Passive BCIs can adapt via online analysis of electrophysiological signals. Despite similar goals, these methods are rarely discussed in combination. Hence, we organized a workshop for the 8th International BCI Meeting 2021 to discuss the combined application of both methods. Here we expand upon the workshop by discussing UCD in more detail regarding its utility for end-users as well as non-end-user-based early-stage BCI development. Furthermore, we explore electrophysiology-based online user state adaptation concerning consciousness and pain detection. The integration of the numerous BCI user state adaptation methods into a unified process remains challenging. Yet, further systematic accumulation of specific knowledge about assessment and integration of internal user states bears great potential for BCI optimization.

KEYWORDS:

• EEG
• BCI
• UCD
• signal diversity
• functional connectivity

1. Introduction

In the International Classification of Functioning of the World Health Organization1 ‘disability’ is defined as the ‘results of the interaction between an individual (with a health condition) and that individual’s contextual factors (personal and environmental factors)’. Over one billion people, which is about 15% of the global population, live with some form of disability (https://www.who.int/health-topics/disability). Article 9 of the Convention on the Rights of Persons with Disabilities underlines the right to accessibility: persons with disabilities should be enabled (on an equal basis with others) ‘to live independently and participate fully in all aspects of life’. This includes access to the physical environment, transportation, information, and communications, including information and communications technologies and systems (https://www.un.org/development/desa/disabilities/convention-on-the-rights-of-persons-with-disabilities/article-9-accessibility.html).

The term assistive technology (AT) indicates any product or technology-based service that enables people with activity limitations in their daily life [2]. The provision of AT represents environmental factors that may facilitate the functioning and improve accessibility, such as controlling a wheelchair or listening to text read out loud by a computer. The ATs can be classified according to their complexity in low (e.g. a pencil grip), mid (e.g. vocal output communication aids) and high-tech ATs (e.g. devices such as eye-trackers), and may include mainstream technologies (personal computer, tablet), software with customized user interfaces, and specific input devices (switches, mouse emulators, simplified keyboards or voice recognition tools) [2]. AT services host the process of designing personalized assistive solutions coping with end-user needs. Such a process involves multidisciplinary professionals and includes the assembly of different components of the AT solution. Furthermore, it considers the user’s motor, cognitive and sensory challenges as well as the peculiarities of their environment. A brain–computer interface (BCI), considered as a high-tech AT component in terms of an additional/alternative input device [3], could improve the inclusiveness of the personal AT solutions. The integration of the user-centered design (UCD; ISO 9241–210 [4]) principles in the design and evaluation processes of usable BCI systems implies a comprehensive understanding of the users in terms of cognitive and motor characteristics and their relation with the abilities to control a BCI [5–8], as well as the interaction with health professionals (also in terms of AT experts), medical companies and caregivers [9]. UCD was adopted, in the last years, as a paradigm in BCI optimization [e.g. 8,10,11]. The UCD focuses on a cycle where BCI design and development is based on the requirements of users, and the evaluation, against the requirements, would be utilized in the device optimization [9]. Hence, the evaluation process is consistent with the construct of usability, the term representing ease of use (ISO 9244–11 [12]). Among the metrics of usability, there are three evaluation factors that will be focused on in the current paper: effectiveness (how accurate and complete a user can accomplish a BCI-controlled application), efficiency (relating the costs invested by the user, i.e. effort and time, to effectiveness), and satisfaction (perceived comfort and acceptability while using the BCI).

A complimenting aspect to UCD-based BCI adaptation would be BCI adaptation based on monitoring the user’s internal state via electrophysiological signals and adjusting the BCI accordingly via machine learning. Indeed, the internal state of the user may change while the BCI is being used and the assessment based on psychological measures may not be feasible. In this scenario, internal states of the users can be assessed using physiological signals underlying attention, consciousness, fatigue, decision-making and sensory processing [13]. In regard to BCIs, the internal state will influence the user’s ability to generate the control signal for an active BCI or maintain attention on the stimuli used to control a reactive BCI [14]. Internal states have various effects on BCI performance [15]. In particular, effects on BCI feasibility and effectiveness are significant. This may occur due to fatigue and mental load [16], fluctuating consciousness [17,18], and attentional processing [5]. The adaptation to the user’s internal state based on brain signals decoded with BCI technology can be implemented by using a passive BCI (to distinguish the approach from BCIs designed for volitional communication or control) [14,19–21].

This paper will explore the utility of UCD in optimizing BCI development as well as the potential of adapting BCIs online by detecting the user’s internal states from the electrophysiological signal. In the first part of the paper, we will treat metrics to evaluate BCI in terms of usability, and their application in studies involving end-users as well as in early-stage BCI development studies. Within such metrics, those reflecting the user’s internal states are self-report metrics. This approach will be exemplified based on a study of training effects with BCIs that utilized auditory and tactile stimulation [based on research published in 22]. Hereafter, we will focus on selected electroencephalographic (EEG) measures as examples of electrophysiologically monitoring of the user’s internal states. We will outline studies that used EEG to determine internal states and then discuss two feature types, signal diversity [using research published in 23] and connectivity [based on 24], in more detail.

2. Summary of the workshop

This paper builds upon a two-hour workshop held at the 8th International BCI Meeting 2021 with the title ‘Optimising BCI performance by integrating information on the user’s internal state’. The workshop consisted of four (10/15-minute) presentations followed by 5–10 minute question and answer sessions. The four presentations covered the topics that are discussed in this paper: (1) UCD metrics and motivational aspects for basic study design, (2) BCI as AT: UCD in a clinical setting, (3) data and metrics for consciousness detection and (4) building a classification model with integrative EEG features. These presentations were followed by discussions in separate groups during which the participants completed a questionnaire, which is available in the supplemental materials of this paper. We incorporated ideas that emerged in the workshop discussion throughout this paper and also included the detailed answers to the questionnaire in our supplementary material for this paper. Further related material such as related programming scripts can be found on https://github.com/Han-YY/vBCI-Meeting_Workshop3.

3. Adapting the BCI based on user needs and usability assessment

One of the main motivations of BCI research has focused on providing optimized AT for end-users to reestablish communication and facilitate daily life activities. A step forward for the improvement of AT solution inclusiveness would be a full deployment of BCIs to end-users, by fully including them in the AT centers. The inclusion of BCIs in ATs [3,25] must include their integration with existing (available in the market) assistive or mainstream technologies. This would overcome the idea of a BCI as a stand-alone device, allowing end-users to use standard means of communication by switching to the BCI channel when the muscular one is fatigued or weak and/or by using them as complementary channels. As a proof of concept, a P300-based BCI serving as an input channel to access a commercial AT software for communication was developed and evaluated in terms of usability with healthy participants [26], end-users, and AT-experts [27]. Hereafter, to improve its usability, the P300-based BCI was endowed (based on user feedback) with an electromyography (EMG) channel, exploiting end-users residual muscular activity to delete errors in BCI selections [6]. The importance of the involvement of AT services in the development of innovative devices and their customization and validation resulted in the development of a P300-based BCI system interfacing with a commercial AT software GRID3 (Smartbox Assistive Technology) [28]. A checklist of the UCD process cycle adapted for BCI optimization as intended for end-users is illustrated in Figure 1 and selected aspects will be described in more detail in the following parts of the text.

A key aspect of the integration of the UCD principles in the design of optimized BCI systems implies user understanding. Two studies [5,7] showed that amyotrophic lateral sclerosis [ALS) disease could compromise the capability to control a P300-based BCI and alter the timing of the allocation of attentional resources in the post-perceptual stage of stimulus processing. Furthermore, the process of temporally filtering a target stimulus within a stream of stimuli was related to BCI control. Geronimo and colleagues [29] also underlined the importance of factors related to cognitive function in end-users with ALS in a successful BCI operation (in P300 and motor-imagery based BCI control].

Beyond thorough initial user understanding, an important specific step of the UCD process, consists of the collection of standardized metrics to measure the usability aspects effectiveness, efficiency, and satisfaction. These standardized metrics are not only beneficial for relatively late BCI development stages, but could already inform BCI development from a relatively early stage on [30]. This will be illustrated subsequently via a more detailed look at one selected early-stage BCI development study with healthy non-end-user participants [22]. An overview of usability and BCI-relevant aspects with relevant examples for standardized metrics and their assessment times in the selected study example [22] can be found in Table 1. Since the importance of optimizing user motivation has been highlighted by several recommendations on BCI study protocol standardization [31–34], we will also discuss motivational study design aspects. An overview of selected aspects to potentially optimize BCI user motivation and performance as implemented in the selected study example [22] is summarized in Table 2. In addition to the measures outlined in Table 1, visual analog scales (VAS) were used to subjectively assess motivation on a global level and the Questionnaire for Current Motivation in Learning and Performance Situations was used in its adapted BCI version (QCM-BCI) to subjectively assess motivation on a more specific level regarding interest, mastery confidence, incompetence fear, and challenge [18,22,35,36].

Figure 1. Checklist of the user-centered design (UCD) step-by-step process cycle adapted for brain-computer interface (BCI) optimization intended for end-users with exemplary questions for each step [based on 11,37]. It has been noted that the UCD implies a careful definition and selection of the targeted end-users. An algorithm for such selection was recently suggested [38]. On a further note, another workshop at the 8th International BCI Meeting 2021 focused on ideas ‘Toward an international consensus on user characterization and BCI outcomes in settings of daily living’ with an accompanying database, summarizing user factors and outcome measures: https://www.notion.so/6c11535322d04977a14cfaa60ba5494f?v=4c92db74d5854f30bef197b8a9cdd327.

Table 1. Overview of usability and BCI-relevant aspects (with examples for relevant standardized metrics) as well as their assessment times with potential end-users (with exemplary studies using these measures) and their assessment times in the selected early-stage BCI development study example with healthy non-end-users [22], based on earlier work [11,37]

Usability aspects	BCI-relevant aspects [with examples for relevant standardized metrics]	Assessment times with potential end-users [with exemplary studies using these measures]	Assessment times in the selected early stage BCI development study example with healthy non-end-users [22]
Effectiveness (how accurate and complete a user can accomplish BCI control)	Accuracy [% correct responses] [preferably measured online via concrete task results vs. offline via estimation from previously collected data)	Each session [many studies, extensively discussed by 39]	Over each session
Efficiency (relating the costs invested by the user, i.e. effort and time, to effectiveness]	Information transfer rate [ITR) [bits/min]	Each session [many studies, extensively discussed by 39]	Over each session
	Adjustment of the ITR: Utility metric [bits/min = 0 if effectiveness < 50%]	Each session [40]	Applied in one occasion of a temporarily demotivated participant (with an online accuracy score of < 30%]
	Subjective Workload [NASA-TLX]	Each session/task [6]	End of each session
Satisfaction (perceived comfort and acceptability while using the BCI)	General aspects of assistive technology [QUEST 2.0, expandable by four additional BCI-related items operationalizing reliability, learnability, speed, and esthetic design]	End of prototype testing [27,40–44]	-(not included due to time constraints)
	Overall satisfaction [visual analog scale ranging from 0–10]	Each session [45,46]	End of each session
	Interview [semistructured; free]	End of prototype testing [47]	“Mini interview” at the end of each session (“Could you imagine using a BCI for communication? What problems would need to be addressed? What would be major improvements? Did you notice anything else?”)

Notes. NASA-TLX = NASA-Task Load Index [53,54]. QUEST = Quebec User Evaluation of Satisfaction with Assistive Technology [82].

Table 2. Overview of selected aspects to potentially optimize BCI user motivation and performance as implemented in the selected study example [22], based on earlier study design guidelines [31,32] and the idea to use elements inspired by a popular science fiction movie franchise (pictures, sounds, etc.), in which characters can use the powers of their mind to positively interact with their environment via non-muscular pathways (‘Star Wars’), to create an overarching and motivating theme [48]

Suggested properties of a good instructional design	Corresponding design aspects in the study example [22]
Feedback
Non-evaluative and supportive feedback that conducts to a feeling of competence	Positive feedback sound after successful BCI use (not given after nonsuccessful BCI use) as well as more elaborated non-evaluative and supportive feedback from the experimenter during breaks
Engaging feedback and environment	Positive ‘Star Wars’ feedback sound (from a popular robot character, based on a mix of harmonic electronic and cheerful baby sounds)
Explanatory and specific feedback	Adjustable breaks during sessions, allowing time for verbal feedback by the experimenter as well as intermediate analysis of potential problems
Instruction
Goals should be clearly defined	Careful briefing and debriefing (explanation of paradigm and research purpose, suggesting strategies for successful use, concluding analysis of potential problems)
The meaning of the feedback should be explained	Illustration and explanation of the study specific BCI signal and ways of subjectively influencing it
The skill to be learned should be demonstrated	Demonstration of (previously recorded) successful BCI use of a end-user (also to illustrate the practical relevance of this line of research)
Task
Motivation and positive emotions promote learning	‘Star Wars’ theme (absolving ‘mission’ that were inspired by movie scenes), creation of a positive atmosphere (little refreshment breaks)
Need for autonomy and work at user’s own pace, adaptation of training procedure to the user	Adjustable breaks during sessions, allowing time to drink or snack little refreshments, eventually chance to try an alternative BCI version (auditory vs. tactile)
Need for variability over tasks and problems, progressive and adaptive tasks	Varying ‘missions’ (each one to a new ‘Star Wars’ movie scene), new BCI calibration at the beginning of each session (allowing users to reach online accuracies > 70% without guaranteeing 100%)

Notes. Originally, these suggestions were not meant for P300-based BCIs as utilized in the selected study example [22], since it seemed that they did not require human training [32]. However, this was before the relevance of training effects could be shown for auditory and tactile P300-based BCIs [22,49–52,84].

As brief background information, the study example [22] examined two P300-based BCI versions, an auditory and a tactile one. Since the current workshop paper’s focus is intended to be relevant for various BCI types, we will focus on selected study results to illustrate the benefits of considering the aforementioned usability measures and motivational aspects. A first exemplary question of interest was, whether the applied BCIs could be successfully used and trained. In this regard, the two objective measures of effectiveness (online accuracy) and efficiency (information transfer rate; ITR) could offer complementary information. While accuracy remained relatively stable around 80% to 85% and indicated successful BCI use, it did not indicate a training effect. Yet, the addition of the time aspect via ITR revealed training effects, since it increased significantly during training for the auditory as well as the tactile BCI version. This would have been unnoticed, if only accuracy would have been considered as a performance measure. A second exemplary question of interest was, whether these training effects could be transferred from using the auditory BCI to using the tactile BCI and vice versa. With regard to this question, online accuracy as well as ITR indicated that the switch from the auditory to the tactile BCI version seemingly occurred more easily than vice versa, but only non-significantly by trend. The subjective measures of efficiency, satisfaction and motivation could help to form a more complete picture. Considering efficiency, the NASA-Task Load Index [NASA-TLX; 53,54] and VAS could show that a significant decrease of subjective workload was experienced when the auditory BCI was trained (which was absent during tactile BCI training) and that the switch from the tactile to the auditory BCI led to a significant increase in subjective workload (which was absent vice versa). Last but not least, satisfaction and motivation were reported to be on a high level overall with a stable overall high QCM-BCI mastery confidence, accompanied by a low QCM-BCI incompetence fear. However, there were significant decreases regarding global motivation when training the tactile BCI, accompanied by declines by trend regarding VAS satisfaction and QCM-BCI feelings of challenge (each absent during auditory BCI training). Additionally, the auditory training group showed significantly more QCM-BCI interest in trying another alternative BCI version (absent in the tactile group), which was not the case for the tactile training group. In summary, this pattern of findings led to the interpretation that even though both BCIs were interpreted as successfully usable and trainable (based on objective effectiveness/online accuracy and efficiency/ITR), the auditory BCI was interpreted as subjectively harder but more rewarding to train, while the tactile BCI was interpreted as more intuitive but more monotonous and therefore less rewarding to train (based on subjective efficiency/NASA-TLX/VAS, satisfaction/VAS, and motivation/QCM-BCI/VAS). This was furthermore supported by the optional ‘mini interview’ statements, which all in all paraphrased this interpretation pattern. As a closing detail, a look at individual performances indicated highly individual BCI version preferences as well as the possible prevention of one dropout case, who despite having a very bad result at one of the training sessions (with one occasion of an online accuracy score of < 30% and therefore an utility metric of zero), was motivated to successfully complete his training. In conclusion, the usability assessment and consideration of user motivation could offer a more complete picture of BCI performance. This could provide a valuable basis for concrete improvement ideas, such as increasing the complexity of the specifically used tactile BCI to make it less monotonous and its training and use therefore more beneficial, while also keeping potential individual BCI preferences in mind.

4. Adapting the BCI using physiological features

The concept of a passive BCI that uses brain signals to monitor the user’s cognitive state was first proposed by [14]. A recent review [20] suggested the application of passive BCIs to detect workload, stress, emotions, fatigue, and attention. The data needed to determine these states can be extracted from the EEG itself but also peripheral measures such as eye-tracking, galvanic skin response, and heart rate. This approach is the basis of adapting a BCI to the user’s state using physiological features.

One of the most intuitive user states to detect during BCI usage is the workload. Increased workload is reflected in power spectral density (PSD) features and can also influence event-related potentials (ERPs), depending on the BCI used [17]. More recent work on predicting workload in a computer game-like task achieved over 90% accuracy by employing spatial filtering [55]. Potentially, measures extracted from the EEG may be augmented with peripheral measures such as heart rate and eye-movements. The transition from high workload to mental fatigue was shown to be reflected in increased delta, theta, and alpha power as well as decreased heart and increased blink rate [56]. Similarly to workload, fatigue detection may be enhanced by decomposing the signal into independent components as shown in a realistic aviation task [57]. Deep learning methods have also been applied to the problem showing successes in mental workload detection [58], which was expanded to emotion recognition based on EEG features and peripheral measures [59]. Adaptation of the BCI to the attention of the user has also been proposed. For example, this may occur before the task is performed via increases in power in the alpha band [60] or after the task is performed via error-related potentials [61]. Asynchronous BCIs can also rely on the detection of attention to allow self-paced control of the BCI. For example, the frequency of visual stimuli elicits a unique response that can be used to determine if the user is paying attention to the BCI, pausing the system when the user does not, thus allowing self-paced control [62].

A measure that has not yet been extensively discussed in the passive BCI literature and which was presented at our workshop at the 8th International BCI Meeting 2021 is signal diversity. Signal diversity could for instance be measured via Synchrony Coalition Entropy [63] or via Lempel-Ziv complexity, which measures the compressibility of the data based on the Lempel-Ziv compression algorithm [64]. At our workshop, we focused on Lempel-Ziv complexity, since it is one of the most commonly used quantifications of signal diversity that has been established in the field of consciousness detection as a reliable feature that can be extracted from the time domain. A detailed example of computing Lempel-Ziv complexity on data that was recorded during the so-called Wada test [previously published in 23] can be found in our supplemental workshop material (https://github.com/Han-YY/vBCI-Meeting_Workshop3).

As a further example, the brain response to pain is a typical case of a brain state involving multiple brain regions, thus our work in pain assessment will be used as an example to show the use of integrative measurements for assessing internal states. The processing of pain relies on the dynamic integration of cognitive, emotional, and motivational processes in different regions of the brain [65,66] and the brain network for pain processing was studied using functional connectivity [67,68]. Pain processing includes delta, theta, alpha, beta, and gamma oscillations [66,69]. Cross-frequency coupling (CFC) was used to reveal the integration between frequency bands in pain processing. A typical example is the role of CFC between alpha and beta bands in sensorimotor areas in acute pain assessment [69,70].

Nickel et al. demonstrated the potential of functional connectivity and CFC in internal state assessment [69]. However, to distinguish different states, it is still required to quantify these integrations, so they can be used as classification features. To analyze the performance of these integrative features, there are still two aspects that should be considered about the features’ comparability: a) In signal processing, there are two properties to be extracted, power and phase, so it is necessary to choose metrics able to show the individual effect of power or phase in the assessment. b) Regardless of the type of studied integration (i.e. functional connectivity or CFC) the measurements always focus on the synchrony between two signal series. Hence, it is essential to use consistent metrics to measure different integrations, so that we can mix different categories of significant integrative features when there are multiple significant integrations showing the characteristics of the corresponding internal state.

For measuring the synchrony based on the signal power, spectral coherence (i.e. square-magnitude coherence) is a common choice [71]. On the other hand, there were several classical metrics proposed for measuring phase synchrony, such as phase lag index [PLI, 72], phase locking value [PLV, 73] and intersite phase clustering (ISPC), which was adapted from intertrial phase clustering [71,74].

Because the only difference between spectral coherence and ISPC in the Euler-like format is the inclusion of the power information (i.e. magnitudes of the series), in our previous work [24], functional connectivity and CFC were utilized as features in tonic pain prediction. Four conditions (including two resting states, one thermal pain condition and one non-painful thermal condition) were classified with these features. For simplifying the comparison, all the functional connectivities were extracted from the alpha band, since it is assumed to be the most important oscillation in pain processing [75–77], and all CFC were extracted with different frequency bands from the same channel. From this work, two advantages of integrative features were revealed. First, the performances of different integrative features can be analyzed and compared directly with the accuracies produced with the corresponding classifier. Based on this information, we are able to determine which features should be utilized for developing an efficient and accurate classification model. Second, because of the considerable amount of features generated with multiple EEG channels or frequency bands, we selected features with neighborhood component analysis (NCA) [78]. This way, the brain regions or frequency bands with the highest predictability of pain processing can be determined. Related analysis scripts can be found in our supplemental workshop material on https://github.com/Han-YY/vBCI-Meeting_Workshop3.

In this section, we focused on the use of integrative measurements as features in pain state assessment. In other applications, the usefulness of selected features will depend on the particular internal state that should be detected. While analyzing the brain signals to assess internal states, one approach is to compare the activities at specific regions. For example, a significant difference was found at parietal and frontal lobes between the responses of vegetative and minimally conscious state patients [79,80]. Furthermore, neuronal oscillations represent various effects, for instance, the change of power in theta and alpha bands have strong correlations with fatigue [81]. In conclusion, for the assessment of internal states with neural signals, both connectivity between brain regions and neuronal oscillations should be utilized.

5. Discussion and future directions

We summarized and expanded upon the topics discussed in the workshop ‘Optimising BCI performance by integrating information on the user’s internal state’, which was held at 8th International BCI Meeting 2021. Two possible approaches were presented, ‘Adapting the BCI based on user needs and usability assessment’ and ‘Adapting the BCI using physiological features’. These two approaches will now be discussed in two separate sections that provide specifically selected discussion aspects with regard to each approach. In the end, both approaches will be presented in a unified way in the third discussion section ‘A unified approach of BCI adaptation’, which provides a more global outlook and closing summary of our key messages.

5.1. Adapting the BCI based on user needs and usability assessment

Various selections of standardized measures that differ from those selected by the described study [22] might prove helpful as well, depending on the specific aim of a study. The relatively exhaustive Quebec User Evaluation of Satisfaction with Assistive Technology could be used in its adapted BCI version as a detailed measure to assess the usability criterion satisfaction [11,82,83] or the relatively economic System Usability Scale could be used as a ‘quick and dirty’ operationalization for overall usability85–88[. This could, for instance, help with designing optimal tactile BCIs from an early stage on, where several promising options have recently been discussed [22,84,89–92]. Especially the idea of training different BCI versions might prove useful. On the one hand, this could allow a more varied training, on the other hand, it might make sense to look for the optimal BCI for each individual user instead of an optimal BCI in general.

The applied motivational aspects could also be adjusted and merit further research on the optimal way of application. Concerning the motivational ideas summarized in Table 2, it was originally noted that the listed suggestions were only based on theory and would be in need of formal validation [32]. Since then, several findings have been reported in support of selected theoretical ideas, underlining the importance of considering BCI user motivation [33,34,93–96]. However, further thoroughly conducted studies could shed more light on which motivational aspects work best under which specific circumstances, such as recent work that examined the optimization of biassed vs. unbiased motor imagery BCI feedback while also considering various user states and traits [97,98].

As a concluding example, which has also been discussed at the 8th International BCI Meeting 2021, the design of optimal BCIs for children has recently led to the formation of research initiatives, where UCD and motivational aspects could again be of key importance [99–101]. It should be ensured that the involved measures are child friendly. Simple and short VAS instead of exhaustive text heavy questionnaires could ensure child user motivation and produce valid results that would not be potentially biassed by monotony or boredom. In the course of this, various findings from the motivational adult BCI literature could be promising for translation to child BCIs, like the robot learning companion PEANUT (Personalized Emotional Agent for Neurotechnology User Training) [96] as well as the inclusion of gamification principles in general, as implemented in various BCI controlled applications [102–104].

5.2. Adapting the BCI using physiological features

With regard to the approach using machine learning to detect internal states of the user from the EEG, other states beyond the two presented in the paper (consciousness detected using signal diversity and pain detected using connectivity), are also worth exploring. Many of these were mentioned and discussed by the participants in the workshop. Generally, metrics measuring the connectivity between brain regions or coupling between frequency bands are powerful tools to predict internal states. Rosenberg et al. demonstrated that attention can be potentially measured with whole-brain functional network strength as a neuromarker [105], and Granger causality-based functional connectivity has been used to detect selective attention [106]. Connectivity patterns have also been shown to be related to intensities of fatigue [107] and during cognitive workload at various task-difficulties [108]. As to the CFC, it has been confirmed that the CFC between delta-alpha and delta-beta bands are potential biomarkers to detect motivation [109] and Dimitriadis et al. also demonstrated cross-frequency phase interaction is a reliable predictor of workload [110]. The advantages of such integrative features for the detection of internal states are significant: a) Compared with the measurements directly extracted from raw neurophysiological signals such as PSD, the integration among different sources of signals (i.e. brain regions or neuronal oscillations) can generate more abundant information. b) The phase-based integrative features have one extra benefit as a feature in machine learning. Because of the specific range of phase (from -π to π), the measurements based on phase synchrony can be produced in a strictly limited range of values, e.g. from 0 to 1 in ISPC, PLI and PLV [71–73]. Such a limited quantitative range makes the levels of integrations comparable in all cases, and within the request of normalization as features in machine learning, it does help. Hence, such integrative measurements can be utilized as a kind of ideal quantified references in assessing the internal states.

Signal diversity is also not limited to the detection of consciousness. For example, entropy estimated from the continuous EEG and classified using a neural network was used to detect epileptic seizures during which entropy drops sharply [111]. The estimation of diversity does not need to be based on the continuous EEG. Inouye and colleagues [112] demonstrated that the entropy of the power spectrum increases with mental workload. Entropy measures have also been applied to evoked responses as opposed to spontaneous EEG. Sitges and colleagues [113] used multiscale entropy to distinguish responses of controls and patients with chronic pain to non-painful stimuli. Overall, this research shows that entropy measures and signal diversity measures in general have the potential to be used as a feature to develop a BCI that adapts to the internal state of the user.

5.3. A unified approach of BCI adaptation

A possible design for a BCI that adapts to the internal state of the user is shown in Figure 2. We distinguish the adaptation to the user’s state into two categories based on the methods used to determine the state. On the one hand, behavioral metrics such as questionnaires and interviews may be used to assess the user’s state (contributing to BCI usability assessment) in regard to workload, satisfaction and motivation. For example, self-reported VAS ratings may be used to determine the user’s perception of specific BCI aspects and interviews may shed light on potentially unknown aspects. This approach could be used to evaluate adaptations of the training paradigms to ensure the changes made to the training made the task more engaging for the user. This in turn should lead to increased satisfaction and motivation. On the other hand, features such as oscillations reflected in the PSD, ERPs, signal diversity and integrative features (such as functional connectivity) may be used to assess the user’s current state during active use of the BCI. For example, if high workload or fatigue are detected, the pauses between trials may be increased or the BCI could switch to another control modality (for instance, from P300 to motor imagery). If measures of consciousness indicate the user is currently not conscious, the BCI may be paused, and if an episode of acute pain is detected, a caregiver may be alerted.

There has been much discussion about the role of internal states in BCI designed with UCD toward health management, but it is also a vital factor in more general BCI systems designed for some non-medical environments, especially in closed-loop BCI. Based on the idea that the adaption adapts the measured internal states, the utility of internal states is an essential element in the design of a closed-loop BCI system. Until now, there have been some researchers proving such an important functionality of the internal states [114]. For example, Müller et al. proposed a closed-loop EEG system that can adapt the setting of a driving system according to the driver’s workload [115]. Moreover, in a study focusing on the closed-loop BCI applied in virtual reality, Luu et al. also realized the potential of internal states as a powerful factor in improving the system’s performance [116]. Though the technical solution to utilize internal states in the development of closed-loop BCI regarding specific applications still involves various challenges, its importance has been disclosed significantly both in theory and in application.

Figure 2. An example of the brain-computer interface (BCI) control cycle and how adaptation to the internal state of the user could be included. The user generates a control signal that is decoded by the BCI and provided as feedback to the user. In this example, the BCI is adapted by applying user-centered design (UCD) to determine the preferences and abilities of the user via self-report measures (such as the NASA-Task Load Index [NASA-TLX; 53,54], the System Usability Scale [SUS; 85,86], the BCI version of the Questionnaire for Current Motivation in Learning and Performance Situations [QCM-BCI; 18,35,36], visual analogue scales (VAS), or using an interview approach) and also via the detection of electrophysiological signal features (such as electroencephalographic oscillations, signal diversity and connectivity) by applying machine learning to determine the current state of the user.

6. Conclusion

We discussed the application of UCD to respond to the users’ needs while using BCI, both in studies with healthy non-end-users and end-users in clinical environments. Furthermore, we highlighted electrophysiological signal features that can, for example, be extracted from EEG, such as Lempel-Ziv complexity and functional connectivity, and can be applied for the detection of specific internal states.

We conclude that the ‘two sides of the same coin’ of internal states credo can help the development of BCIs to involve more interactive feedback, by adapting BCIs to diverse internal states in different conditions to improve BCI performances. This may further inspire the development of bi-directional BCIs and may help the user to adjust the internal states to the current condition [117]. With the help of UCD and novel approaches for internal state detectionfor instance, with effective EEG features, it will be possible to make BCIs more flexible and will help with optimizing development of closed-loop neurofeedback systems [118].

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.