Heading for new shores! Overcoming pitfalls in BCI design

References

• Wolpaw J, Wolpaw E. Brain–computer interfacesprinciples and practice. New York (NY): Oxford University Press; 2012.10.1093/acprof:oso/9780195388855.001.0001
   Google Scholar
• Lotte F, Bougrain L, Clerc M. Electroencephalography (EEG)-based brain-computer interfaces. In: Wiley encyclopedia on electrical and electronics engineering. Wiley; 2015. doi:http://dx.doi.org/10.1002/047134608X.W8278
   Google Scholar
• Borton D, Micera S, Millán JDR, et al. Personalized neuroprosthetics. Sci Transl Med. 2013;5:210rv2.
   PubMed Web of Science ®Google Scholar
• Thakor NV. Translating the brain-machine interface. Sci Transl Med. 2013;5:210ps17.
   PubMed Web of Science ®Google Scholar
• Bashashati A, Fatourechi M, Ward RK, et al. A survey of signal processing algorithms in brain–computer interfaces based on electrical brain signals. J Neural Eng. 2007;4:R32–R57.10.1088/1741-2560/4/2/R03
   PubMed Web of Science ®Google Scholar
• Lotte F, Congedo M, Lécuyer A, et al. A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng. 2007;4:R1–R13.10.1088/1741-2560/4/2/R01
   PubMed Web of Science ®Google Scholar
• Allison B, Neuper C. Could anyone use a BCI? In: Brain-Computer Interfaces. London: Springer; 2010. p. 35–54.10.1007/978-1-84996-272-8
   Google Scholar
• Makeig S, Kothe C, Mullen T, et al. Evolving signal processing for brain–computer interfaces. Proc. IEEE. 2012;100:1567–1584.10.1109/JPROC.2012.2185009
   Web of Science ®Google Scholar
• Allison B, Luth T, Valbuena D, et al. BCI demographics: how many (and what kinds of) people can use an SSVEP BCI? IEEE Trans Neural Syst Rehabil Eng. 2010;18(2):107–116. doi:10.1109/TNSRE.2009.2039495
   PubMed Web of Science ®Google Scholar
• Guger C, Daban S, Sellers E, et al. How many people are able to control a P300-based brain-computer interface (BCI)? Neurosci Lett. 2009;462:94–98.10.1016/j.neulet.2009.06.045
   PubMed Web of Science ®Google Scholar
• Lotte F, Larrue F, Mühl C. Flaws in current human training protocols for spontaneous brain-computer interfaces: lessons learned from instructional design. Front Hum Neurosci. 2013;7:568. doi:10.3389/fnhum.2013.0056
   PubMed Web of Science ®Google Scholar
• Scherer R, Faller J, Balderas D, et al. Brain-computer interfacing: more than the sum of its parts. Soft Comput. 2013;17(2):317–331.10.1007/s00500-012-0895-4
   Web of Science ®Google Scholar
• Scherer R, Pfurtscheller G. Thought-based interaction with the physical world. Trends Cogn Sci. 2013;17(10):490–492.10.1016/j.tics.2013.08.004
   PubMed Web of Science ®Google Scholar
• Kübler A, Holz E, Riccio A, et al. The user-centered design as novel perspective for evaluating the usability of BCI-controlled applications. PLoS ONE. 2014;9:e112392.10.1371/journal.pone.0112392
   PubMed Web of Science ®Google Scholar
• Kübler A, Holz EM, Sellers EW, et al. Toward independent home use of brain-computer interfaces: a decision algorithm for selection of potential end-users. Arch Phys Med Rehabil. 2015;96(3):S27–S32. doi:10.1016/j.apmr.2014.03.036
   PubMed Web of Science ®Google Scholar
• Zickler C, Halder S, Kleih SC, et al. Brain painting: usability testing according to the user-centered design in end users with severe motor paralysis. Artif Intell Med. 2013;59(2):99–110. doi:10.1016/j.artmed.2013.08.003
   PubMed Web of Science ®Google Scholar
• Nijboer F, Plass-Oude Bos D, Blokland Y, et al. Design requirements and potential target users for brain-computer interfaces–recommendations from rehabilitation professionals. Brain-Computer Interfaces. 2014;1(1):50–61.10.1080/2326263X.2013.877210
   Google Scholar
• Huggins JE, Wren PA, Gruis KL. What would brain–computer interface users want? opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis. 2011;12(5):318–324.10.3109/17482968.2011.572978
   PubMed Web of Science ®Google Scholar
• Blain-Moraes S, Schaff R, Gruis KL, et al. Barriers to and mediators of brain–computer interface user acceptance: focus group findings. Ergonomics. 2012;55(5):516–525.10.1080/00140139.2012.661082
   PubMed Web of Science ®Google Scholar
• Andresen EM, Fried-Oken M, Peters B, et al. Initial constructs for patient-centered outcome measures to evaluate brain–computer interfaces. Disabil Rehabil Assist Technol. 2016;11(7):548–557.
   PubMed Web of Science ®Google Scholar
• Huggins JE, Moinuddin AA, Chiodo AE, et al. What would brain-computer interface users want: opinions and priorities of potential users with spinal cord injury. Arch Phys Med Rehabil. 2015;96(3):S38–S45.e5. e5.10.1016/j.apmr.2014.05.028
   PubMed Web of Science ®Google Scholar
• Peters B, Bieker G, Heckman SM, et al. Brain-computer interface users speak up: the virtual users’ forum at the 2013 international brain-computer interface meeting. Arch Phys Med Rehabil. 2015;96(3):S33–S37.10.1016/j.apmr.2014.03.037
   PubMed Web of Science ®Google Scholar
• Holz EM, Botrel L, Kaufmann T, et al. Long-term independent brain-computer interface home use improves quality of life of a patient in the locked-in state: a case study. Arch Phys Med Rehabil. 2015;96(3):S16–S26. doi:10.1016/j.apmr.2014.03.035
   PubMed Web of Science ®Google Scholar
• Sellers EW, Vaughan TM, Wolpaw JR. A brain-computer interface for long-term independent home use. Amyotrophic Lateral Sclerosis. 2010;11(5):449–455. doi:10.3109/17482961003777470
   PubMed Web of Science ®Google Scholar
• Sellers EW, Ryan DB, Hauser CK. Noninvasive brain-computer interface enables communication after brainstem stroke. Sci Transl Med. 2014;6:257re7.
   PubMed Web of Science ®Google Scholar
• Blankertz B, Sannelli C, Halder S, et al. Neurophysiological predictor of SMR-based BCI performance. NeuroImage. 2010;51(4):1303–1309. doi:10.1016/j.neuroimage.2010.03.022
   PubMed Web of Science ®Google Scholar
• Ahn M, Ahn S, Hong JH, et al. Gamma band activity associated with BCI performance – simultaneous MEG/EEG study. Front Hum Neurosci. 2013;7. doi:10.3389/fnhum.2013.00848
   PubMed Web of Science ®Google Scholar
• Zhang Y, Xu P, Guo D, et al. Prediction of SSVEP-based BCI performance by the resting-state EEG network. J Neural Eng. 2013;10(6):066017.10.1088/1741-2560/10/6/066017
   PubMed Web of Science ®Google Scholar
• Saeedi S, Chavarriaga R, Leeb R, et al. Adaptive assistance for brain-computer interfaces by online prediction of command reliability. IEEE Comput Intell Mag. 2015;11:32–29. doi:10.1109/MCI.2015.2501550
   Web of Science ®Google Scholar
• Saeedi S, Chavarriaga R, Millán JDR. Long-term stable control of motor-imagery BCI by a locked-in user through adaptive assistance. IEEE Trans Neural Syst Rehab Eng; Forthcoming.
   Web of Science ®Google Scholar
• Kaufmann T, Vögele C, Sütterlin S, et al. Effects of resting heart rate variability on performance in the P300 brain-computer interface. Int J Psychophysiol. 2012;83(3):336–341. doi:10.1016/j.ijpsycho.2011.11.018
   PubMed Web of Science ®Google Scholar
• Sutton S, Braren M, Zubin J, et al. Evoked-potential correlates of stimulus uncertainty. Science. 1965;150(3700):1187–1188. doi:10.1126/science.150.3700.1187
   PubMed Web of Science ®Google Scholar
• Halder S, Furdea A, Varkuti B, et al. Auditory standard oddball and visual P300 brain-computer interface performance. Int J Bioelectromagn. 2011;13(1):5–6.
   Google Scholar
• Kaufmann T, Holz EM, Kubler A. Comparison of tactile, auditory, and visual modality for brain-computer interface use: a case study with a patient in the locked-in state. Front Neurosci. 2013;7:129. doi:10.3389/fnins.2013.00129
   PubMed Web of Science ®Google Scholar
• Hammer EM, Halder S, Blankertz B, et al. Psychological predictors of SMR-BCI performance. Biol Psychol. 2012;89(1):80–86. doi:10.1016/j.biopsycho.2011.09.006
   PubMed Web of Science ®Google Scholar
• Vukovic A. Motor imagery questionnaire as a method to detect BCI illiteracy. Paper presented at: 2010 3rd International Symposium on Applied Sciences in Biomedical and Communication Technologies (ISABEL 2010); 2010 7-10 Nov; 2010.
   Google Scholar
• Jeunet C, N’Kaoua B, Lotte F. Advances in user-training for mental-imagery based BCI control: psychological and cognitive factors and their neural correlates. Prog Brain Res. 2016;228:3–35.
   PubMed Web of Science ®Google Scholar
• Riccio A, Simione L, Schettini F, et al. Attention and P300-based BCI performance in people with amyotrophic lateral sclerosis. Front Hum Neurosci. 2013;732. doi: 10.3389/fnhum.2013.00732
   Web of Science ®Google Scholar
• Kleih SC, Nijboer F, Halder S, et al. Motivation modulates the P300 amplitude during brain-computer interface use. Clin Neurophysiol. 2010;121(7):1023–1031. doi:10.1016/j.clinph.2010.01.034
   PubMed Web of Science ®Google Scholar
• Käthner I, Ruf CA, Pasqualotto E, et al. A portable auditory P300 brain–computer interface with directional cues. Clin Neurophysiol. 2013;124(2):327–338. doi:10.1016/j.clinph.2012.08.006
   PubMed Web of Science ®Google Scholar
• Fried-Oken M, Mooney A, Peters B, et al. A clinical screening protocol for the RSVP keyboard brain–computer interface. Disabil Rehabil Assist Technol. 2015;10(1):11–18. doi:10.3109/17483107.2013.836684
   PubMed Web of Science ®Google Scholar
• Zander TO, Ihme K, Gärtner M, et al. A public data hub for benchmarking common brain–computer interface algorithms. J Neural Eng. 2011;8(2):025021.10.1088/1741-2560/8/2/025021
   PubMed Web of Science ®Google Scholar
• Ledesma-Ramirez C, Bojorges-Valdez E, Yáñez-Suarez O, et al. An open-access P300 speller database. Paper presented at: Fourth International Brain-Computer Interface Meeting; 2010; Monterey, USA; 2010. Available from: https://hal.inria.fr/inria-00549242
   Google Scholar
• Bernabeu M, Laxe S, Lopez R, et al. Developing core sets for persons with traumatic brain injury based on the international classification of functioning, disability, and health. Neurorehab Neural Re. 2009;23(5):464–467.10.1177/1545968308328725
   PubMed Web of Science ®Google Scholar
• Schwarzkopf SR, Ewert T, Dreinhofer KE, et al. Towards an ICF core set for chronic musculoskeletal conditions: commonalities across ICF sets for osteoarthritis, rheumatoid arthritis, osteoporosis, low back pain and chronic widespread pain. Chin Rheumatol. 2008;27:1355–1361.
   Web of Science ®Google Scholar
• Cieza A, Ewer T, Ustun B, et al. Development of ICF core sets for patients with chronic conditions. J Rehabil Med. 2004; Suppl.:9–11.
   PubMed Web of Science ®Google Scholar
• Peters B, Mooney A, Oken B, et al. Soliciting BCI user experience feedback from people with severe speech and physical impairments. Brain Computer Interfaces. 2016;3:47–58.doi:10.1080/2326263X.2015.1138056.
   Web of Science ®Google Scholar
• Scherer R, Wagner J, Billinger M, et al. Augmenting communication, emotion expression and interaction capabilities of individuals with cerebral palsy. Paper presented at: 2014 6th International Brain-Computer Interface Conference; 2014; Graz, Austria. p. 312–315. doi:10.3217/978-3-85125-378-8-78
   Google Scholar
• Daly I, Billinger M, Laparra-Hernández J, et al. On the control of brain-computer interfaces by users with cerebral palsy. Clin Neurophysiol. 2013;124(9):1787–1797. doi:10.1016/j.clinph.2013.02.118
   PubMed Web of Science ®Google Scholar
• Scherer R, Billinger M, Wagner J, et al. Thought-based row-column scanning communication board for individuals with cerebral palsy. Ann Phys Rehabil Med. 2015;58(1):14–22. doi:10.1016/j.rehab.2014.11.005
   PubMed Web of Science ®Google Scholar
• Scherer R, Schwarz A, Müller-Putz GR, et al. Game-based BCI training: interactive design for individuals with cerebral palsy. Paper presented at: 2015 IEEE International Conference on Systems, Man, and Cybernetics (SMC); 2015 Oct; Hong-Kong; 2015a; p. 3175–3180.
   Google Scholar
• Scherer R, Schwarz A, Müller-Putz GR, et al. Let’s play Tic-Tac-Toe: a brain-computer interface case study in cerebral palsy. Paper presented at: 2016 IEEE International Conference on Systems, Man and Cybernetics (SMC); 2016 Oct 9-12; Budapest, Hungary.
   Google Scholar
• Neumann N, Kubler A. Training locked-in patients: a challenge for the use of brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng. 2003;11(2):169–172. doi:10.1109/TNSRE.2003.814431
   PubMed Web of Science ®Google Scholar
• Neuper C, Pfurtscheller G. Neurofeedback Training for BCI Control Brain-Computer Interfaces: Revolutionizing Human-Computer Interaction. In: Graimann B, Pfurtscheller G, Allison B, editors. Frontiers Collection. Springer Berlin Heidelberg; 2010:65–78.
   Google Scholar
• Wander JD, Blakely T, Miller KJ, et al. Distributed cortical adaptation during learning of a brain-computer interface task. Proc Nat Acad Sci. 2013;110(26):10818–10823.10.1073/pnas.1221127110
   PubMed Web of Science ®Google Scholar
• Neuper C, Scherer R, Reiner M, et al. Imagery of motor actions: differential effects of kinesthetic and visual–motor mode of imagery in single-trial EEG. Cognitive Brain Res. 2005;25(3):668–677. doi:10.1016/j.cogbrainres.2005.08.014
   PubMedGoogle Scholar
• Lotte F, Jeunet C. Towards improved BCI based on human learning principles. Paper presented at: 3rd International Brain-Computer Interfaces Winter Conference; 2015.
   Google Scholar
• Shute V. Focus on formative feedback. Rev Educ Res. 2008;78(1):153–189.10.3102/0034654307313795
   Web of Science ®Google Scholar
• Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81–112.10.3102/003465430298487
   Web of Science ®Google Scholar
• Merrill M. First principles of instruction: a synthesis. In: Reiser RA, Dempsey JV, editor. Trends and issues in instructional design and technology. 2nd ed. Upper Saddle River, NJ: Merrill/Prentice Hall; 2007. p. 62–71.
   Google Scholar
• Jeunet C, Jahanpour E, Lotte F. Why standard brain-computer interface (BCI) training protocols should be changed: an experimental study. J Neural Eng. 2016;13(3):036024.10.1088/1741-2560/13/3/036024
   PubMed Web of Science ®Google Scholar
• Jeunet C, N’Kaoua B, N’Kambou R, et al. Why and how to use intelligent tutoring systems to adapt MI-BCI training to each user? Paper presented at: International BCI meeting; 2016; Monterey, USA.
   Google Scholar
• Jeunet C, N’Kaoua B, Subramanian S, et al. Predicting mental imagery-based BCI performance from personality, cognitive profile and neurophysiological patterns. PLoS ONE. 2015;10(12):e0143962.10.1371/journal.pone.0143962
   PubMed Web of Science ®Google Scholar
• Kleih SC, Kübler A. Psychological factors influencing brain-computer interface (BCI) performance. Paper presented at: 2015 IEEE International Conference on Systems, Man, and Cybernetics (SMC); 2015; Hong-Kong. p. 3192–3196.
   Google Scholar
• Jeunet C, Vi C, Spelmezan D, et al. Continuous tactile feedback for motor-imagery based brain-computer interaction in a multitasking context. Paper presented at Proc. Interact; 2015. p. 2015.
   Google Scholar
• Gomez Rodriguez M, Peters J, Hill J, et al. Closing the sensorimotor loop: haptic feedback helps decoding of motor imagery. J Neural Eng. 2011;8:036005.10.1088/1741-2560/8/3/036005
   PubMed Web of Science ®Google Scholar
• Sollfrank T, Ramsay A, Perdikis S, et al. The effect of multimodal and enriched feedback on SMR-BCI performance. Clin Neurophysiol. 2016;127:490–498.10.1016/j.clinph.2015.06.004
   PubMed Web of Science ®Google Scholar
• Lotte F, Faller J, Guger C, et al., editors. Combining BCI with virtual reality: towards new applications and improved BCI towards practical brain-computer interfaces. Berlin Heidelberg: Springer; 2013. p. 197–220.
   Google Scholar
• Birbaumer N, Ghanayim N, Hinterberger T, et al. A spelling device for the paralysed. Nature. 1999;398(6725):297–298. doi:10.1038/18581
   PubMed Web of Science ®Google Scholar
• Blankertz B, Dornhege G, Krauledat M, et al. The non-invasive berlin brain-computer interface: fast acquisition of effective performance in untrained subjects. NeuroImage. 2007;37(2):539–550. doi:10.1016/j.neuroimage.2007.01.051
   PubMed Web of Science ®Google Scholar
• Mason SG, Bashashati A, Fatourechi M, et al. A comprehensive survey of brain interface technology designs. Ann Biomed Eng. 2007;35(2):137–169. doi:10.1007/s10439-006-9170-0
   PubMed Web of Science ®Google Scholar
• Vidaurre C, Kawanabe M, von Bünau P, et al. Toward unsupervised adaptation of LDA for brain-computer interfaces. IEEE Trans Biomed Eng. 2011;58(3):587–597. doi:10.1109/TBME.2010.2093133
   PubMed Web of Science ®Google Scholar
• Faller J, Vidaurre C, Solis-Escalante T, et al. Autocalibration and recurrent adaptation: towards a plug and play online ERD-BCI. IEEE Trans Neural Syst Rehabil Eng. 2012;20(3):313–319. doi:10.1109/TNSRE.2012.2189584
   PubMed Web of Science ®Google Scholar
• Faller J, Scherer R, Costa U, et al. A co-adaptive brain-computer interface for end users with severe motor impairment. PLoS ONE. 2014;9(7):e101168. doi:10.1371/journal.pone.0101168
   PubMed Web of Science ®Google Scholar
• Samek W, Meinecke FC, Muller KR. Transferring subspaces between subjects in brain--computer interfacing. IEEE Trans Biomed Eng. 2013;60(8):2289–2298. doi:10.1109/TBME.2013.2253608
   PubMed Web of Science ®Google Scholar
• Perdikis S, Leeb R, Millán JDR. Subject-oriented training for motor imagery brain-computer interfaces. Paper presented at: 2014 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC’14; 2014. p. 1259–1262.
   Google Scholar
• Kobler R, Scherer R. Restricted Boltzmann machines in sensory motor rhythm brain-computer interfacing: a study on inter-subject transfer and co-adaptation. Paper presented at: 2016 IEEE International Conference on Systems, Man and Cybernetics (SMC); 2016 Oct 9-12; Budapest, Hungary.
   Google Scholar
• Cecotti H. A self-paced and calibration-less SSVEP-based brain–computer interface speller. IEEE Trans Neural Syst Rehabil Eng. 2010;18(2):127–133.10.1109/TNSRE.2009.2039594
   PubMed Web of Science ®Google Scholar
• Kindermans PJ, Schreuder M, Schrauwen B, et al. True zero-training brain-computer interfacing – an online study. PLoS ONE. 2014;9(7):e102504.10.1371/journal.pone.0102504
   PubMed Web of Science ®Google Scholar
• Baykara E, Ruf CA, Fioravanti C, et al. Effects of training and motivation on auditory P300 brain–computer interface performance. Clin Neurophysiol. 2016;127(1):379–387.10.1016/j.clinph.2015.04.054
   PubMed Web of Science ®Google Scholar
• Hastie T, Tibshirani R, Friedman J. The elements of statistical learning. 2nd ed. New York, NY: Springer-Verlag; 2009. doi:10.1007/978-0-387-84858-7
   Google Scholar
• Ledoit O, Wolf M. Honey, i shrunk the sample covariance matrix. J Port Manage. 2004;30(4):110–119.10.3905/jpm.2004.110
   Web of Science ®Google Scholar
• Lemm S, Blankertz B, Dickhaus T, et al. Introduction to machine learning for brain imaging. NeuroImage. 2011;56:387–399.10.1016/j.neuroimage.2010.11.004
   PubMed Web of Science ®Google Scholar
• Lotte F. Signal processing approaches to minimize or suppress calibration time in oscillatory activity-based brain-computer interfaces. Proc IEEE. 2015;103(6):871–890.10.1109/JPROC.2015.2404941
   Web of Science ®Google Scholar
• Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Mag. 2008;25:41–56. doi:10.1109/MSP.2008.4408441
   Web of Science ®Google Scholar
• Xu R, Jiang N, Mrachacz-Kersting N, et al. Factors of influence on the performance of a short-latency non-invasive brain switch: evidence in healthy individuals and implication for motor function rehabilitation. Front Neurosci. 2015;9.
   Web of Science ®Google Scholar
• Lotte F. Guide to brain-computer music interfacing. London: In Guide to Brain-Computer Music Interfacing. Springer; 2014. p. 133–161.10.1007/978-1-4471-6584-2
   Google Scholar
• Ramoser H, Muller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehab Eng. 2000;8:441–446. doi:10.1109/86.895946
   PubMedGoogle Scholar
• Samek W, Kawanabe M, Muller K-R. Divergence-based framework for common spatial patterns algorithms. IEEE Rev Biomed Eng. 2013;2014(7):50–72.
   Google Scholar
• Seeber M, Scherer R, Wagner J, et al. EEG beta suppression and low gamma modulation are different elements of human upright walking. Front Hum Neurosci. 2014;8:485. doi:10.3389/fnhum.2014.00485
   PubMed Web of Science ®Google Scholar
• Seeber M, Scherer R, Wagner J, et al. High and low gamma EEG oscillations in central sensorimotor areas are conversely modulated during the human gait cycle. NeuroImage. 2015;112:318–326. doi:10.1016/j.neuroimage.2015.03.045
   PubMed Web of Science ®Google Scholar
• Friedrich EVC, Scherer R, Neuper C. The effect of distinct mental strategies on classification performance for brain-computer interfaces. Int J Psychophysiol. 2012;84(1):86–94. doi:10.1016/j.ijpsycho.2012.01.014
   PubMed Web of Science ®Google Scholar
• Friedrich EVC, Neuper C, Scherer R. Whatever works: a systematic user-centered training protocol to optimize brain-computer interfacing individually. PLoS ONE. 2013;8(9):e76214. doi:10.1371/journal.pone.0076214
   PubMed Web of Science ®Google Scholar
• Scherer R, Faller J, Friedrich EVC, et al. Individually adapted imagery improves brain-computer interface performance in end-users with disability. PLoS ONE. 2015b;10(5):e0123727. doi:10.1371/journal.pone.0123727
   PubMed Web of Science ®Google Scholar
• Scherer R, Faller J, Opisso E, et al. Bring mental activity into action! self-tuning brain-computer interfaces. Paper presented at: 2015 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC’15; 2015c.
   Google Scholar
• Fatourechi M, Bashashati A, Ward RK, et al. EMG and EOG artifacts in brain computer interface systems: a survey. Clin Neurophysiol. 2007;118(3):480–494. doi:10.1016/j.clinph.2006.10.019
   PubMed Web of Science ®Google Scholar
• Schlögl A, Keinrath C, Zimmermann D, et al. A fully automated correction method of EOG artifacts in EEG recordings. Clin Neurophysiol. 2007;118(1):98–104. doi:10.1016/j.clinph.2006.09.003
   PubMed Web of Science ®Google Scholar
• Daly I, Scherer R, Billinger M, et al. FORCe: fully online and automated artifact removal for brain-computer interfacing. IEEE Trans Neural Syst Rehabil Eng. 2015;23(5):725–736.10.1109/TNSRE.2014.2346621
   PubMed Web of Science ®Google Scholar
• Pfurtscheller G, Allison B, Bauernfeind G, et al. The hybrid BCI. Front Neurosci. 2010;4:42.
   Web of Science ®Google Scholar
• Leeb R, Sagha H, Chavarriaga R, et al. A hybrid brain-computer interface based on the fusion of electroencephalographic and electromyographic activities. J Neural Eng. 2011;8:025011.10.1088/1741-2560/8/2/025011
   PubMed Web of Science ®Google Scholar
• Muller-Putz G, Leeb R, Tangermann M, et al. Towards noninvasive hybrid brain-computer interfaces: framework, practice, clinical application, and beyond. Proc IEEE. 2015;103:926–943.10.1109/JPROC.2015.2411333
   Web of Science ®Google Scholar
• Urigüen JA, Garcia-Zapirain B. EEG artifact removal-state-of-the-art and guidelines. J Neural Eng. 2015;12:031001.10.1088/1741-2560/12/3/031001
   PubMed Web of Science ®Google Scholar
• Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett. 2006;27:861–874.10.1016/j.patrec.2005.10.010
   Web of Science ®Google Scholar
• Wolpaw JR, Birbaumer N, Heetderks WJ, et al. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehab Eng. 2000;8:164–173.
   PubMedGoogle Scholar
• Quitadamo LR, Abbafati M, Cardarilli GC, et al. Evaluation of the performances of different P300 based brain-computer interfaces by means of the efficiency metric. J Neurosci Methods. 2012;203:361–368.10.1016/j.jneumeth.2011.10.010
   PubMed Web of Science ®Google Scholar
• Thomas E, Dyson M, Clerc M. An analysis of performance evaluation for motor-imagery based BCI. J Neural Eng. 2013;10:031001.10.1088/1741-2560/10/3/031001
   PubMed Web of Science ®Google Scholar
• Antelis JM, Montesano L, Ramos-Murguialday A, et al. On the usage of linear regression models to reconstruct limb kinematics from low frequency EEG signals. PLoS ONE. 2013;8:e61976.10.1371/journal.pone.0061976
   PubMed Web of Science ®Google Scholar
• Spüler M, Sarasola-Sanz A, Birbaumer N, et al. Comparing metrics to evaluate performance of regression methods for decoding of neural signals. In: Conf Proc IEEE Eng Med Biol Soc, 2015; 2015; Milano (Italy). p. 1083–1086.
   Google Scholar
• Müller-Putz G, Scherer R, Brunner C, et al. Better than random: a closer look on BCI results. Int J Bioelectromagn. 2008;10:52–55.
   Google Scholar
• Maris E, Oostenveld R. Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods. 2007;164:177–190.10.1016/j.jneumeth.2007.03.024
   PubMed Web of Science ®Google Scholar
• Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat Taylor & Francis. 2016;70(2):129–133.
   Web of Science ®Google Scholar
• Nijboer F, Birbaumer N, Kübler A. The influence of psychological state and motivation on brain-computer interface performance in patients with amyotrophic lateral sclerosis - a longitudinal study. Front Neurosci. 2010;4:55.
   PubMed Web of Science ®Google Scholar
• McCane LM, Sellers EW, McFarland DJ, et al. Brain-computer interface (BCI) evaluation in people with amyotrophic lateral sclerosis. Amyotroph Lateral Scler Frontotemporal Degener. 2014;15(3-4):207–215.10.3109/21678421.2013.865750
   PubMed Web of Science ®Google Scholar
• Schlögl A, Kronegg J, Huggins JE, et al. Evaluation criteria for BCI research. In: Dornhege G, Millán JDR, Hinterberger T. et al, editor. Toward brain-computer interfacing. Cambridge, MA: MIT press; 2007. p. 373–391.
   Google Scholar
• Dal Seno B, Matteucci M, Mainardi LT. The utility metric: a novel method to assess the overall performance of discrete brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng. 2010;18(1):20–28.10.1109/TNSRE.2009.2032642
   PubMed Web of Science ®Google Scholar
• Hill NJ, Häuser AK, Schalk G. A general method for assessing brain–computer interface performance and its limitations. J Neural Eng. 2014;11(2):026018.10.1088/1741-2560/11/2/026018
   PubMed Web of Science ®Google Scholar
• Riccio A, Leotta F, Bianchi L, et al. Workload measurement in a communication application operated through a P300-based brain–computer interface. J Neural Eng. 2011;8(2):025028.10.1088/1741-2560/8/2/025028
   PubMed Web of Science ®Google Scholar
• Hart S, Staveland L. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock P, Meshkati N, editors. Human Mental Workload. North-Holland: Human Mental Workload; 1988. p. 139–183.
   Google Scholar
• Nuzzo R. How scientists fool themselves - and how they can stop. Nature. 2015;526:182–185.10.1038/526182a
   PubMed Web of Science ®Google Scholar
• Brouwer AM, Zander TO, Van Erp JB, et al. Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls. Front Neurosci. 2015;9:136.
   PubMed Web of Science ®Google Scholar
• Duncan CC, Barry RJ, Connolly JF, et al. Event-related potentials in clinical research: guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400. Clin Neurophysiol. 2009;120:1883–1908.10.1016/j.clinph.2009.07.045
   PubMed Web of Science ®Google Scholar
• Kass RE, Caffo BS, Davidian M, et al. Ten simple rules for effective statistical practice. PLoS Comput Biol. 2016;12:e1004961.10.1371/journal.pcbi.1004961
   PubMed Web of Science ®Google Scholar
• Scherer MJ, Craddock G. Matching person & technology (MPT) assessment process. Technol Disabil. 2002;14(3):125–131.
   Google Scholar
• Scherer M, Jutai J, Fuhrer M, et al. A framework for modelling the selection of assistive technology devices (ATDs). Disabil Rehabil Assist Techno. 2007;2(1):1–8.10.1080/17483100600845414
   PubMedGoogle Scholar