A bimodal registration and attention method for speed imagery brain-computer interface

Abstract

Brain-computer interface (BCI) has received attention from researchers in many research fields as an emerging control technology that translates human thoughts into actions. Existing motor imagery (MI)-based BCI systems can only decode a limited number of neural intentions, thus limiting the scope of BCI applications. We propose a speed imagery (SI)-BCI paradigm, which aims to decode spontaneous SI intentions. Thus, the number of decodable intentions is increased by using the natural continuity of the physical quantity of speed. We further build a synchronous bimodal acquisition system of spontaneous SI intentions, which is capable of acquiring EEG signals and functional near-infrared spectroscopy (fNIRS) signals simultaneously. Specifically, an interpretable bimodal signal registration and attention algorithm, called STformer, is proposed for SI classification, which consists of two parts: 1) a bimodal registration algorithm for signal fusion that improves the tightness of spatio-temporal coupling of EEG and fNIRS signals. 2) a dual-path spatio-temporal feature extraction and global attention network that makes full use of bimodal spatio-temporal features for SI intention classification. Experimental results on two datasets show that the proposed SI-BCI system outperforms state-of-the-art methods in terms of data reliability, classification performance and interpretability.

Keywords:

• Speed imagery
• brain-computer interface
• electroencephalography
• functional near-infrared spectroscopy and bimodal signal registration

Disclosure statement

The authors report there are no competing interests to declare.

Additional information

Funding

This work was supported by the National Natural Science Foundation of China under Grant T2322020, 61971303 and 62371329.

Notes on contributors

Zhengkun Liu

Zhengkun Liu received the B.Eng. degree in automation from the School of Electrical and Information Engineering, Tianjin University, Tianjin, China, in 2021. He is currently working toward the master’s degree with the School of Electrical and Information Engineering, Tianjin University, Tianjin, China. His research interests include brain computer interface, spiking neural network and neural architecture search.

Xiaoqian Hao

Xiaoqian Hao received the B.Eng. degree in Yanshan University, Qinhuangdao, China, in 2020. She is currently working toward the master’s degree with the School of Electrical and Information Engineering, Tianjin University, Tianjin, China. Her research interests include brain-computer interface and deep learning.

Tengyu Wu

Tengyu Wu is currently an undergraduate student at the School of Electrical and Information Engineering, Tianjin University, Tianjin, China. His research focuses on the design of fNIRS systems and deep learning.

Guchuan Wang

Guchuan Wang is currently an undergraduate student at the School of Electrical and Information Engineering, Tianjin University, Tianjin, China. His research interests lie in the realm of circuit and systems design.

Yong Li

Yong Li, born in Tianjin in 1981, is currently a Deputy Chief Physician and Master’s Supervisor in the Department of Interventional Oncology at Tianjin Medical University Cancer Hospital, Tianjin, China. His main research interests include oncology, interventional oncology, and automation of minimally invasive cancer treatments.

Biao Sun

Biao Sun received the Diploma degree in electrical information science and technology from Central South University, Changsha, China, in 2004, and the Ph.D. degree in electrical science and technology from Huazhong University of Science and Technology, Wuhan, China, in 2013.From 2015 to 2016, he was a Visiting Research Fellow with the Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore. He is currently an Associate Professor with the School of Electrical and Information Engineering, Tianjin University, Tianjin, China. His research interests include compressed sensing, machine learning, and brain–computer interface.