Feasibility of decoding visual information from EEG

ABSTRACT

Decoding visual information, such as visual imagery and perception, from EEG data can be used to improve understanding of the neural representation of visual information and to provide commands for BCI systems. The appeal of EEG as a neuroimaging tool lies in its high temporal resolution, cost-effectiveness, and portability. Nevertheless, the feasibility of using EEG for visual information decoding remains a subject of ongoing inquiry. In this review, we explore the neural correlates of this visual information, specifically focusing on visual features such as colour, shapes, texture, and also naturalistic whole objects. We begin to examine which visual features can be effectively measured using EEG, taking into account its inherent characteristics, such as its measurement depth, limited spatial resolution, and high temporal resolution. Using a systematic approach, the review provides an in-depth analysis of the current state-of-the-art in EEG-based decoding of visual features for BCI purposes. Finally, we address some potential methodological improvements that can be made to the experimental design in EEG visual information decoding studies, such as palette cleansing, augmentation to bolster dataset size, and fusion of neuroimaging techniques.

KEYWORDS:

• Electroencephalography
• visual imagery
• visual perception
• classification
• machine learning
• neural decoding
• brain computer interface
• review

1. The question of feasibility

In recent years, advances have been made in decoding visual imagery and perception for Brain–Computer Interface (BCI) inputs, using a range of neuroimaging techniques, such as functional Magnetic Resonance Imaging (fMRI). The exploration of Electroencephalography (EEG) for decoding visual information has yielded mixed results, despite its appealing attributes and advantages. EEG, as a noninvasive alternative to intracranial electroencephalography (iEEG), offers higher portability, superior temporal resolution, and a more cost-effective solution compared to fMRI and functional Near-Infrared Spectroscopy (fNIRS). However, there are also concerns about EEG’s suitability for decoding visual information due to its limited spatial resolution and restricted signal acquisition from deep brain regions. This review examines the current state of, and potential for decoding visual information using EEG. Notably, significant progress and growing interest in this area have emerged since the last review in 2015, which combined EEG with fMRI [1]. Thus, this review is particularly timely for researchers intrigued by the prospect of decoding visual information from EEG data.

We begin in Section 2 by introducing visual information. Section 3 outlines the motivations behind decoding visual information via EEG, while Section 3.2 explores the visual processing in the brain. Following this, we investigate the current state-of-the-art in EEG-based decoding of visual information. We consider the viability of EEG for decoding different aspects of visual information divided into simple types of visual information in Section 5 such as colour, texture, shape and position, whilst Section 6 addresses complex decoding. In both sections, we assess the quality of existing studies and datasets, pinpointing key neural regions and time-points evidenced to be usable for decoding specific types of visual information. This information aids in optimal channel placement and temporal window selection. By collating and reviewing widely used EEG visual information decoding datasets, our goal is to assist researchers in making informed choices about dataset selection and determining when generating new datasets might be necessary. As creating new datasets can be time-consuming and expensive, making use of existing resources is a valuable option. Section 7 considers constraints and advantages of the EEG hardware itself, such as the spatial and temporal resolution, practicality, cost, and safety. Whilst Section 8 and 9 shift the focus to challenges related to data gathering and explores potential strategies to enhance data quality and, consequently, improve decoding performance.

2. What is visual information

Visual information includes visual properties such as colour, shape, orientation, size, contrast, luminance, spatial location, movement, and texture. These properties can be experienced via visual perception and imagination.

Visual perception involves receiving and then interpreting visual information in the surrounding environment such as the vibrant orange and red hues of a sunset, or the roundness of a beach ball, when they are within sight. This process is largely, though not exclusively driven by external input. Perception is also impacted by beliefs and past experiences [2]. Visual imagery, is referred to in the literature as visual mental imagery, visual imagination, or the mind’s eye. Visual imagery is the capacity to evoke the appearance of things such as the sunset or beach ball, in their absence – meaning the object is not in the line of the visual field [3,4], or rather not seeable. This internally generated appearance should include some of the sensory visual characteristics of the object, such as shape, colour, and size, and is considered to be a conscious visual experience [5]. Though mental imagery is sometimes used to refer specifically to visual imagination in the literature, we regard mental imagery as encompassing other sensory modalities such as inner-speech, imagined taste, touch and also imagined movement, i.e. motor imagery.

We note that event-related potentials (ERP) are a response of the brain, to visual information, which can be captured with EEG. ERPs have a long history of successful use as inputs for BCI inputs [6] and for assessment of cognitive [7], sensory and perceptual capacities Woodman [8]. For example, visual-P300 is a component which occurs approximately 300 ms after presentation of a visual stimulus, with an increased amplitude for surprising stimuli [9], and has been used in spelling-based BCIs [10,11]. Visually evoked potentials are a subtype of ERPs. Steady stage visually evoked potential (SSVEPs) are brain signals that occur in response to a visual stimulus flickering at a fixed frequency, and are also used as BCI input [10,11]. Whilst SSVEP and visual-P300 are visual, they relate less to visual and semantic properties of the visual stimulus such as colours, shapes, and category. For this reason, they are out of the scope of this review which aims to focus on the aforementioned visual properties. Further, though machine learning is an important part of the decoding pipeline, and innovations in which may boost decoding performance, this review does this review does not cover the specific algorithms used in EEG visual information decoding literature and more generally. This is because there is a wealth of review papers dedicated to this, for example, Aggarwal and Chugh [12]; Cao [13]; Roy et al. [14]; Xu et al. [15].

3. Decoding visual information from EEG

3.1. Why decode visual information

There are many motives for researchers to decode visual imagery and perception from neural activity, such as 1) for inputs to BCIs as mentioned previously, which can have potential applications in communication [16] and gaming [17] 2) studying neural dynamics and the mechanisms of visual imagery and perception which can provide insights into disorders related to imagery, such as post-traumatic stress disorder (PTSD) and schizophrenia, within clinical psychology [18], and 3) on a philosophical level, this search can help externalize internal processes and address questions about the nature of thought. Neural decoding can use classification and reconstruction to retrieve stimulus content [19,20]; see Figure 1. Classification involves the prediction of the class a stimulus belonged to given some brain recording data. For example, was the person perceiving an image of an animal or a tool? Reconstruction entails generating a pictorial representation of the perceived animal picture, based on the brain activity. On the other hand, with feature extraction, a set of features such as colour or shape that capture important stimuli characteristics may be obtained from the brain activity.

Figure 1. Classic pipeline for decoding visual information. First the neural signal is measured whilst the individual performs a visual imagery or perception task. This neural signal is then decoded. The output contents include an example of pictorial reconstruction (a), semantic classification (b), and identification of low level visual information such as shape (c) and colour (d).

3.1.1. Visual based BCIs

BCIs as mentioned in Section 2, can be driven by visual information, whether by visual input into the sensory system or internal generation of visual content. BCI research has predominantly focused on neural decoding of VEPs, the P300 component, motor movement, and motor imagery [21], and more recently on speech decoding [22]. In contrast, decoding visual pictorial-level content for BCI use is relatively unexplored. While there are a few studies comparing which mental states users prefer [23,24], the BCI field lacks research directly comparing the three modalities, auditory, visual, and motor, for information transmission and user experience. Information transmission quality can include the amount of information conveyed, the information redundancy, accuracy and stability of the command, and speed of information transmission. User experience can include comfort, the cognitive load of processing and generating the information, and how intuitive the modality is for a given task. Visual and auditory commands have some other advantages over motor commands, providing an alternative option in cases where motor command is not appropriate. Visual and auditory commands may also enable individuals to communicate a more semantically rich variety of information, not restricted to the motor domain.

There may be benefits to using the visual modality as input over auditory. Many BCIs focus on simple auditory properties such as auditory event-related potentials [25]. Though, here we take auditory commands to refer to inner-speech, as this is an increasingly explored input for BCIs in the auditory domain [26–28]. It is said that a ‘picture paints a thousand words’. In the context of BCI usage, this claim requires empirical validation. However, it is feasible that for BCI usage imagining or perceiving a single picture may very rapidly convey rich information. For example, basic dissociation of scene categories from visual perception seems achievable from just the first $\sim$100 ms of EEG data [29]. This would likely be faster than the auditory presentation and comprehension of a sentence describing this same picture. For instance, seeing a picture of ”dark green trees in front of a blue glacier and a snow-capped mountain” will be quicker than internally speaking or hearing this sentence, which take $\sim$4000 ms (see Figure 2). Thus, a BCI system driven by visual perception may enable fast identification of, and response to, an individual’s input. Though of course, the ideal system would be driven by visual imagery rather than visual perception, it is possible that the internal generation speed of visual imagery varies substantially depending on the stimulus complexity, and also between individuals. Indeed, it has been demonstrated that individuals exhibit varying dominance in sensory modalities for mental imagery, with auditory and visual modalities generally the more dominant [30]. Research directly comparing the two would make for interesting future work. For now, such individual differences should be noted when selecting the sensory modality for BCI input commands. We discuss these individual differences in more depth in Section 8.1. A disadvantage of visually driven BCIs is that good vision is required – therefore potentially excluding those with vision impairments.

Figure 2. This demonstrates the relative speed taken to process similar information content in either a. visual or b. auditory modalities. Object recognition, such as this glacier scene, can be detected as early as 100ms from visual perception, figure taken and adapted from Lowe et al. [29]. How long does it take to say the sentence out loud in B?

3.1.2. Understanding neural mechanisms

The bulk of the visual information decoding literature is motivated by deepening our understanding of neural dynamics and the mechanisms behind visual imagery and perception. A pivotal contribution of visual-decoding research has been on ending the debate as to whether mental imagery is propositional (symbolic and language based) or depictive (i.e pictorial) – visual mental images involve depictive representations [31], as visual information can be decoded from V1, also known as the primary visual cortex, activity. Whilst it seems there is phenomenal depictive similarity between perception and imagery, another goal has been to understand the extent that the two overlap in their spatiotemporal dynamics. Visual decoding in this context has been applied by cross-decoding, i.e. training a classifier on perception neural data then testing it on imagination data and vice versa. This cross-decoding has revealed that perception and imagery show similar top-down activity in visual and frontoparietal regions of the brain [32]. Further by applying the classifier to different time points of MEG data, it shows that imagery and perception most overlap in time-windows related to high-level processing. The authors used this finding to show support that these two processes rely on the same prediction processes. A further motivation of visual decoding is to externalize internal subjective experiences, such as dreams and visual imagery experienced during waking, and learn more about their respective contents and their neural representation. One pivotal study trying to decode visual contents of dreams from fMRI activity [33]. They trained decoders with brain activity obtained during waking in which participants saw images labeled with feature values extracted from a deep convolutional neural network. These decoders were then applied to brain activity captured whilst dreaming, which revealed that dreams have a hierarchical visual representation, not just categorical. Additional future motivations may include gaining insight into disorders related to imagery, such as PTSD and schizophrenia [18] as a way to externalize related hallucinations or intrusive visual imagery thoughts, or another case is investigating how psychedelics modulate visual imagery experience by examining the spatiotemporal dynamics and how these relate to vividness, change in altered conscious experience and to the imagined content itself.

3.2. How is visual information represented in the brain

Here, we make the distinction between spatial-temporal dynamics that are related to the processing of visual information, and the neural correlates which actually encode the visual information. We can group stimuli into two levels when decoding – low and high level information. Some researchers focus on one level exclusively, others combine both, or rather do not consider a distinction. In this context, low-level refers to pictorial qualities such as colour, shape, edges and orientation, whereas high-level relates to complex compositions and eventually semantic representation, for example whether the stimulus can be labeled as a ’dog’ or a ’cat’. This high level information can be invariant to the low level pictorial details, and in some cases invariant across sensory modalities, such as in work by Man et al. [34]; Viganò et al. [35]. We provide a brief overview to introduce brain regions discussed later within the wider context of visual processing dynamics.

3.2.1. Visual processing

In visual perception, light passes into the eye to produce a reversed image of the visual stimulus on the retina. The light-sensitive receptors in the retina, rods, and cones, then convert this light into electrical signals. This visual information is relayed to the visual cortex, part of the occipital lobe, at the back of the brain, via the lateral geniculate nucleus (LGN) in the dorsal thalamus [36]. The visual cortex is composed of six core regions, the V1, V2, V3, V4, V5 and inferotemporal cortex – these can considered hierarchical in the complexity of information they process (see Figure 3 for a schematic of some key regions related to visual processing). The LGN sends visual information first to V1, also known as the primary visual cortex, which then is passed through to V2, the secondary visual cortex, and so forth. The visual system can be divided into a ventral stream for object visual information, and a dorsal stream for spatial visual information [37]. Visual perception is not exclusively driven by external input, as it entails both top-down and bottom-up processing, and is demonstrated to evoke recurrent processing [38]. In contrast, visual imagery involves predominantly top-down processing. The subjective vividness of visual imagery is evidenced to be modulated by the strength of coupling from intraparietal sulcus to the early visual cortex [39]. Both imagery and perception overlap in the neural regions involved, namely the visual cortex and parietal and premotor/frontal areas [3,40], yet, as mentioned, they have distinct temporal dynamics [41]. Still, not all areas involved in visual processing encode visual information relevant to the stimulus-specific properties. For example, subcortical regions are often content invariant. Levinson et al. [42] found subcortical regions such as the basal ganglia and brain stem to be involved in visual processing but not to encode detectable category information. This may be due to the sensitivity of neuroimaging techniques, though in this specific case, 7 Tesla fMRI was used, which has high spatial resolution.

Figure 3. This demonstrates key regions related to visual processing, such as V1–3 and the lateral geniculate nucleus.This image is adapted and taken from BodyParts3D, © the Database center for Life Science licensed under CC attribution-share alike 2.1 Japan.

4. Review methodology

Here, we outline our approach for the following review of the literature for EEG-based decoding of simple and complex stimuli. Google Scholar and PubMed search engines were used to find papers. The targets for simple stimuli included shapes, colour and texture; whereas, the targets for complex stimuli included naturalistic compositions such as faces, scenes, places, and characters (see Table 1 for the keyword variations). Note, that predominantly only peer-reviewed work is included, however exceptions were made where a non-peer reviewed paper has been foundational in the field, with many citations. We have chosen to include these as we are evaluating the current state-of-the-art, and such papers have already substantially influenced the field.

Table 1. Key words used to identify papers included in the following literature review for each target.

Target	Key word variations
BCI	BCI; brain computer interface; brain-computer interface; BMI; brain-machine interace; brain machine interface
EEG	eeg; electroencephalography; electroencephalogram
visual imagery	visual imagery; mental imagery; visual imagination; mind’s eye; imagined picture(s); picture imagery; mental picture(s)
visual perception	visual perception; seeing; vision; looking;
shape	shape(s); orientation(s); location(s); position(s); geometric; size(s); spatial
colour	colour(s); colour(s); RGB; hue(s); colourimetry
texture	texture(s);
naturalistic	naturalistic; natural; complex; scene(s); face(s); place(s); object(s); digit(s); letter(s); character(s)

Information transfer rate (ITR) is a metric used to measure the amount of information in a set period of time, in bits. This metrics is useful when comparing across studies, as it takes into account the number of classes and the time window used to calculate the decoding accuracy. ITR was defined in 1998 by Wolpaw et al. [43], and is as follows in equation 1:

(1) $B=lo{g}_{2}N+Plo{g}_{2}P+(1-P)lo{g}_2\frac{1-P}{N-1}$(1)

B is the bit rate ITR. N is the number of classes. P is the decoding accuracy. To obtain the BCI ITR, time in seconds (T) is incorporated in equation 2:

(2) $ITR=Bx\frac{60}{T}$(2)

Where sufficient information is provided in the decoding papers, we include the ITR alongside the reported decoding accuracy.

5. Simple visual features feasibility

Simple visual features refers to visual properties such as colour, shape, texture, orientation, and position, which can be considered as the building blocks of vision. Visual perception and imagination decoding of simple features via EEG mostly focus on colour decoding which we discuss first. Next, we discuss spatial information such as shape, orientation, and position, combined into one section. Finally, we discuss texture decoding which is a relatively unexplored area. For an overview of key EEG datasets relating to simple visual features see Table 4.

Table 4. Simple visual features EEG datasets.

Author	Task	Stimuli	Device	Channels	Sampling Rate	Duration
Spatial Information Decoding
Llorella et al. [44]	Imagination	7 geometric shapes and then 2 geometric shapes	g.Nautilus g.tec	8 channels - P3, P4, PO3, POz, PO4, PO7, PO8 and Oz	Recorded at 250 Hz	Imagination duration 5000 ms
Qiao et al. [45]	Perception	8 spatial patterns	Neuroscan	64 channels	Recorded at 1000 Hz	Perception duration of 4000 ms
Esfahani and Sundararajan [46],	Perception and Imagination	5 shapes	emotiv neuroheadset	14 channels - AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4	Recorded at 2048 Hz, then down sampled to 128 Hz	Perception duration of 2000 ms and imagination duration of 5000 ms
Gurariy et al. [47]	Perception	Shape vs Tool Images	HydroCel Geodesic Sensor Net	256 channels	Recorded at 1000 Hz	Perception duration of 300 ms
Texture Decoding
Portilla and Simoncelli [48],	Perception	166 natural texture images	Brainvision	19 channels - Fp1, Fp2, F3, F4, C3, C4, P3, P4, O1, O2, F7, F8, T7, T8, P7, P8, Fz, Cz, and Pz	Recorded at 1000 Hz	Perception 500 ms
Colour Decoding
Yu and Sim [49]	Passive perception, and imagination	red, green, blue, white, and yellow LED lights	Emotiv Epoc	14 channels - AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4	128 Hz	Perception 7000 ms and imagination 7000 ms
Hajonides et al. [50]	Perceive then recall orientation and colour of two simultaneous gabor patches	Gabor patches, 48 colour variations, 48 orientation variations. 900 trials per participant	EasyCap	61 channels	1000 Hz	Perception 300 ms
Rasheed [51]	Passive perception, and imagination	red, green, blue and grey squares; 60 trials for each colour per participant	G.tec’s g.MOBIlab+ portable device	4 channels - P3, P4, O1 and O2	256 Hz	Perception 3000 ms; Imagination 3000 ms

5.1. Colour

colour decoding involves determining the colour contents of a stimulus. Everyday stimuli generally contain a multitude of colours. Still most colour decoding focuses on monochromatic stimuli. Past studies typically divide colours into discrete categories, such as red, blue, green, yellow. colour could also be decoded along a continuous colour spectrum rather than discrete categories, though to the best of our knowledge this has not been attempted.

5.1.1. Neural representation of colour

When considering the neural correlates of colour, it is important to distinguish the processing of chromatic content from actual colour perception. In this context, chromatic content refers to hues, colour intensity, and spectral information, whereas colour perception relates to the subjective experience of seeing and interpreting colour. In the early stage of colour processing, chromatic light stimuli from retinal input contributes to object-contour perception, which enables objects to be distinguished from their backgrounds. An fMRI experiment [52] showed this chromatic information to be most decodable from early retinotopic areas including V1,2,3. Though there were some cells in V2 that also contained information about perceived colour surface of objects, this level of colour perception was most decodable from V4, specifically in the lateral occipital cortex. Further evidence for V4’s involvement in perceived colour comes from a colour perception and imagination fMRI study which decoded imagined green, yellow, or red stimuli. V4 was found to be the best predictor of colour category [53]. It is possible there a verbal and/or semantic representation of colour that can can also be used for decoding in higher-level regions.

5.1.2. Colour state of the art

There are at least two EEG-based imagery and perception colour datasets, and two which are restricted to perception. The first perception dataset includes neural data from 30 participants who perceived Gabor patches (sinusoidal gratings usually with a Gaussian envelope) with 48 colour variations and 48 different orientations [50]. Two Gabor patches were presented simultaneously. Rather than passive viewing, participants were asked to remember the colours and orientation for a later recall test, to encourage engagement. Though data was gathered from 61 electrodes at 1000 Hz, in their decoding pipeline, the researchers used a subset of the 17 most posterior electrodes, as well as binning the orientation and colours into 12 bins, to achieve reliable classification of features using Linear Discriminant Analysis. This was because a searchlight analysis showed posterior electrodes at 150 to 350 ms post stimulus to be optimal for decoding. This identification of posterior electrodes supports that the colour decoding was based on visual information as opposed to semantic colour labels. In the other colour decoding perception dataset [54], 9 participants perceived red, blue and green, with EEG data measured from 59 channels. The data was down-sampled to 200 Hz, and decoded via binary classification using a Support vector Machine (SVM) and a feedforward neural network, red vs blue; blue vs green; green vs red. When decoding between two colours using an SVM, the average accuracy was 79.1% (ITR = 51.5 bpm). Channel selection for decoding included the posterior electrodes P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO3, POz, PO4, PO8, O1, Oz and O2. This study did not consider frontal electrodes, however found that the best performing electrode out of the posterior channels, was POz. Interestingly, the visual second-layer channels (POz – PO8), which would be involved in a slightly later stage of processing, demonstrated higher performance than the first-layer channels (Oz – O2), related to early visual processing. This result seems consistent with the fMRI literature as discussed in 5.1.1, showing information related to perceived colour to be encoded in V4 rather than earlier visual processing areas.

The two imagery and perception datasets use a mix of RGB colours, and white, yellow, and gray. One dataset uses 14 channels to record data whilst 10 participants perceive, then imagine, red, green, blue, white, and yellow LED lights [49]. On this dataset, 61.5% accuracy has been achieved for imagination (ITR = 0.3 bpm). The other dataset is recorded from four, electrodes positioned in occipital and posterior regions, from 7 participants who perceived and imagined RGB and gray-coloured squares displayed on a large curved projector [51]. This dataset was used by other researchers to achieve an average of 38% accuracy with random forest classification for the three perceived colours (ITR = 1.2 bpm), and 36% using an SVM classifier for the imagined colours (ITR = 0.9 bpm) [55] For a comparison table of the colour decoding studies see Table 2, and for examples of colour stimuli see Figure 4.

Figure 4. Types of stimuli used in the three mentioned colour decoding studies; a) shows examples of coloured Gabor patch stimuli, b) depicts the coloured square on a curved projector in Rasheed [51] and c) shows the Arduino with led lights used in Yu and Sim [49]. B and C images are adapted from their respective papers.

Table 2. Decoding performance metrics for colour, including details on the classifier employed, feature extraction methods, spatial areas selected, chosen time windows, and the calculation of information transfer rate (ITR).

Study	N -Class	Classifier	Features	Spatial selection	Time (sec)	Accuracy	ITR (bpm)
Hajonides (Perception)	2	LDA	PCA	posterior occipital	0.35	$\sim$	$\sim$
Wu (Perception)	2	SVM	Cleaned EEG	primary visual cortex	1	>70%	7.12
Yu (Imagination)	5	ANN	alpha, beta waves	T7, F4, AF4	7	61.5	0.3
Torres-Garcia (Perception)	3	RF	statistical features	P1, P2, O1, O2	3	38	1.2
Torres-Garcia (Imagination)	3	SVM	statistical features	P1, P2, O1, O2	3	36	0.9

5.1.3. Colour feasibility

In summary, categorical colour decoding seems possible with EEG for both perception and imagination. This can be achieved via channels placed in posterior and occipital regions which is compatible with findings that colour is represented in V1-V4. Whether decoding category from the verbal/semantic labeling as opposed to the low-level representation is possible from EEG, requires further research where frontal channels are used for decoding. Potentially, combining posterior and occipital relating to low level and frontal related to high-level representation could boost decoding performance.

5.2. Spatial information

An object for decoding can vary in its spatial location such as its shape, orientation, and location. For example, it may be a triangle, square, or circle, and positioned in the center of the field of vision, or it may be toward the top right corner and angled 45°. There are a few ways of operationalizing shape, for example, by its contours and edges, or by its skeletal structure, a geometric model based on the medial axis – these are commonly used in pose estimation-based decoding. Orientation can be operationalized categorically such as horizontal or vertical, or on a continuous spectrum of angle degrees. Location refers to the positioning of an object, and is generally operationalized categorically, such as up, down, left, right, or center.

5.2.1. Neural representation of spatial information

Contour shape information is neurally represented in mostly mid-to-higher visual regions. One fMRI study found shape identities of a circle and square to be represented in V3, lateral occipital cortex, intraparietal sulcus and the parahippocampal cortex. Though some decodability of shape was also found in early visual regions such as V1–3 [56]. This is somewhat surprising, as a previous study found reduced V1 activity during shape perception [57]. The decodability of V1 in Erlikhman et al. [56] was interpreted as potentially due to feedback connections.

An object’s shape features are often correlated with its category-level information. For example, the shape of a car or a hammer are fairly consistent across within class variations. One study explicitly disentangled shape and category-level information [58]. They found that low-level pixel-based shape (image silhouette) to high-level perceived shape (perceived shape similarity) followed a posterior-to-anterior gradient, they found ventral category-sensitive regions also showed sensitivity to shape information.

Skeletal shape is revealed to be represented in V3 and LO through two fMRI studies [59,60]. The second presented 20 participants with 20 novel objects which were devoid of semantic meaning, they explored representation in V1–V4, LO, and posterior fusiform sulcus (pFs). Unsurprisingly, these skeletal structures were not found to be represented in pFs, which is often implicated in object recognition [61]. pFs is a new functionally defined region which mostly refers to the fusiform gyrus [62]).

5.2.2. Spatial state of the art

It is shown that shape is encoded throughout the visual cortex, lateral occipital cortex and can be entangled with semantic representation via fMRI studies. To what extent is shape information decodable using EEG data? There are several studies which have classified perceived or imagined shape categories from EEG data. In one study, 7 participants perceived and then imagined 7 geometric shapes including a circle, square, horizontal line, triangle, pentagon, hexagon, and parallelogram. The EEG channels included parietal-occipital regions P3, P4, PO3, POz, PO4, PO7, PO8, and Oz, recorded at 250 Hz. When classifying between all 7 shapes, 35% accuracy was achieved for imagination (ITR = 3.8 bpm), with baseline at 14.3%. On another dataset but using the same hardware, the same authors had 11 participants, for which they classified between two imagined geometric shapes. Perception of the shape was followed by a 5 seconds imagination task, with an average of 70% accuracy across participants (ITR = 5.8 bpm) [44]. This indicates decodability of shape category from posterior brain regions. However, as there was no gap between perception and imagination, it may have been the case that the imagined shape was actually decoded from remnant perceptual activity (see Section 8.5).

Another EEG study chose more distributed electrode positioning − 14 including AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4. EEG data was recorded from 10 participants, at 128 Hz [46]. Participants viewed five shapes, a cube, sphere, cylinder, pyramid and cone (see Figure 5). Colour and size was kept constant, with 2 seconds to perceive and 5 second to imagine. Using a linear discriminant classifier, 44.6% accuracy was achieved (ITR = 2.6 bpm). There was a second iteration of the experiment in which a verbal cue for the image was given to ensure the classification result was not due to remnant visual perception (see Section 8.5). Interestingly, the classifier trained on session 1 performed just as well on session 2 with the verbal cues. Though the difference in performance between testing (100 images) and image testing (50 training images) may be partly attributed to the variation in training data, making a direct comparison less fair. As it was not explored which electrodes are contributing to decoding performance, the decoding-relevant information at different levels cannot be teased apart. This includes the possibility of semantic labels in frontal electrodes and higher-level shape perception in posterior electrodes.

Figure 5. In this figure depicted are: a. The geometric shapes used in [46] b. seven geometric shapes from [44].

Shapes can vary in their spatial location and orientation. A study using 21 electrodes in posterior-occipital regions successfully classified eight spatial patterns; up, down, middle horizontal, left, middle, right vertical, and oblique 45° and 135°, which were flashed at 2 Hz [45]. Each perception block was 4 seconds, followed by a checkerboard stimulus for 0.5 second to clear the previous pattern. Averaged across participants, binary classification was 92.9%, and classification between 4 categories was $\tilde{7}$0% (ITR = 9.7 bpm).

In another perception study, with high spatial resolution (256 channels), visual stimuli were varied along two dimensions, shape (how elongated or stubby the profile is) and toolness (tool or graspable object). Classification of shape was highest around 100−200 ms and again at 350 ms. This study also used source localization to enhance the spatial resolution by acquiring MRI images of the participants – recorded at 1000 Hz [47]. See Table 3 for a comparison of these decoding studies.

Table 3. Decoding performance metrics for spatial features, including details on the classifier employed, feature extraction methods, spatial areas selected, chosen time windows, and the calculation of information transfer rate (ITR).

Study	Class (N)	Classifier	Features	Spatial selection	Time window (sec)	Accuracy	ITR (bpm)
Llorella et al. (imagination)	2	CNN	EEG	posterior occipital	5	70%	5.8
Esfahani et al. (imagination)	5	LDA	Hilbert-Huang Transform	global	5	44.6%	2.6
Qiao et al (perception)	8	CNN-LSTM	$\sim$	posterior occipital	4	70	9.7
Gurariy et al. (Perception)	2	MVPA	EEG	ventral dorsal	0.055 onwards	$\sim$	$\sim$

5.2.3. Spatial feasibility

In summary, shape and positioning decoding both seems feasible by EEG with channel placement on posterior-occipital regions, and at least for shape, distributed channel placement across the head. Future studies should disentangle whether decoding is possible with frontal regions alone. It is interesting that previous studies have chosen to select posterior-occipital regions when shape and category-level information representation is evidenced in the ventral temporal cortex [63]. Further, the neural representation of shape literature indicates it should be possible due to its correlation with category information. For time window selection at least for perception, timepoints corresponding to mid to late stages of visual processing have been implicated as having peak decodability for shapes. Further investigation is required for time-window selection for orientation and positioning/location, and for imagination for all types of spatial information, as currently spatial imagery decoding literature is scarce. Imagined spatial information could be an interesting feature to decode in the case of aphantasia, an impairment related to visual imagery [64] suggest a distinction between object and spatial aphantasia. For a more in depth discussion of individual differences in visual imagery see Section 8.1.

5.3. Texture

Image texture refers to spatially global complex image regions, often with repeated elements. Texture information provides a useful cue for discriminating between image classes, facilitating object recognition. In fact, there is an interesting study [65] showing that, convolutional neural networks, which have been used to model the human visual cortex [66], have a bias toward relying on texture rather than shape information when classifying images. Decoding of texture can either be operationalized as classifying categorisable, nameable and typically artificial textures, such as checkerboards or stripes, or through reconstruction of perceived and imagined textures, whether naturalistic or artificial, which involves a more continuous representation.

5.3.1. Neural representation of texture

Texture can be considered a mid-level representation, captured in the intermediate visual cortex. This representation is composed of low-level visual representations such as orientation and intensity. Yet, texture relates to summary-statistics which are invariant to transformations such as translation and moments of intensity, therefore the spatial arrangement of visual features becomes less important. An fMRI study showed that V2 response strength well predicted the naturalistic structure of different texture types. However, the early visual cortex i.e. V1 did not [67]. Additionally, in an fMRI-adaption paradigm which enables identification of finer-scale neural responses, neural selectivity in the fusiform face and parahippocampal place area was found to be sensitive to an image’s texture properties [68] when participants perceived faces and places. These results indicates that texture representation is distributed throughout the brain but predominantly in early visual cortex such as V2.

5.3.2. Texture state of the art

To the best of our knowledge, there are only two studies which reconstruct texture from EEG activity. Specifically, visually evoked potentials (VEPs) relating to 500 ms perception of 166 natural texture images, were fed into a reconstruction pipeline. The dataset used in both studies consists of 15 participants, with a BrainVision 19 electrode cap, consisting of Fp1, Fp2, F3, F4, C3, C4, P3, P4, O1, O2, F7, F8, T7, T8, P7, P8, Fz, Cz, and Pz, and recorded at 1000 Hz. In the first study, texture statistics for each image are obtained via a Portilla-Simoncelli (PS) texture synthesis algorithm [48], in which high-order statistical constraints are applied to local image features [69]. These statistics were used a response variables in a linear regression with the VEPs as the predictors.

In the second study, which uses the same dataset, a multimodal variational auto encoder (MVAE) is emplyed [70]. The texture image and corresponding VEP are passed into separate encoding models, which then form a joint latent space, after which the image and VEP pass through separate decoders for reconstruction. This approach enables reconstruction from partially observable information, i.e. reconstructing the texture image from just the VEP input, as the latent variables can be inferred.

In Figure 6 we illustrate the respective results of the two studies. Notably, the second study shows improved reconstruction, both in terms of correlations between the original and reconstructed images, and from naive observer ratings.

Figure 6. a) shows reconstructions using MVAE wakita et al. [70] and b) are reconstructions from Orima and Motoyoshi [71] using the figure is adapted from Orima and Motoyoshi [71]; wakita et al. [70].

5.3.3. Texture feasibility

The results depicted in Figure 6 indicate promising feasibility for EEG texture decoding. This is particularly the case as only 19 electrodes were used, which indicates texture can be decoded from very low spatial resolution. Interestingly, there seems to be a distinction for decoding naturalistic vs non-naturalistic texture images; different VEPs were found for each at 200–300 ms, from occipital electrodes. To the best of our knowledge there is no existing work on decoding imagined texture from EEG. As texture can rely on fine spatial resolution of an image, it will be interesting to see the general capacity of individuals to generate stable representations of texture – likely decoding will depend on the vividness of the internal representation.

Here, we present existing datasets for visual imagery and/or visual perception decoding from EEG activity. The information in Tables 4 and 6 refers to the raw datasets only, as often the researchers reusing the datasets apply electrode/channel selection and down sampling during pre-processing.

Table 5. Decoding performance metrics for complex stimuli, including details on the classifier employed, feature extraction methods, spatial areas selected, chosen time windows, and the calculation of information transfer rate (ITR).

Study	N -Class	Classifier	Features	Spatial selection	Time (sec)	Accuracy	ITR (bpm)
Kumar et al. (imagination)	3	RF	Statistical features	global	10	85.2%	5
Tirupattur et al (imagination)	10	CNN LSTM	EEG	global	10	$\sim$72%	2.7
Nemrodov et al. (perception)	2	SVM	EEG	occipito-temporal	0.05 to 650	71%	13.1
Shatek et al. (perception)	2	Linear Discriminant	EEG	global	0.12	$\sim$58%	9.27
Kosmyna et al. (perception)	2	Logistic Regression	SpecCSP	global	2	60%	1.1
Kosmyna et al. (imagination)	2	Logistic Regression	SpecCSP	global	2	52%	0.03

Table 6. Datasets for naturalistic/complex stimuli relaying stimuli details, acquisition device, sampling rate and duration of the imagination or perception task.

Author	Task	Stimuli	Device	Channels	Sampling Rate	Duration
Kumar et al. [72]	Perceive and imagine	colour digits (0–9); non-text objects; letters	Emotiv EPOC+	14 channels	128Hz (downsampled from 2048 Hz)	Perception: 10000ms, Imagination: 1,000ms
Nemrodov et al. [73]	Perceive, go trials in a go/no-go gender recognition task	Unfamiliar male faces, 54 happy and 54 neutral expressions	Biosemi ActiveTwo EEG recording system	64 channels	512 Hz	Perception: 300 ms
Shatek et al. [74]	Perceive four images, then imagine one of these, then select which one chosen for imagination	Variations of Sydney Harbour Bridge, and Santa-Claus (40 trials per class, 80 in total)	Brain Products ActiCAP	64 channels	1000 Hz	Perception: 1,500 ms, Imagination: 3,000ms
Kosmyna et al. [75]	Visual observation followed by visual imagination	Two simplified pictures on black background 1) daisy, 2) hammer	2 g.tec USBAmp amplifiers	36 channels		Perception: 4,000ms, Imagination: 4,000ms
Spampinato et al. [76]	Passively perceive images	Various objects scuh as dog, cat, butterfly, sorrel, capuchin, elephant, panda, fish, airliner, broom, canoe, phone, mug, convertible, computer	Brainvision DAQs and amplifiers with actiCAP	128 channels	1000 Hz	Perception: 500ms
Rashkov et al. [77]	Passively perceive video clips of different objects	Abstract geometric shapes, waterfalls, human faces, goldberg mechanisms, extreme sports	NVX-136 amplifier developed by Medical Computer Systems Ltd. (Moscow, Zelenograd)	128 channels	500 Hz	Single clip was 6,000 to 10000ms

6. Complex visual stimuli feasibility

Complex visual stimuli refers to compositions of whole objects and images, rather than the building blocks of visual features such as colour, orientation, shape. For the neural decoding use-case of BCIs, decoding complex visual stimuli is arguably more valuable as a goal than simple stimuli, as these complex stimuli can carry and convey more information. Further naturalistic complex stimuli relate more to the visual information that we are exposed to on a daily basis. In this section, we cover both imagined and perceived complex stimuli. Common stimuli choices for decoding include naturalistic stimuli such as faces, scenes, and objects, and artificial stimuli such as simple objects, and also artificial stimuli such as digits and letters. We consider faces separately as there are distinct neural representations for face processing [78]. For an overview of key EEG datasets relating to complex visual features see Table 6. For a comparison of the decoding results, and time and spatial features selection for complex stimuli, see Table 5.

6.1. Neural representation

When considering the neural representation of complex stimuli, we consider both semantic and pictorial content to be important. There are some specific areas related to object identity, for example the fusiform gyrus is considered a computational hub for face processing [78,79]. The parahippocampal place area has been implicated in processing of place-related stimuli, for example pictures of landscapes or scenes [80]. Additionally, inanimate and animate stimuli distinctions, such an animal vs. a tool can be decoded in the inferior temporal cortex [81] Of course, it can be argued that category-level information can become amodal, and no longer debatedly visual information. On one hand, it can be hard to disentangle the two to ascertain which is leading to decodability. On the other, amodal category-level information can be useful extra information, which can boost the decoding of visual information.

6.2. Imagined objects, digit, letters

Overall, there are fewer datasets for imagined than for perceived visual information. This is in part because studies that focus on decoding imagery will also capture perceptual data, whereas the reverse is not usually the case. For visual imagery, there is one core dataset used by many researchers which consists of visually imagined coloured digits (0–9), letters, and non-text objects from 23 participants [72]. This dataset was recorded using a 14 channel Emotiv EPOC+ device, at a sampling frequency of 2048 Hz, though this was later down sampled to 128 Hz. Each participant saw and imagined each image just once. There was a 10-second duration for perception followed immediately by a 10-second imagination of the same stimulus. Between each stimulus was a 20 second gap. While this gap prevented contamination between stimuli, the lack of gap between the perception and imagination trial is problematic, as the prior perception neural signals can contaminate the visual imagery brain signals. This is a common methodological flaw in visual information decoding work, not unique to this study, which we discuss in depth in Section 8.5. Presentation order of the stimuli was kept constant and not randomized. Classification performance on this dataset was high. Kumar et al. [72] achieved an average classification accuracy of 85.2% when decoding which of the three categories the neural activity relates to, when combining all participants’ data (ITR = 5 bpm). Tirupattur et al. [82] achieved within category classification accuracies of 72.9% for digits, 71.2% for characters and 72.9% for objects. This dataset was further used by Jolly et al. [83] to create a GRU-based universal encoding pipeline which can be used to extract meaningful features on multiple datasets. Whilst the results would indicate that decoding imagined objects, digits and letters is feasible from EEG data – the methodological issues within the dataset make this conclusion shaky.

6.3. Perceived faces

One body of research aims to classify and reconstruct perceived faces [73]. The dataset produced for this work includes 64-electrode EEG recordings from 13 participants obtained whilst perceiving faces. Specifically, 70 different individuals’ faces were displayed. The focal stimuli of the experiment was 54 images of unfamiliar Caucasian males, of which a happy and neutral expression was shown for each (see Figure 7). Though an additional six famous males were included to promote alertness, and 10 female faces used as part of a gender recognition go/no-go task. Each face is shown on a black background, with an oval mask which keeps the shape consistent throughout. Images were presented in a pseudo-random order, with each male face presented twice and female faces presented once in each block. As there were 32 blocks, this results in 64 trials for each unfamiliar male face. The original authors achieved 64% (ITR = 5.7 bpm) and 71% (ITR = 13.1 bpm) accuracy for across and within expression facial identity respectively, which remains almost unchanged after normalization. Though, it is worth noting here that chance was at 50% due to the classification approach used. They also explored the temporal dynamics, so identified which time points were most important for decoding. In terms of which timepoints were most valuable for decoding for high-level information, it was found that from 150 ms across-expression was most robustly discriminated, whereas within expression decoding was discriminated at earlier time points.

Figure 7. Examples of the unfamiliar male faces used. The top row are the original displayed faces, the bottom row are their respective reconstructions. Adapted from Nemrodov et al. [73].

6.4. Imagination and perception - binary classification of naturalistic stimuli

Several datasets have been created that are binary classification tasks of naturalistic stimuli. Though these are binary classification problems, the stimuli used are 3D and colour, meaning they are relatively complex, which may explain the low performance.

One such dataset was recorded from 16 participants, using a 1000 Hz 64-electrode Brain Products ActiCAP system [74]. The two decoding categories include the Sydney Harbour bridge images (places) and Santa-Claus images (faces), though exemplar-level decoding is also attempted. Participants viewed four sequential images, an ensemble of santa-claus and Sydney Harbour bridge photographs (see Figure 8. Once cued, the participant visually imagined one of these four images and then indicated which one they had chosen by clicking a picture of it on the screen. An advantage of this dataset is that they provide Vividness of Visual Imagery Questionnaire (VVIQ) scores for each participant. For perception, above-chance accuracy was obtained for category-level decoding, there was also some success for exemplar-level decoding. Though neither category nor exemplar level decoding was successful for imagination.

Another dataset involves 26 participants who perceived then imagined a picture of a hammer, and a flower against black backgrounds [75]. The EEG data was gathered using a 36 channel g.tec kit (see Figure 8). Using a spectrally weighted common spatial patterns classifier, 60% accuracy (ITR = 1.1 bpm) was achieved for category level perceived images, and 52% accuracy for the imagery task (ITR = 0.03 bpm).

Figure 8. Showing the types of binary stimuli used. In a) classification of an imagined/perceived hammer vs. A flower adapted from Kosmyna et al. [75], b) classification of Sydney Harbour bridge vs. Santa Claus adapted from Shatek et al. [74].

6.5. Perception classification and reconstruction of naturalistic stimuli

A core dataset for visual perception [76] decoding, has been used in at least 9 further studies (see Figure 10 [85–89]). This is likely due to the large number of classes (40), complex stimuli, and high electrode count. This EEG dataset was gathered using a 128 channel Brainvision system, and consists of neural responses from 6 participants (5 male, 1 female). The stimuli used were easily identifiable, naturalistic stimuli. Specifically, they use a subset of images from ImageNet − 40 classes with 50 exemplars from each class. The task involved passively perceiving images from each class being displayed in bursts for 500 ms each. A common evaluation method for assessing how realistic and high-quality generated images are, is the Inception Score (IS) [90]. The lowest IS score is 0, and the highest is bounded by the number of classes. A higher IS indicates a higher quality generated image. The generated images are passed through the pre-trained Inception V3 to predict the class probabilities. The IS algorithm uses these conditional probabilities to output a score based on the quality and diversity of the generated images. This score is shown to correlate with human observer ratings of generated image realism. Zheng et al. [84] achieve an IS of 5.53 (chance is 2.5%), a slight improvement on Palazzo et al. [91] in which the IS is 5.07. See Figure 9 for example reconstructions from Zheng et al. [84].

There is another dataset of naturalistic stimuli, in which participants passively perceive video clips belonging to five different categories [77]. These categories include abstract geometric shapes, human faces displaying different emotions, goldberg mechanisms, natural waterfalls and extreme sports. The extreme sports category includes first person videos of high-speed motion activities. This introduces the confound that the brain states associated with each category will also differ by emotional state; for example, the waterfall will be relaxing whereas the sports will be stressful. This dataset was recorded using a 128 channel system on 17 participants. Reconstruction performance was assessed by human observers who found roughly 90% of the reconstructed images to be identifiable as belonging to their intended category. One motivation for using videos rather than images, is that participants may be more motivated and engaged during the experiment. Problematically, undergoing long sessions of EEG results in boredom, mental fatigue and low alertness, which reduce attention on the task. In fact, changes in alpha and theta [92,93] power is found to change over time as a consequence of mental fatigue and boredom. Therefore using videos as stimuli which are more engaging than passive viewing of single images may enable longer testing sessions and therefore more data per participant.

Figure 9. Examples of reconstruction results from Zheng et al. [84].

Figure 10. Illustrates that at the time of this review, at least eight further visual decoding studies have been carried out which use Spampinato et al. [76]’s foundational dataset.

7. Feasibility of EEG hardware

When selecting a neuroimaging modality to use for visual information decoding, there are trade-offs to consider including spatial and temporal resolution, practicality, cost and safety. These all impact the feasibility of using EEG for visual information decoding. We examine each in turn with consideration of how the feasibility with EEG compares to other neuroimaging techniques (see Table 7 for a comparison table).

Table 7. This table compares the relative cost, spatial and temporal resolution, portability, and invasiveness of the neuroimaging modalities discussed in this current section on EEG hardware.

7.1. Spatial resolution

Throughout the visual cortex, visual information is represented in a retinotopic reference frame which relates to where the perceived or imagined stimuli are spatially located [94]. Exploiting the rich spatial information via neuroimaging technique with high spatial resolution can boost decoding. Unsurprisingly then, fMRI has been a popular neuroimaging choice for visual decoding due to its high spatial resolution – around 2–3 mm spatial resolution with 3 Tesla fMRI and under 1 mm with 7 Tesla fMRI. Consequently, it has been used to produce impressive visual decoding results [95,96]. One of the main concerns for visual information decoding with EEG, is its low spatial resolution. Improvements can be made by using high-density devices, though even a 128 channel system is limited to around 6–8 cm³ spatial resolution. Deep learning super resolution algorithms [97] may be applied to boost the spatial resolution of EEG, though to the best our knowledge these have not yet been applied in the visual information decoding context. To boost spatial accuracy, or rather, the localization of the signal origin, source localization algorithms can be applied. This is a more commonly used technique for EEG data than super resolution, though it is less typically used in visual decoding studies as generally source reconstruction requires obtaining an MRI anatomical image.

Another feasibility question is to what depth and extent EEG can record from the brain. Electrical activity picked up by scalp electrodes, must first pass through cerebrospinal fluid, insulating skull and finally skin, which distorts the signal. Additionally, the cortex is not flat, it undulates with gyri, sulci and deep fissures. This makes establishing the source of a signal difficult. To record from the primary visual cortex which mostly sits in fissures, electrodes can be placed over the calcarine fissure, yet just the central 10 degrees of the visual field is located on the surface [98]. There is also a general consensus that deep brain regions are not visible via EEG. Such subcortical regions include the thalamus, nucleus acumbens and hippocampus. One recent study recorded high density (256 channel) scalp EEG alongside intracranial electrodes which provide a ground truth for subcortical activity [99]. An MRI scan was acquired to enable EEG source reconstruction. Importantly, the authors found high correlation between the intracranial and source reconstructed EEG recordings. Whilst this does not demonstrate that the extent of subcortical signal obtainable via EEG will improve visual information decoding, it illustrates that with further progress in source reconstruction algorithms, there is scope for EEG to measure at a deeper capacity.

7.2. Temporal resolution

Visual processing has complex and quick temporal dynamics, for example a study using 1200 Hz MEG showed image categories of faces and scenes could be distinguished from 85 ms after the image was presented van de Nieuwenhuijzen et al. [100]. EEG’s superior temporal resolution makes it a potentially strong candidate for visual imagery and perception decoding, which contain temporal information that can be exploited for improved decoding. These two processes have some temporal similarities – Xie et al. [40] demonstrate that visual perception and imagery overlap in the alpha band, which is associated with top-down or rather feedback processing. However, they also have distinct temporal profiles, which enables identification of whether an individual is engaging in visual perception or imagination [75]. For example, imagery ERPs are shown to have a longer latency than the perceptual condition, with a delay of about about 200–400 ms [101]. The authors suggest this may be because accessing mental representation and reconstructing the image, take more time than solely sensory processes. The temporal information related to visual processing may be exploitable for decoding meaningful information extraction of the visual image. Non-invasive technologies such as fNIRs and fMRI which have high spatial resolution, rely on the hemodynamic response, which is slow which results in a typical sampling rate of around 2 seconds in fMRI. These consequently suffer from poor temporal resolution which can be a limitation in visual decoding.

7.3. Practicality and cost

We suggest that a lot of the motivation behind selecting EEG in visual decoding studies is due to its accessibility. EEG is accessible due to its relatively low cost and the wide range of EEG-based technology, from consumer to research grade, which differ in the number of electrodes, sampling frequency and cost. EEG is also very versatile for use cases which require portability. While traditionally EEG use was confined to a faraday cage and large amplifiers which can be suitable for a clinical setting, there are now systems with amplifiers smaller than an A4 notepad which can operate wirelessly and with stability in the field. MEG offers relatively high spatial and temporal resolution and is consequently used to research visual processing spatio-temporal dynamics [100,102]. Yet, whilst MEG is a potential solution to the spatial temporal trade-off, it is costly and lacks mobility. Still, EEG setup can be time-consuming depending on the system used. For example, channels may need to be carefully placed, and for wet electrode systems, application of conductive gel can be a lengthy process particularly with high-density channels. Wet electrodes remain the gold standard for acquiring optimal signal quality compared to dry electrodes, which whilst quicker to setup, can result in signal degradation and susceptibility to artifacts [103]. We recommend researchers to select wet electrodes to boost the feasibility of decoding visual information via EEG.

7.4. Safety

Some of the more successful attempts to decode visual information have involved invasive techniques such as microelectrode arrays on mice [104] and electrocortiography (ECoG) on humans [105,106] who are usually undergoing surgery for conditions such as epilepsy. Microelectrode arrays, often inserted inside the cortex, can yield single-unit or multi-unit activity, enabling extremely high spatial resolution. ECoG electrodes placed subdurally on the surface of the brain also offer high spatial resolution. Consequently, there is growing interest in using invasive techniques for people without neurological injuries. For example, the NeuraLink device has impressive spatial resolution and wireless capability [107]. The company aims to develop a neural implant that can be used to control a computer or mobile device anywhere the user goes, for those with disabilities but also those without. So far, this has only been implanted in pigs and monkeys, though there is a large body of invasive BCI research in humans [108,109]. For example Willett et al. [110] developed an intracortical BCI that facilitated decoding of handwriting movements in a participant with hand paralysis. Further, stereo-electroencephalography (sEEG), due to the deep brain regions it can reach with its implantation of intracranial electrodes, has a high signal-to-noise ratio, and demonstrated efficacy in a P300 paradigm [111]. Still, there are safety and ethical concerns surrounding invasive neuroimaging techniques (see Burwell et al. [112] for a review), such as the risks of surgery and infection alongside glial scarring which can impede the implant and impair the signal,

Alongside the development of invasive technologies, there remains a need for further advances in alternative, noninvasive technologies. EEG alongside other noninvasive techniques such as fMRI, fNIRS and MEG can be considered safer, and also are not limited to participant pools of solely those already undergoing surgery.

8. Challenges during data gathering

The design choices made during data gathering can impact decoding performance, validity of results and the generalizability of approaches and findings to other contexts. Well thought out experimental design can boost the feasibility of using EEG for visual information decoding. We outline and discuss the design factors which should be carefully considered for visual perception and imagery decoding studies. These include, (1) individual differences in visual imagery ability when recruiting participants; (2) block versus event-related design; (3) controlling for confounding perceptual input; (4) whether the task is active or passive and the impact of eye movement; and (5) the need for gaps between trial components, referred to as ‘palette cleansing’.

8.1. Participants − individual differences in visual imagery ability

There is large variance in how different individuals experience visual imagery. When recruiting participants for an EEG study involving visual imagery tasks, it is important to assess their experience of visual imagery, for optimal task design and decoding. Depending on the specific task, the potential participant’s visualization ability may serve as exclusion or inclusion criteria or be useful information when designing the decoding pipeline.

Vividness is one common imagery variation. Recently, the terms aphantasia and hyperphantasia were coined to refer to extreme ends of a vividness of visual imagery spectrum [113]. Aphantasia refers to the inability to voluntarily visually imagine, though with some nuances: there is a case study where an hallucinogen improved visual imagery [114] and some aphantasics report having visual content in dreams [115]. Hyperaphantasia refers to the capacity for extremely vivid, as detailed as if ‘seeing’, visual images. During data gathering for a visual imagery task, if a person experiences no visual imagery, then they might interpret the instruction ‘imagine the face you saw previously’ to mean ‘think of the semantic details of that face’ as opposed to its visual content. Additionally, neural activity will be modulated by the vividness of the imagined stimulus [3], as well as the participant’s ability to visualize demonstrated in both resting state and task-based fMRI [116]. We recommend assessing a potential participant’s visualization experience by using the Vividness of Visual Imagery Questionnaire (VVIQ) which measures how vivid the visualization is, alongside asking them to describe any nuances or characteristics of the visual imagery they generate. For example, individuals report nuances in the way of three-dimensionality, movement, and ability to project the mental imagery into space [117].

While it is interesting to include participants with a wide range of imagery strategies in studies, it is useful to consider these strategies when creating decoding pipelines. For example, if one person has a widely different experience of imagery, it may be sensible to create separate decoding pipelines on an individual basis rather than combining across participants.

8.2. Is block design problematic?

Block design and event-related design are two types of experimental design often employed in neuroimaging studies. In a block design, each condition is presented continuously for an extended time period. The block relating to one condition is compared to the block for another condition, rather than comparing individual trials. In contrast, event-related design involves discrete trials of short duration for each condition, often in a randomized order. There is debate in the visual information decoding from EEG literature on whether block designs are appropriate to use or if stimulus randomization is essential [118–120]. Contention surrounds one of the core datasets mentioned in Section 6.0.5, heavily critiqued for its block design [119,121]. If this critique is warranted, then this would weaken the dataset and the fairly large body of work already built upon it as well as emerging new decoding studies.

The dataset, as described in Section 6.0.5, consists of 40 image classes. For each class, 50 image exemplars are shown consecutively, in a block, in 0.5 second bursts. High classification accuracy by the original authors and later researchers for these naturalistic, complex stimuli has resulted in extensive media attention. Li et al. [119] draw attention to the lack of stimuli randomization and jittering, as problematic. The classification could be driven by long-term static neural activity relating to the time block rather to the stimuli class itself. In fact, they demonstrate a reduction in classification accuracy when temporal correlates are accounted for. Similar critique has been made of a perception and imagery decoding dataset in which stimuli presentation order was kept constant and not randomized. As this could result in unwanted temporal correlations driving the classification accuracy, the study setup has been questioned by Rekrut et al. [122] and by Li et al. [118]. The original authors of the 40-class dataset contest this critique of block designs, arguing that people react faster and more consistently when conditions are presented in blocks, resulting in more stable responses with a higher signal to noise ratio.

Indeed, block design can facilitate a higher signal to noise ratio and was standard until event-related fMRI work began in the early 1990s [123]. Liu [124] notes that the convolutional model for fMRI analysis [125] was one of the key advances resulting in more complex designs for event-related work. EEG research in this area is still in its early stages and suffers a notoriously low signal to noise ratio. Hence, given the current state of the art, classification work which decodes from block design datasets is arguably the most suitable approach until advances are made to reduce noise.

Nonetheless, it should be acknowledged that using block design can limit ecological validity. When applying BCIs in a real-world setting, it is essential to decode visual stimuli which are temporally sandwiched between other, irrelevant visual stimuli. In this context, it is unrealistic to rely on successive presentation of visual stimuli from the same category as is the case with block design. While currently there are advantages to using block design to increase the signal to noise ratio, and thus decodability, the findings will generally not have the same applicability as an event-related design. Researchers creating or reusing block design datasets should be aware of these limitations.

8.3. Confounding external sensory input

In perception decoding experiments, a participant must keep their eyes open to perceive the stimuli. Problematically, perceptual inputs unrelated to the stimulus can interfere with the stimulus related neural signal. Similarly, in imagery decoding experiments, perceptual inputs can interfere with the neural activity related to the visual imagery tasks. This is due to the overlap in visual perception and imagery systems. Decoding success may then be attributable to erroneous correlations between the perceptual information and imagery task rather than to imagery activation itself. Most confounding perceptual inputs will be those that exhibit a systematic bias, i.e. they vary in parallel with the stimuli. One example here, are text and image-based cues. These can result in systematic eye movements whilst reading the word or scanning the image. A more nuanced confounding perceptual input can be the environment itself, such as a room full of different object that are in the field of view of the participant. Visual attention may be deployed to different aspects of this external environment throughout the duration of the experiment. This may become systematic if there are, for example, semantic associations between the experimental task and objects in the visual field. It is therefore important to control for perceptual inputs external to the stimuli by keeping these inputs constant and, when unrelated to the experiment, out of view of the participant. Another potential benefit of eradicating perceptual interference is an increase in visual working memory capacity due to reducing perceptual input, allowing for more complex stimuli to be decoded, although this hypothesis remains to be tested.

Two possibilities for controlling these external perceptual inputs during imagery tasks include (1) eye closure (2) a pitch black environment with eyes open. Both options have relative advantages and disadvantages. Requiring a pitch black setting constrains the generalizability of results to artificial settings. Eye closure is more feasible in naturalistic settings than a pitch black room, however, it is linked to strong alpha and related motor signals which can create interference. These could create difficulties in picking out the more subtle signals related to the stimuli. Further, eye closure impairs tracking eye movements in hybrid designs, where eye tracking is used in conjunction with EEG. Though a pitch black room can also be used for visual perception tasks, past studies tend to rely on participants’ focusing on the stimuli display and ignoring extra perceptual information.

Each approach to reduce perceptual interference has been used in past EEG imagery decoding studies. Yu and Sim [49] use a dim, though not pitch black, room. Alharbi et al. [126] and Rasheed [51] also use a dark room. To the best of our knowledge, Kumar et al. [72] is the only study to employ eye closure, though this dataset has also been used in several subsequent studies. Several studies use neither eye closure nor lightless conditions to prevent perceptual interference.

8.4. Passive vs active tasks – fixation points and eye movements

Here, we discuss the impact of eye movements in passive vs active viewing tasks. In a passive task, participants are asked to simply perceive a visual stimulus without any additional actions. Whereas, in the case of an active task, the participant may be required to perform an additional task based on the contents of the stimulus they perceive, such as pressing a button – therefore, they must be actively perceiving the stimulus. Eye movements have many important roles in visual perception, for example, preventing perceptual fading [127] and supporting pattern completion [128]. However, there are indications that when decoding perceived stimuli from neural activity, eye movement can result in confounding early visual cortex activity [129]. We also consider the impact of eye movement in visual imagery decoding, with consideration of fixation points.

In general, eye movement is considered to be confounding. Eye movement has been demonstrated to be systematic when active viewing is required to perform the task [129]. This can drive decodability in visual perception tasks, which confounds neural activity related to visual information. In Thielen et al’.s experiment [129], participants performed both passive viewing of oriented square-wave gratings under fixation constraints, and active viewing requiring a button press when the stimulus was perturbed. In the active task, eye movements drove classification of the stimulus perceived, whereas this was not possible in the passive version. The authors recommend post hoc analyses to check for confounding effects of eye movements in active studies; they found removing saccades reduced the confounding effect. Though, we note that when the aim is not to understand neural mechanisms, there will be some BCI researchers for whom the activity driving decoding performance is not important so long as performance is still high.

Most perceptual decoding studies considered in this paper use passive tasks [51,55,75,76,87,126,130–132]. Only a few studies use active viewing tasks, such as Shatek et al. [74] and the go/no-go paradigm used by Ling et al. [133] and Nemrodov et al. [73]. The majority of perceptual and imagery decoding studies incorporate fixation points. This is desirable for reducing confounding eye movement driven neural activity. Fixation points also ensure that overt attention is consistent across trials. However, preventing eye movement may be problematic in imagery decoding studies. This is because there is evidence of image-specific fixation reinstatement during imagery [134], and reenacted scan paths [135]. It is debated whether eye movement plays a functional role in visual imagery [136]. Yet, recent work does support a functional role of eye movement in memory retrieval [128,137]. Problematically, preventing eye movement seems to impair visual imagery [135]. Consequently, when decoding studies use fixation points as a response to reducing noisy signals driven by eye movements, the compromise is that this may reduce the quality of the mental image formed and thus the decodability of the neural signal.

It has not been directly tested whether eye movement can result in confounding decodability in imagery decoding as has been demonstrated in perception decoding [129]. Yet the overlap between the two processes means that we may infer that eye movement will also be problematic in visual imagery decoding paradigms. Consequently, there is a trade-off between preventing eye movement, and thus impairing visual imagery, and including it, risking eye driven neural activity and confounded decoding.

8.5. Palette cleansing

Studies that lack a sufficient gap between visual trials, risk contaminating neural signals. Addressing this methodological flaw is essential to ensure accurate decoding of visual information.

In particular, when conducting perceptual decoding tasks, it is crucial to ensure that neural activity associated with perceiving the target stimulus is not impacted by perception of a prior stimuli. Similarly, in visual imagery decoding studies, neural activity for imagery and perceptual tasks can become confounded by previous stimuli, especially as the stimulus is often shown to the participant prior to the imagination task. To mitigate this, it is important to introduce a temporal gap between the perception task and imagery task. Additional measures should be taken to minimize overlap in the neural representations of the two tasks, ensuring that visual imagery is not solely decoded from residual neural activity associated with perception. Visual noise masks can help reduce this interference. We explore various considerations when selecting a visual noise mask, including factors like structured versus unstructured and static versus dynamic options.

Figure 11. The left side depicts the difference between unstructured, structured and dynamic visual noise. For unstructured dynamic, black and white dots are selected at random with multiple iterations. This example shows three random dot variations shown consecutively. The right hand side depicts how visual noise can be injected into the experiment pipeline, before and after each perception and imagery task.

The literature indicates that visual noise should be unstructured, for example random black and white dots, rather than structured, such as fragments of letters [138]. Visual noise can also be static or dynamic (see Figure 11). Dynamic means the unstructured noise changes during one presentation. Direct comparison between the dynamic and static masks indicates that they are each effective at disrupting representation of images, and there are no unintended side effects of dynamic masks [139,140].

In previous decoding work visual noise masks, when used, are generally static. However, there are some likely advantages of using dynamic noise. Static noise can result in an ‘imprint’ of the previously seen visual stimulus onto the displayed noise, after the disappearance of the actual stimulus. As dynamic visual noise disrupts the imprinted perceptual trace, it prevents ‘imprinting’. We therefore recommend the use of dynamic unstructured visual noise in perceptual and imagery decoding studies.

In most perception decoding studies, participants are shown a blank screen for varying intervals of time between each perceived stimulus [76,130,131], though Nemrodov et al. [73] used a screen with a fixation point. A white noise mask was used in just one study, Ling et al. [133], with a duration of 100 ms. A mixture of ‘palette cleansers’ have been used in decoding studies with both perception and imagination tasks, In one study, ‘REST’ in white letters was shown after the imagination task followed by a blank screen [75], though no interlude was given between perception and imagination. Kumar et al. [72] leave a gap of 20 seconds. Yu and Sim [49] turn off the LED colour light for 5000 ms. A stimulus cue is shown between perception and imagination in Shatek et al. [74]. While Rasheed [51] uses a 9-second rest with closed eyes after 3 seconds, it occurs only after every three sequences. All of these approaches may partly reduce the impact of the previous perceived or imagined stimulus, however, we recommend using dynamic visual noise masks as an evidence backed and consistent method of palette cleansing.

9. EEG dataset size limitations

We begin by discussing synthetic data generation and augmentation (Section 9.1). This solution addresses the common challenge of limited datasets, and sits between data gathering and data decoding of limited datasets. We break down this augmentation section into first traditional machine learning techniques in Section 9.2, then deep learning augmentation approaches in Section 9.3. Finally, in Section 9.4 we discuss how multimodal or rather fusing EEG with additional neuroimaging modalities may boost decoding performance.

9.1. Small datasets - augmentation

In the previous section, we established the importance of gathering high-quality datasets with robust design choices, to improve performance and validity in the decoding stage. Data gathering for visual information-related processing can be a time consuming, resource intensive and financially costly process. Further, visual imagery tasks can be cognitively taxing, which limits the amount of data that can be gathered in one session. Using multiple sessions can be negatively impacted by the non-stationarity of EEG time-series data. Non-stationarity refers to the change over time in the EEG signal due to technical changes such as dried electrodes, changes in mental state such as fatigue and disengagement, and to changes in neural dynamics [141]. This creates difficulties in creating large real-world datasets. Yet, large, high-quality datasets are a prerequisite for decoding, particularly when using deep learning models. One solution to having only small datasets is to create synthetic data, typically by augmenting empirical data. Synthetic data generation can be used to boost the size of an existing dataset and also to increase its generalizability to a test set. This generalisabiltiy is facilitated by the increase in data variance that augmentation provides. While synthetic datasets are becoming quite common in fields such as computer vision [142], this technique is only recently gaining traction for BCI applications [143].

9.2. Traditional data augmentation techniques

There is a wide variety of traditional methods that can be used for EEG data augmentation. These augmentations differ somewhat from those applied to image data, although EEG signal data can be transformed into images to enable augmentation and decoding techniques from the image analysis domain to be more readily applied [144]. Typically, augmentation is applied to the time-series data. For example, evoke CutCat for motor imagery data, an approach that cuts the time window related to one trial and concatenating it with a time window from another related trial. Only afterward is the time-series converted into 2D images (spectrograms) to input to a CNN.

In their review of data augmentation approaches for EEG, Lashgari et al. [145] show that a sliding windows technique is the most commonly used approach. This involves sliding a window over the time series file, and cropping it at various time points. Jittering is also a popular technique, and entails adding noise to the raw EEG or image transformed input data, such as Gaussian, Salt and pepper or Poisson noise [145]. Rotation, a common image augmentation technique, in the case of EEG, can be used to create robustness for the differently shifted electrode locations that can occur when testing the same participant across multiple sessions [146]. The spatial data, where the electrodes are located on the head, can be considered in 3D. The data can then be augmented by rotating it around these three main head axes.

Identifying the best augmentation technique can be tricky as in prior work the baseline decoding performance without augmentation is rarely provided. However, an empirical comparison of data augmentation techniques on 128 time series datasets [147] found magnitude domain transformation such as jittering, magnitude warping, and scaling all achieved similar performance for boosting accuracy. As did the time domain transformation time-warping. However, flipping, slicing and window-warping achieved lower performance. This comparison used eight types of data, varying from ECG and ECoG, to traffic and trajectory data. Therefore, these findings may not generalize to visual perception and imagery EEG data.

While it is not visual information decoding, Lashgari et al. [145] directly compare data augmentation techniques on the same motor imagery EEG dataset, BCI IV-2a. The techniques compared include noise addition, sliding window, Fourier transform, recombination of segmentation and the deep learning augmentation technique Generative Adversarial Networks (GAN). Augmentation using GAN showed the largest boost in accuracy. We discuss such deep learning approaches to data augmentation in the following section.

9.3. Deep learning data augmentation techniques

One of the more popular deep learning techniques for data augmentation is to generate synthetic data using Generative Adversarial Networks (GANs) [148]. In this context, a GAN is used to generate additional EEG time series data to be used as input to the decoding model. GANs consist of a jointly optimized generator and discriminator neural network: the generator attempts to produce realistic synthetic EEG data while the discriminator tries to identify whether the data produced by the generator is real or synthetic. We note that GANs can also be used in the decoding process, such as a reconstruction of an image that a person perceived.

There are many variants of GAN that can be used for synthetic data generation [149], such as deep convolutional GAN, conditional GAN, auxillary GAN, and Wasserstein GAN (W-GAN) [150]. Synthetic data generated using W-GAN has led to improvement in decoding performance in applications such as emotion recognition [151] and producing synthetic high-resolution EEG data from low-resolution samples [152].

Conditional deep convolutional GANs, in which label information is given to the generator and discriminator, have also been used for augmenting motor imagery datasets resulting in performance enhancement [153]. Fahimi et al. [153] have used these GANs to boost performance on datasets where attention is diverted.

Of course, synthetic data, while increasing dataset size and variation and thus filling gaps, will vary in quality. In some circumstances it will introduce substantially more noise into the dataset, even reducing performance. In fact, GAN approaches to data generation have received criticism due to the low likelihood of the generated synthetic data being based on the same data distribution as the original target distribution [154]. Consequently, it is important to evaluate the quality of synthetic data produced by comparing it to the real-world data. Some evaluation metrics include Euclidean distance, sliced-wasserstein distance, Frechet Inception Distance [155], Inception Score [90] and Jensen-Shannon Divergence which is based on KL divergence [156]. To the best of our knowledge, these synthetic data generation techniques have not yet been applied to BCI EEG-based visual information decoding. Researchers in this field should weigh up the potential costs and benefits of using synthetic EEG data and use appropriate evaluation techniques to assess the quality of the synthetic data. We recommend Lashgari et al. [145]’s recent review for a more general overview on data augmentation techniques for deep learning-based EEG.

9.4. Multimodal neuroimaging fusion

As detailed in Section 7, each neuroimaging modality has its own set of limitations. For instance, fMRI offers high spatial resolution but suffers from low temporal resolution, while EEG, conversely, provides high temporal resolution but has limited spatial resolution. Combining the superior temporal resolution of EEG with the enhanced spatial resolution of fNIRS or fMRI has the potential to significantly enhance decoding capabilities, mitigating the shortcomings of each individual modality. This approach holds particular promise for harnessing fine-grained spatial features and retinotopic mapping related to visual information. Although there is limited prior research involving the fusion of EEG and fMRI for visual information decoding, and none that combines EEG with fNIRS, there are relevant studies, such as Cichy and Oliva [157] for MEG and fMRI fusion in the investigation of spatiotemporal dynamics in visual object perception and Muukkonen et al. [158] for face perception.

Currently, there is only one dataset that combines intracranial EEG and fMRI, consisting of 18 participants watching an audiovisual film; however, these tasks were performed non-simultaneously [159]. As this dataset is relatively recent, no comparisons between bimodal and unimodal decoding have been conducted. For speech decoding, EEG has shown promising results when fused with fNIRS data [160]. In this study, a recording device with 64 EEG and 16 fNIRS channels was used to collect data from 16 participants during overt and imagined speech tasks. Bimodal decoding outperformed both unimodal methods for both overt and imagined speech, with an average decoding performance of 46.41% for bimodal, 43.83% for EEG unimodal, and 32.62% for fNIRS unimodal. The fusion approach used in this study included concatenating the feature extractions from separate sub-networks for each modality, followed by additional processing in subsequent layers. This is an example of intermediate fusion, one type of fusion approach.

Different fusion approaches can be categorized into early, intermediate, or late fusion. Early fusion combines the original input data before it is fed into the decoder architecture, intermediate fusion seeks marginal data representations, and late fusion combines decisions from unimodal architectures. Early and intermediate fusion can be valuable for finding cross-modal relationships between the two modalities. These inter-dependencies may provide extra information for decoding. An example of early fusion is fMRI-informed EEG source localization to enable decoding of music from EEG signal [161]. Here, the locations of significant voxels in the fMRI data are identified on a source model which is used to create the EEG features. As well as facilitating decoding on participants whose fMRI and EEG had been recorded, this approach enabled decoding on a separate dataset of only EEG recording.

We recommend further exploration of EEG fusion with fNIRS or fMRI to enhance decoding performance, especially techniques like fMRI-informed EEG source localization which holds promise in BCI applications. This approach leverages the advantages of fMRI while remaining practical for EEG-only contexts. Unfortunately, for visual information, there is a lack of datasets that provide either simultaneous or non-simultaneous acquisition of multimodal neuroimaging data. Therefore, creating an open-source dataset for researchers to investigate optimal data representation, feature extraction, and fusion techniques is a logical next step.

10. Overall evaluation and recommendations

Using EEG for decoding visual information is gaining traction for BCI applications, and also for furthering our understanding of the brain. There are appealing reasons for selecting EEG, it is noninvasive, portable, is relatively cheap and has high temporal resolution, Yet, this growing interest is matched with concerns about the feasibility of using EEG for these purposes. The last review of decoding visual information from EEG was in 2015 [1], and one prior to that on fMRI encoding and decoding in 2011 [162]. As a substantial amount of new work has emerged since this time, this current work provides a more up to date review to account for the new developments in this field, so researchers can make informed choices about using EEG for visual information decoding purposes.

In this review paper, we evaluated the feasibility of using EEG by examining the capacity of the hardware, and identifying the current state of the art for decoding via EEG both simple visual information and complex information. Here we summarize the conclusions drawn around feasibility and reiterate methodology recommendations for future researchers to consider if they choose to select EEG as their neuroimaging technique for visual information decoding.

EEG has low spatial resolution and is considered limited in recording from subcortical regions. Yet visual encoding and processing involve fine-grained retinotopic mapping [94] and subcortical regions [42]. There is emerging evidence that scalp-EEG has the power to measure from subcortical regions [99], though we cannot claim that this will be meaningful information in the context of visual information decoding, it does hint at further unlocked potential in EEG that can be explored and exploited to boost decoding performance. Whilst EEG cannot compete with the high spatial resolution of invasive techniques such as ECoG, some improvements can be made to boost EEG’s resolution. For example, MRI acquisition can be used for source localization. Though, we also note that as MRI scans are generally expensive and non-portable, this measure may negate the practical benefits of choosing EEG in the first place.

Through reviewing previously used datasets, we demonstrate that researchers have selected complex naturalistic images, basic images such as digits and letters, and low level visual features such as colour for decoding. The simple features we considered in this review include colour, texture and spatial information (orientation, position and shape). Perceptual decoding has been shown to be possible for all the aforementioned categories [44,45,47,48,50,51,54,70]. Imagination decoding lacks exploration for texture and spatial position and orientation, but attempts to decode colour and shape have shown success [46,49,55]. We recommend future research explore the channel placement and time points to use for decoding more explicitly, as often the optimal spatial and temporal selection have not been fully explored, instead all channels and time-points are used for decoding. For example, a proper comparison of frontal vs posterior channels may indicate whether decoding is occurring based on low-level feature information of high-level categorical information such as verbal labeling. In the context of simple features, this would be whether decoding is occurring from representation in the early visual cortex, or based on categorical representation in more frontal regions, for example, the shape category square vs. triangle. At least for colour, perceptual decoding has been achieved from posterior regions, indicating feasibility of decoding colour from visual information alone. Excitingly there is also feasibility indicated for imagination colour decoding.

The complex stimuli previously used for decoding are varied, including faces, places, objects, tools, characters, and basic shapes and colours. For complex stimuli, most of the decoding work has focused on perception [72–75,77], which in general indicate that this is a feasible aim. Though the impressive decoding performances with naturalistic stimuli are potentially limited by the methodology used to gather this particular dataset [76]. Performance for decoding perceived faces is relatively low – chance was at 50% [73].

When considering what tweaks can be made to existing methodologies to boost performance, and what challenges can be mitigated (see Sections 8 and 9), we begin by returning to an ongoing debate on block vs event-related design, which impacts many previously collected datasets in this domain (Section 8.2). There is merit in using block designs as EEG-based visual information decoding is still in its early stages. However, we caution researchers that block design may hamper ecological validity. These limits of block design should be noted, and event-related designs should be considered when possible.

We next drew attention to individual differences in visual imagery, such as vividness and other nuanced characteristics (Section 8.1). Visual imagery is a noisy experience, differing not just across individuals but also within individuals over time. We recommend using the VVIQ in the participant screening phase to identify individuals’ visual imagery abilities, as well as noting characteristics of their experience such as faded edges or distorted perceptions. Through this process, it may be ascertained that visual imagery is not a suitable BCI command choice for a particular individual, and alternatives can then be considered.

Not just imagery but also visual perception is a noisy experience, therefore, to achieve high decoding performance it becomes important to reduce extra neural noise and confounding inputs. During visual perception, we recommend using fixation crosses to reduce confounding eye movements, which is a technique used in the majority of past studies (Section 8.4). With visual imagery, eye movements may be a functional part of image generation, and disrupting eye movement can reduce imagery quality. Researchers should take this into consideration, particularly in studies that measure both perception then subsequent visual imagery of that same image. Although this will also depend on whether or not the imagery task involves eye closure. We recommend reducing confounding external perceptual inputs, other than the stimuli, in both perceptual and imagery tasks. In some previous studies, researchers have addressed this for visual imagery tasks through eye closure or using a pitch black room. Darkening the testing room is not commonly done for visual perception tasks, and we recommend that researchers consider it in future studies. Researchers should weigh up the two strategies, noting that eye closure is more naturalistic than a pitch black room but also induces strong alpha and related motor signals [163] which can add unwanted noise to the neural signal.

Another important design choice for reducing confounding and noisy neural signals is using palette cleansers (Section 8.5). We recommend using unstructured, dynamic visual noise before and after every perception and imagery task. This helps remove lingering signal left over from previous tasks or other irrelevant internally generated content.

We also offer data decoding recommendations. EEG datasets are often small due to time, money and participant concentration constraints. We recommend synthetic data generation for augmenting EEG datasets (Section 9.1, notes some popular algorithms choices for this). This has been shown to enhance performance decoding in other EEG domains, however, has not yet been applied to visual information decoding. Additionally, future researchers can explore fusion with other neuroimaging techniques, for example complementing the high temporal resolution of EEG with another high spatial resolution neuroimaging technique such as fMRI.

11. Conclusion

In summary, decoding visual information has interesting use-cases for BCIs and understanding neural representation of visual imagination and perception. EEG is an appealing modality for doing this due to its low cost, portability, the facts its noninvasive, and high temporal resolution. As described in this review there are concerns over the feasibility of using EEG for visual information decoding due to its low spatial resolution. It is essential to choose a neuroimaging technique that has the capacity to acquire meaningful information for decoding visual features or high-level categories. The current review evaluated the feasibility of decoding visual information from EEG data first from a theoretical perspective, by assessing the capacity of EEG hardware in comparison to other neuroimaging techniques, and second by evaluating what types of visual information have already been successfully decoded with EEG. Finally, potential problems can arise from the methodology used to gather the datasets and the dataset size we proposed solutions for these, and discussed the merits of fusing EEG data with another neuroimaging technique with higher spatial resolution.

Acknowledgements

We thank Max Townsend at University of Leeds for his valuable comments on earlier iterations of this work. We acknowledge our funding bodies.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Holly Wilson’s research is funded by the UK Engineering and Physical Sciences Research Council (EPSRC). Eamonn O’Neill and Michael J. Proulx’s research is supported and partly funded by the UKRI Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA 2.0; EP/T022523/1) at the University of Bath.