I Know What You're Thinking; Can Neuroimaging Truly Reveal Our Innermost Thoughts?

Abstract

Advances in neuroimaging, combined with developments in artificial intelligence software, have allowed researchers to noninvasively decode the brain and ‘read the mind’.

For decades, reading the mind or ‘decoding the brain’ fell into the realm of science fiction. In recent years, however, the concept has left the pages of books and entered the lab. With the right technology, scientists are able to elucidate the thoughts of individuals without a word being uttered. Implantable electrodes in the brain have allowed those living with paralysis to speak for the first time, the electrodes converting brain activity into text on a screen [1]. Similar technology has enabled those living with paralysis to use a computer [2] or control a robotic limb using only their thoughts [3]. Brain–computer interfaces like these are seen by many as the future, and even commercial companies such as Neuralink and Meta (both CA, USA) are working on variations of the technology that can be adapted for commercial use [4,5].

However, as it stands, this technology is highly invasive and requires a complex surgery to insert the electrodes. Looking for a noninvasive approach, researchers turned to functional magnetic resonance imaging (fMRI). Over the last few decades, research teams have developed this technique, enhancing our knowledge of specific regions of the brain and using this information to interpret meaning from patterns of neural activity. Now, with the integration of artificial intelligence (AI), could a noninvasive technique truly decode the brain and ‘read the mind’?

Mapping the brain to read the mind

In its more basic form, the principle of using fMRI to ‘read the mind’ relies on linking specific feelings, experiences or even words with specific areas or networks in the brain. By mapping brain areas, researchers were able to associate meaning to the activation of a given region or network – if the so-called pain matrix is activated, the person is likely in pain [6]; if amygdala and insula are activated the person is likely scared [7], and so on.

Utilizing this approach to communicate with patients with disorders of consciousness, one research team took advantage of the distinct brain regions for motor control and spatial awareness to develop a rudimentary form of mind reading. In the 2010 study, patients were asked a series of questions about their family. To answer, they were instructed to visualize playing tennis for ‘yes’ or walking around their home for ‘no’. Thinking about tennis activated a movement-focused region of the brain (the supplementary motor area), whereas visualizing their home activated a region associated with spatial awareness (the parahippocampal gyrus). The technique was game changing in its field, and had the potential to restore otherwise unresponsive patients with the ability to communicate [8].

The technique of mapping specific activities or words to regions of the brain accelerated our understanding of the brain and its areas. One study even developed a so-called ‘Brain Dictionary’ – detailing which regions of the brain responded to individual words [9]. However, the process was slow going and real-time ‘mind reading’ seemed off the cards – until the integration of AI changed things.

Painting a mental picture

Despite being around for some time now, 2022–2023 has seen a boom in the use of AI-based software. While such programs are yet to be wholly accepted by the general population, neuroscientists have embraced the potential AI-based software can bring. For many, the computing power enabled by AI allows us to come close to mimicking, and thereby understanding, the vast power of the brain.

Publishing their results on bioRxiv in March 2023, one Japan-based research group utilized the text-to-image generative software Stable Diffusion to generate de novo images from fMRI scans [10]. Similar to the popular DALL-E 2, Stable Diffusion is trained on a bank of images with text descriptions; it can then draw from this bank of examples to create a de novo image based on any given prompt. To adapt this software to recognize a neuroimaging-based prompt, the team trained the algorithm with an online data set of brain scans from 4 individuals who each viewed 10,000 images [10,11].

Once trained, the algorithm was tested on a new set of brain scans, the goal being to recreate the image the individual originally viewed. Utilizing the patterns of brain activity –predominantly those from the occipital lobe, a region associated with registering image layout and perception – the algorithm could recreate an abstract version of the image in question; yet it struggled to form more concrete recognizable objects. To overcome this and create more identifiable images, the researchers linked text descriptors of the images to the patterns of neural activity elicited when said images were viewed [10,11].

The keywords provided in the text descriptors meant that Stable Diffusion could draw on its vast databank of objects and ensure that the correct item was included in the resulting image. This, combined with the layout and perceptual information from the occipital lobe, meant that the final image was a very close approximation of the original source (Figure 1) [10].

Figure 1. (A) Stable diffusion recreated images from original source.

(B) Neural activity alone created an abstract image with similar layout but lacked identifiable contents. (C) Adding in text descriptors created an image of the desired object, but lacked similarity to the original. (D) A combination of neural and text information resulted in a close approximation of the original image.

Reprinted from [10].

Further refinement is needed, and the model requires testing on a larger sample than the scans of the 4 individuals included in the present study. That being said, for preliminary work the results are impressive, and open the door for a wide range of potential applications. The study authors hope that the technology could be used to intercept and interpret thoughts and dreams, or allow scientists to view how other animals perceive reality [11].

Brain like an open book

In May 2023, scientists from the University of Texas at Austin (TX, USA) took it to the next level, using a combination fMRI and AI to convert thoughts into text in real time – a feat some have called true mind reading. Publishing their results in Nature Neuroscience, they showed that their ‘brain decoder’ could translate thoughts with surprising accuracy, getting the gist and meaning of a sentence if not a word-for-word transcript [12].

To train the decoder on patterns of brain activity, study participants listened to nearly 16 hours-worth of podcasts while an fMRI scanner generated a bank of scans. The resulting images were used to generate maps that predict how each participant reacts to specific phrases and concepts [12,13].

In all studies attempting to measure real-time neural activity, researchers face the challenge of the inherent time lag that occurs with fMRI. fMRI uses blood flow as a proxy for neural activity, measuring the increase of blood flow that occurs in a specific region of the brain when that area is activated [12]. This physiological process peaks and returns to normal over a period of around 10 seconds, a time not even the most advanced scanner can improve on. For a native English speaker, this can mean that each image could represent up to 20 words [12].

To help overcome this, the researchers integrated the natural language program GTP. The language software was used to predict what word might come next, allowing full sentences to be generated upon the first formation of a thought. These predicted sentences were then combined with the fMRI-generated maps for neural activity to create predictions of neural activity for that sentence. The system then compared the set of predicted brain responses to that actually recorded and selected the best candidate for the true sentence [12,13].

Once the algorithm was trained, it was put to the test. Participants listen to a new podcast, and the algorithm put together a prediction of what they were listening to based purely on their neuronal activity. While not a perfect word-for-word recreation, the algorithm was able to generate the gist of the story and convey its key messages [12,13]. “Our system works at the level of ideas, semantics, meaning,” explained corresponding author Alexander Huth (University of Texas at Austin). “This is the reason why what we get out is not the exact words, it's the gist.” [14].

Similar results were generated when the participants watched a sound-less video, with the output giving a close approximation of the action on the screen. The authors proposed that this suggests the system works at a deeper level than just language and could help in uncovering the secrets of how we comprehend the world around us [12,13].

“For a noninvasive method, this is a real leap forward compared to what's been done before, which is typically single words or short sentences,” commented Huth; “We're getting the model to decode continuous language for extended periods of time with complicated ideas.” [14].

Implications for this technology are wide reaching, particularly for individuals otherwise unable to communicate, such as those affected by stroke or other neurological conditions.

We can, but should we?

Commenting on the results, bioethicist Nita Farahany (Duke University, NC, USA) highlighted that privacy is a key ethical concern with this sort of advancement; “We need everybody to participate in ensuring that [the development of safeguards against misuse] happens ethically. … [The technology] could be really transformational for people who need the ability to be able to communicate again, but the implications for the rest of us are profound.” [13].

As it stands, consent is critical for results – therefore it is unlikely anyone will be reading your mind without your say so. Cooperation with the activity and focusing on language is required for the algorithm to work – counting to ten rather than listening to the podcast would give null results. Further, we are unique in our brain activity, therefore an algorithm trained on the data of one person would not be able to decode the neural activity of another. For the decoder to work on you, you would first need to undertake hours of reference scans [12,15].

“I don't think that this is going to be used to violate people's privacy,” commented Francisco Pereira of the National Institute of Mental Health (MD, USA), a scientist who has worked on brain decoders for decades and was not involved in the study; “Or if it is, and people are in a situation where they can compel you to be in the scanner for 16 hours and somehow get you to think about things, you're already in a pretty compromised situation.” [15].

However, the study's authors highlighted the potential dangers that this technology can hold if misused. They noted that, as technology advances, future iterations of this software may be developed that mitigate the need for participant cooperation. Data could also be “misinterpreted for malicious purposes,” incorrectly attributing thoughts or opinions to individuals [12].

Commenting on the study, BioTechniques Editor-in-Chief Michelle Itano noted: “As imaging data become more and more complex and widespread, it is also critically important to ensure that data storage, sharing and analysis infrastructure is broadly available. It is also becoming more and more critical that early career researchers have access to educational resources for training in how to understand and best utilize these powerful technologies in their research.”

Whether this technology should have stayed confined to fiction books is a topic of hot debate, with the potential for what future iterations of this technology may hold causing concern. However, you don't need to be a mind reader to tell that AI and brain–computer interface technology is well and truly here to stay.