16,629
Views
63
CrossRef citations to date
0
Altmetric
Original Article

Virtual reality as a clinical tool in mental health research and practice


La realidad virtual como una herramienta clínica en la investigación y práctica de la salud metal

Quand la santé mentale a recours à la Réalité Virtuelle en recherche et en pratique clinique

, , , &

Abstract

Virtual reality (VR) is a potentially powerful technology for enhancing assessment in mental health. At any time or place, individuals can be transported into immersive and interactive virtual worlds that are fully controlled by the researcher or clinician. This capability is central to recent interest in how VR might be harnessed in both treatment and assessment of mental health conditions. The current review provides a summary of the advantages of using VR for assessment in mental health, focusing on increasing ecological validity of highly controlled environments, enhancing personalization and engagement, and capturing real-time, automated data in real-world contexts. Considerations for the implementation of VR in research and clinical settings are discussed, including current issues with cost and access, developing evidence base, technical challenges, and ethical implications. The opportunities and challenges of VR are important to understand as researchers and clinicians look to harness this technology to improve mental health outcomes.


La realidad virtual (RV) es una tecnología potencialmente poderosa para mejorar la evaluación en salud mental. En cualquier momento o lugar, las personas pueden ser transportadas y quedar inmersos en mundos virtuales interactivos que están totalmente controlados por el investigador o el clínico. Esta capacidad es parte fundamental del interés reciente por la manera en cómo se podría aprovechar la RV tanto en el tratamiento como en la evaluación de las condiciones de salud mental. Esta revisión proporciona un resumen de las ventajas del empleo de la RV para la evaluación en salud mental, enfocándose en el aumento de la validez ecológica de entornos altamente controlados, en mejorar la personalización y el compromiso, y capturar datos automatizados en tiempo real en contextos del mundo real. Se discuten las consideraciones para la implementación de la realidad virtual en investigación y en situaciones clínicas, incluidos los problemas actuales de costo y acceso, el desarrollo en base a la evidencia, los desafíos técnicos y las implicancias éticas. Es importante comprender las oportunidades y los desafíos de la realidad virtual, ya que los investigadores y los médicos buscan aprovechar esta tecnología para mejorar los resultados de salud mental.

L’évaluation de la santé mentale pourrait bénéficier de la puissance de la technologie de la réalité virtuelle (RV). Les sujets peuvent être transportés en immersion à tout moment ou en tout lieu dans des mondes virtuels interactifs totalement contrôlés par le chercheur ou le médecin. C’est ce qui motive l'intérêt récent sur la façon d’exploiter la RV à la fois dans le traitement et l'évaluation des maladies mentales. Notre présentation résume les avantages de la RV dans ce cadre ; elle est centrée sur la validité écologique renforcée des environnements très contrôlés, sur une meilleure personnalisation et participation du sujet, et sur la saisie de données automatisées en temps et contextes réels. Nous analysons les conditions de l’introduction de la RV en recherche et en clinique, y compris les enjeux actuels de coût et d'accès, le développement d’une base de données, les défis techniques et les questions éthiques. Chercheurs et médecins souhaitent expérimenter la réalité virtuelle pour améliorer les résultats dans le contexte de la santé mentale, il est donc important d’en comprendre les opportunités et les difficultés.

Introduction


Developments in virtual reality (VR) have the potential to radically transform the landscape of assessment in mental health. Immersive VR involves wearing an enclosed head-mounted device (HMD) that displays three dimensional images on a screen so that the person is fully immersed in a virtual environment (eg, Figure 1 ).Citation1 Images are continuously rendered relative to the position of the head and can capture movements of the body, allowing users to explore and interact with objects and avatars (digital agents) in the virtual space. These virtual environments are either programmed using specialist software to create computer-generated, photorealistic images, or filmed with specialized cameras to create 360-degree videos of real-world scenes that can be replayed within VR. Together, these capabilities make it possible for researchers and clinicians to observe and record individuals in highly controlled and near-natural environments, in real time.


Figure 1. Examples of virtual reality environments 
(King’s College London, IoPPN Virtual Reality Research Lab, Developer: Jerome Di Pietro).
Figure 1. Examples of virtual reality environments 
(King’s College London, IoPPN Virtual Reality Research Lab, Developer: Jerome Di Pietro).

VR has been applied for the delivery of exposure-based treatments, whereby individuals can experience feared situations or contexts in a safe and controlled manner, without leaving the clinical setting. Indeed, VR exposure treatments have proven effective across a range of mental health conditions. A number of reviews have been written on the topic of VR-based treatments for psychiatric conditions more broadly.Citation2-Citation7 Freeman et alCitation4 conducted a systematic review in 2017, finding 154 studies on VR treatments for a range of mental health disorders. Further reviews have been published on VR treatments for schizophrenia,Citation5 anxiety disorders,Citation2,Citation8 and mental disorders more generally.Citation3,Citation6 Evidence for the efficacy of VR treatments is noted within these reviews; however, the methodological quality of studies is generally low and the implementation of VR treatments beyond research settings is yet to be examined. Exposure therapy is notably dominant within this literature, highlighting how the ability of VR to recreate real-world environments has been leveraged to target the mechanism underpinning exposure. However, in general, innovation has not moved far from this approach, with more novel clinical applications of VR yet to be explored. One such application of VR within treatment is its use as a clinical assessment tool, a topic of relatively limited discussion within the literature. Clinical assessment is an integral element of mental health treatment, from diagnosis to treatment planning and monitoring. As such, the aim of this review is to orient the reader to the clinical application of VR as an assessment tool within mental health research and practice. The specific capabilities that underlie the utility of VR for clinical assessment is provided, as well as examples from high-quality research studies, finishing with a discussion of current considerations and limitations in the field.


Benefits of virtual reality for assessment


From symptoms and cognition to functioning and capacity, the measurement of psychological phenomena is central to research and practice in mental health.Citation9 Although we can have confidence in current assessment instruments to a certain degree, there are many threats to reliability and validity.Citation10 Differences between the assessment context (eg, a lab or clinic) and the real world generate multiple sources of potential bias, threatening the accuracy of results. Real-world assessments are possible, but can be costly and time consuming, and access can be limited by location and mobility. VR may overcome many of these limitations through the ability to generate highly controlled, real-world experiences.Citation11

Enhancing ecological validity


With technological advances over the past decade in particular, VR has become increasingly immersive. By immersing individuals in real-world situations through VR, it is possible to conduct assessments that more closely emulate what happens in daily life. This capability overcomes the issue of ecological validity, that is, the degree to which the findings of research studies generalize to real-world settings.Citation11 Research has consistently shown that individuals respond to virtual environments as if they were experiencing them in real life.Citation12-Citation15 Virtual environments are known to produce physiological changes consistent with emotional responses to real-world scenariosCitation13,Citation16,Citation17 and have the ability to elicit symptoms such as paranoia,Citation18,Citation19 cravings,Citation20,Citation21 anxiety,Citation14,Citation15,Citation22 and fear.Citation23 A meta-analysis of nine randomized controlled trials comparing VR exposure with in vivo exposure for phobias found equivalent effects for both interventions, suggesting similar processes occur.Citation24 Experiences in VR have also been found to elicit predictable behaviors, with one study finding that people with higher levels of paranoia kept a greater interpersonal distance from avatars within a virtual environment, which was considered a reflection of perceived trust and social threat.Citation25 These findings highlight one of the main advantages of VR within mental health: the ability to simulate experiences in everyday life.


Whereas validation studies are still lacking, research has demonstrated that VR-based assessments can perform comparatively to assessments conducted in the real world.Citation12,Citation26-Citation28 Gorini et alCitation12 examined emotional reactions to food in a virtual kitchen, a real kitchen, and in photographs, among individuals with eating disorders. They found that both real and virtual exposure to food cues elicited the same emotional reactions, which were greater than those elicited by the photographs. VR also enables access to situations and experiences previously difficult to attain in research, such as hard to reach or dangerous environments. For example, VR has been used in neuroimaging research to study brain activation in naturalistic scenarios, a method previously impossible from inside the scanner.Citation29 Furthermore, since recent developments have resulted in VR becoming completely mobile, assessments do not need to occur within the clinic or lab, allowing people to access them remotely.Citation30 The possibility to deliver automated assessments in people’s homes, independent from a clinician, is an exciting opportunity to increase efficiency, improve accessibility, and reduce cost. 


Control and manipulation of the virtual environment


VR offers the ability to control and manipulate features of the environment that can be used to test and assess relevant variables, such as eliciting paranoia in social situationsCitation18,Citation31,Citation32 or examining responses to cues within the environment.Citation12,Citation20,Citation21 Experimental control is a cornerstone of psychological research, enabling direct comparison between conditions to determine causal relationships between variables. Strong methodological rigor can be achieved in VR through careful manipulation of variables across conditions in virtual environments.Citation11 For example, Veling et alCitation31 randomized participants with psychosis, siblings, and health controls to conditions with varying levels of social stress within a virtual environment. They found a dose-response relationship between social stress and paranoia, which was associated with psychosis liability, supporting the theory that social stress may account for the relationship between the environment and psychosis. Another example comes from Freeman et alCitation33 who examined experiences of paranoia and perceptions of social rank within a virtual underground train. Participants were placed in two conditions: one where they were taller than others on the train and another where they were shorter. Findings showed that under the condition where they were shorter, participants had a more negative view of themselves relative to others and greater levels of paranoia. The differences in paranoia were fully mediated by social comparison, suggesting that negative views of the self, relative to others, may drive feelings of mistrust.


Another capability of VR is that individuals are able to interact with objects within the system, rather than simply observing or imagining different scenarios. This not only enhances ecological validity, as interaction is inherent to the real world, but also allows researchers to examine behaviors of individuals within the virtual environment and their impact by manipulating different contingencies.Citation34 Previously, this type of research was only possible using actors or “confederates” who performed certain roles within situations, which was costly and limited in ecological validity. Furthermore, the controlled nature of the experience enables greater reproducibility relative to field studies where the environment is constantly changing. These capabilities have important implications for social psychology,Citation35 but also for identifying differences characteristic of mental disorders.Citation5 For example, previous studies have demonstrated that exposure of each participant to the same virtual scenario can allow controlled examination of the determinants of paranoid ideation.Citation18,Citation33,Citation36,Citation37 As such, the enhanced environmental control and interactivity of VR allows for standardization of otherwise dynamic variables, ensuring a consistent assessment experience, both over time and across individuals.


Personalization and tailoring


Because virtual environments are computer-generated or recorded, it is also possible to program tailored VR experiences that match individual needs, abilities, or preferences (eg, slowing down a sequence, using text or audio instructions, minimizing distraction). Rizzo et alCitation38 developed a VR-based exposure therapy for posttraumatic stress disorder (PTSD) that enabled the therapist to customize various features of a combat scenario most relevant to the trauma experienced by soldiers. Since effective treatment of PTSD requires exposure to cues of highly idiosyncratic experiences, customization of the virtual environment is an important feature. Such customization also has important utility for assessment purposes, though very little research has examined this to date. For example, functional analysis involves the examination of how symptoms change in relation to different triggers and responses.Citation39 VR could be used to examine changes in symptoms across different situations and in relation to certain cues, for example visual, auditory, and olfactory, allowing precise insight into the determinants of relevant clinical events. Gatti et alCitation40 describe a VR protocol whereby clinicians were able to customize the virtual environment with personally relevant cues to alcohol craving and behaviors for the purposes of clinical formulation. The ability to perform psychiatric assessment via personalized tests and tasks meaningful to the individual and their situation is powerful but is currently understudied within VR.


Real-time, automated data capture


The benefits of using technology for automatic data capture has been well recognized.Citation41 Mobile apps for tracking symptoms and other clinically relevant information over timeCitation42 has been a major area of progress in digital mental health, with findings suggesting that they are feasible and acceptable to individuals.Citation43,Citation44 Researchers are starting to consider how data collected from devices such as smartphone apps and VR might be clinically informative. Patterns in data collected from technology devices, such as movement, speech, and geolocation, have been associated with changes in symptomsCitation45 and may even be used to predict relapse among people with psychosis.Citation42

It is possible to capture data automatically from users during VR experiences. Eye-tracking software can be integrated with VR,Citation30 capturing a source of data commonly used to identify markers of psychiatric disorders and cognition.Citation46 It is also possible to measure behaviors within the virtual environment, such as decisions about how to navigate and interact with different objects and agents.Citation25 Capturing the temporal relationship between variables in real time (ie, how thoughts, emotions, and behaviors unfold in relation to changes in the environment), makes it possible to test hypothesized processes and causal interactions. Furthermore, physiological measures commonly used as objective indicators of psychological states, such as heart rate and galvanic skin response, can be recorded and synced with virtual content or even eye gaze. Mühlberger et alCitation47 developed a VR-delivered behavioral avoidance test (VR-BAT) to assess fear in specific phobia while automatically collecting heart rate, skin conductance, subjective rating of discomfort, and stimulus-approach distance. Results indicated that the physiological measures were good predictors of fear intensity. Others have begun to investigate integrating more sophisticated biosensors, such as wireless electroencephalography (EEG), with VR to assess psychologically relevant constructs such as emotional state, with the aim to feed the data back to clinicians for enhanced decision makingCitation48 or even within VR itself as biofeedback.Citation49

Increasing engagement


Another benefit of delivering clinical evaluation via VR is the potential to enhance an individual’s engagement with the test or assessment. Traditional testing procedures undertaken in mental health can be lengthy, repetitive, and/or laborious, which may impact individual performance, especially for measures of attention or memory. Therefore, replacing traditional assessment with more engaging, meaningful, or enjoyable methods has substantial appeal. This is particularly important given that symptoms such as poor concentration and lack of motivation are common across mental health conditions.Citation50 This application of VR may also be of benefit in young people experiencing mental ill-health, a population where digital technologies are common and their use for mental health is promising.Citation51

VR may enhance engagement with clinical evaluations through the immersive, realistic, enjoyable environments enabled by the technology.Citation52,Citation53 The term “presence” has been used in VR research to refer to the subjective experience of being in a place or environment, even when one is physically situated in another.Citation1 The feeling of presence is influenced by the degree of immersion, defined as the extent to which the system generates sensory stimulation consistent with sensorimotor experience (eg, images are updated as the head moves). Sense of presence has traditionally been linked to an individual’s level of engagementCitation54 and motivation.Citation55 As such, high rates of presence and immersion reported within investigations of clinical uses of VR point to the ability of the technology to enhance engagement with clinical evaluation.Citation56-Citation58 However, it is currently difficult to create VR environments with a high level of realism, which can create a strong aversion to the experience.Citation59

VR could further increase engagement with clinical evaluation by adding elements of digital games to the assessments, a process known as “gamification.”Citation60 Incorporating features of games, such as rewards and feedback, within VR may engage individuals more fully in the evaluation process, limiting distractions that may compete for attention. Pollak et alCitation53 reported that young participants (aged 9 to 17) rated a VR attention-deficit/hyperactivity disorder (ADHD) assessment significantly more enjoyable than a computerized test of attention. The affordances inherent within VR technology have the potential to increase engagement with clinical evaluation for all individuals across presenting disorders, potentially improving the reliability and validity of the assessments.


Applications of VR for assessment in clinical research studies


Owing to the ecological validity, immersive capabilities, and ease of standardized data collection as discussed above, the field has begun to explore the use of VR for the assessment of clinically relevant outcomes. To date, these have fallen into three main areas: social functioning, cognition, and symptomatology. Social functioning can be assessed using automatic data capture such as eye gaze, proximity to VR avatars, and recording of responses to simulated social situations.Citation25,Citation56 Across studies, individuals with mental health conditions known to impact social functioning differed from control participants on VR-recorded measures. Cognition was among the first outcomes to be assessed with early-stage VR programs,Citation61-Citation63 focusing on memory and executive function, commonly assessed using maze navigation and attention tasks. For example, response to a VR-administered continuous performance task assessed attention and response inhibition in children and teens with ADHD compared with controls.Citation64 Finally, VR environments have also been used to elicit and assess symptoms, such as paranoid ideation in the general population,Citation19,Citation65 individuals at risk for psychosis,Citation32,Citation37 those with a first-episode psychosis,Citation57 and individuals with long-standing persecutory delusions.Citation66 Other symptoms such as auditory hallucinations,Citation67 disordered eating,Citation68 addiction,Citation69 and phobiaCitation47 have also been studied using VR.


Considerations and limitations


Whereas VR offers exciting opportunities to advance multiple areas of mental health, it is important to remain cautiously optimistic. Mirroring broader issues within the field of digital mental health, the following are major areas where caution is needed.


Cost and access


Although VR has been around for several decades, only recently has the technology advanced to the point of commercial readiness. A major milestone occurred in 2010 with the release of the Oculus Rift , a relatively affordable VR device directed at consumers. Prior to this, VR technology mainly existed behind the closed doors of software companies and specialized research labs.Citation1 Over the past decade, we have seen rapid technological advancement and proliferation of the marketplace, with various companies offering a range of devices suited to different consumers and budgets. Critical to VRs viability, 2019 saw the release of the fully mobile HMD Oculus Quest .Citation70 Mobile HMDs can be run without a cable connecting them to a computer, overcoming earlier problems with mobility and ease of use. It is now easier to imagine VR becoming commonplace in clinics, hospitals, and people’s homes because it is easier to set up and more convenient to use. However, the cost of these devices is still prohibitive for many, with the most recent version of the Oculus Quest costing around US $400. 


The biggest limiting factor to the implementation of VR into clinical practice at present is the lack of evidence-based VR programs that can be bought off the shelf and used by clinicians and researchers. A number of labs around the world are developing their own software packages and testing them, but they are not yet commercially available. The few commercially available products developed by software companies have not been tested to show whether they are safe and effective. Additionally, because the technology is advancing so rapidly, hardware becomes outdated quickly and proprietary issues limit the availability of VR applications across newer platforms. Consequently, we are yet to see VR have the same market penetration as smartphones, with only a small proportion of consumers currently owning these devices. Some clinicians have integrated VR into their practice,Citation71 but again we are yet to see these treatments become widely available despite good evidence base for some approaches, particularly exposure therapy for anxiety disorders.Citation2 In comparison, there has been a more steady increase in the use of VR within mental health research.Citation72 The slow integration of VR into clinical care is due to many factors. These include a lack in infrastructure to support the technology within services, absence of training and standardized evidence-based VR packages, the learning curve and costs associated with adopting new technologies, and more broadly, fears that technology may hinder engagement or even replace mental health professionals’ roles.Citation73 For these reasons, it is imperative that new VR applications, and in fact any digital technology, are designed with considerations for the systems and context in which they will be implemented.Citation74,Citation75

Developing evidence base


Traditionally, medical research is painstakingly slow at developing new evidence.Citation76 Research is costly, time consuming, and in many ways, inefficient. This is particularly problematic in digital mental health, where research operates at a much slower pace compared with technology development. Therefore, research often lags behind in providing the evidence base necessary to justify the hype surrounding many technologies.Citation74 As a result, the marketplace is largely dominated by mental health technologies that lack the evidence to substantiate their claims.Citation77 VR is no exception, with a booming market of HMDs, games, and applications ready for consumer use.


In addition to the complexities in developing and validating new VR technologies, it is also difficult to maintain them. Updates to software and hardware require ongoing maintenance costs that are not afforded in the traditional funding model offered by research grants. Potential solutions are still unfolding; however, currently, it is rare for the technologies and applications developed by researchers to be widely available for consumers and clinicians.Citation78 They are also often developed in isolation, with collaboration across research centers lacking. Better partnerships across institutions as well as with commercial companies may offer solutions to this problem, but until this time, clinicians and researchers may find themselves frustrated by the lack of readily available VR assessment tools.


Technical requirements and issues


Technology can be complex and unstable, with teams of experts required for effective development and maintenance. Anyone interested in developing new VR applications for research should be warned that considerable technical, design, and computational expertise is required.Citation79 Enhancing ecological validity is a major offering of VR; however, the degree of immersion necessary to generate the sense of presence is a matter of ongoing research. Features that affect this sense of immersion include the display parameters of the VR system (eg, frame rate and resolution), design features (eg, realism of visual objects), and multisensory feedback.Citation1 An interesting phenomenon related to this that is often observed in VR is termed the “uncanny” valley.Citation59 This refers to the way in which humanoid characters in a virtual world can elicit a feeling of unease or revulsion if they look very human-like but are imperfect (eg, their breathing movements are not realistic), reducing the overall realism of the environment. Interestingly, avatars with more cartoon-like feature might overcome this problem as they are easily accepted and do not seem to elicit the same sense of unease as more real-like humanoids.Citation80 Clearly, the design of reliable and immersive VR experiences is complex. This, combined with the computing power and hardware required to build and run VR experiences, is a challenge for those in the field. 


Ethical issues


A number of reviews and commentaries have been written about the ethical landscape of digital mental health.Citation81-Citation83 Common themes across this literature include privacy, confidentiality, transparency, security, and ownership of data. These issues are also relevant to VR, especially when the application involves the collection of personal information and is accessed via the Internet. VR brings about more specific ethical challenges, however. More conceptually, the blurring of realities may have undue consequences on how we relate to and understand the world, perhaps especially so for conditions such as psychosis where reality distortion is already a challenge. As discussed previously, research has demonstrated that experiences in VR can have the same impact as if they occurred in the real world, necessitating considerable caution when conducting VR experiments designed to manipulate behavior.Citation84 Some have commented on the dangers of enabling continual access to alternate realities, potentially perpetuating escape from the discomfort of the real world.Citation85 The side effects of VR—namely, eye strain, cybersickness, and reality distortionCitation1 —are also important considerations, and limited research exists on their long-term effects. In considering these factors, Madary and Metzinger,Citation84 and Rizzo and Schulheis,Citation85 provide recommendations for the ethical use of VR within research, drawing on general principles underpinning respect for rights and protection from harm.


Conclusion


Sufficient evidence has accumulated to support the benefits of VR for a variety of assessment purposes in mental health. VR elicits similar psychological and physiological reactions to real-world environments, extending the reach of current assessments beyond the lab or clinic. Superior capabilities for experimental manipulation and controlled exposure could significantly advance the field of mental health by improving methodological rigor, as well as enabling more accurate and individualized assessment. Automatic data capture of behaviors and signals from VR experiences can reveal important insights that might improve our understanding of mental health conditions and inform more tailored treatments. Advances in hardware, software, and research evidence has progressed in recent years; however, more studies are clearly needed to establish the reliability and validity of VR-based assessments, and issues with access to these resources and ethics require attention, thought, and research as the field develops.


The authors declare there are no conflicts of interest

REFERENCES

  • SlaterMSanchez-VivesMVEnhancing our lives with immersive virtual realityFront Robot AI2016374https://doi.org/10.3389/frobt.2016.00074
  • CarlESteinATLevihn-CoonAet alVirtual reality exposure therapy for anxiety and related disorders: a meta-analysis of randomized controlled trialsJ Anxiety Disord201961273630287083
  • EichenbergCWoltersCEichenbergEVirtual realities in the treatment of mental disorders: a review of the current state of research Virtual Reality in Psychological, Medical and Pedagogical Applications2012 IntechOpen 3564
  • FreemanDReeveSRobinsonAet alVirtual reality in the assessment, understanding, and treatment of mental health disordersPsychol Med201747142393240028325167
  • Rus-CalafellMGaretyPSasonECraigTJValmaggiaLRVirtual reality in the assessment and treatment of psychosis: a systematic review of its utility, acceptability and effectivenessPsychol Med201848336239128735593
  • ValmaggiaLRLatifLKemptonMJRus-CalafellMVirtual reality in the psychological treatment for mental health problems: a systematic review of recent evidencePsychiatry Res201623618919526795129
  • VelingWMoritzSvan der GaagMBrave new worlds—review and update on virtual reality assessment and treatment in psychosisSchizophr Bull20144061194119725193975
  • Maples-KellerJLBunnellBEKimSJRothbaumBOThe use of virtual reality technology in the treatment of anxiety and other psychiatric disordersHarv Rev Psychiatry201725310311328475502
  • BaerLBlaisMAHandbook of Clinical Rating Scales and Assessment in Psychiatry and Mental Health2010 New York, NY Humana Press
  • MyersKWintersNCTen-year review of rating scales. I: overview of scale functioning, psychometric properties, and selectionJ Am Acad Child Adolesc Psychiatry200241211412211837400
  • ParsonsTDVirtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciencesFront Hum Neurosci2015966026696869
  • GoriniAGriezEPetrovaARivaGAssessment of the emotional responses produced by exposure to real food, virtual food and photographs of food in patients affected by eating disordersAnn Gen Psychiatry201093020602749
  • MeehanMInskoBWhittonMBrooksFP JrPhysiological measures of presence in stressful virtual environmentsACM Trans Graph2002213645652https://doi.org/10.1145/566570.566630
  • RivaGMantovaniFCapidevilleCSet alAffective interactions using virtual reality: the link between presence and emotionsCyber Psychol Behav20071014556
  • SlaterMPertaubDPSteedAPublic speaking in virtual reality: facing an audience of avatarsIEEE Comput Graph Appl199919269
  • CostanzoMELeamanSJovanovicTet alPsychophysiological response to virtual reality and subthreshold posttraumatic stress disorder symptoms in recently deployed militaryPsychosom Med201476967067725333498
  • DiemerJAlpersGWPeperkornHMShibanYMühlbergerAThe impact of perception and presence on emotional reactions: a review of research in virtual realityFront Psychol201562625688218
  • FreemanDSlaterMBebbingtonPEet alCan virtual reality be used to investigate persecutory ideation?J Nerv Ment Dis2003191850951412972853
  • RichesSGaretyPRus-CalafellMet alUsing virtual reality to assess associations between paranoid ideation and components of social performance: a pilot validation studyCyberpsychol Behav Soc Netw2019221515930346808
  • LeeJHKuJKimKet alExperimental application of virtual reality for nicotine craving through cue exposureCyberpsychol Behav20036327528012855083
  • KuntzeMFStoermerRMagerRRoesslerAMueller-SpahnFBullingerAHImmersive virtual environments in cue exposureCyberpsychol Behav20014449750111708729
  • OwensMEBeidelDCCan virtual reality effectively elicit distress associated with social anxiety disorder?J Psychopathol Behav Assess2015372296305
  • DiemerJLohkampNMühlbergerAZwanzgerPFear and physiological arousal during a virtual height challenge—effects in patients with acrophobia and healthy controlsJ Anxiety Disord201637303926600469
  • WechslerTFMühlbergerAKümpersFInferiority or even superiority of virtual reality exposure therapy in phobias? A systematic review and quantitative meta-analysis on randomized controlled trials specifically comparing the efficacy of virtual reality exposure to gold standard in vivo exposure in agoraphobia, specific phobia and social phobiaFront Psycol2019101758
  • Fornells-AmbrojoMElenbaasMBarkerCet alHypersensitivity to contingent behavior in paranoia: a new virtual reality paradigmJ Nerv Ment Dis2016204214815226825264
  • KoenigSCrucianGDalrymple-AlfordJDünserAAssessing navigation in real and virtual environments: a validation studyInt J Disabil Hum Dev2011104325330
  • RenisonBPonsfordJTestaRRichardsonBBrownfieldKThe ecological and construct validity of a newly developed measure of executive function: the virtual library taskJ Int Neuropsychol Soc201218344045022339816
  • RoseFDAttreeEABrooksBMParslowDMPennPTraining in virtual environments: transfer to real world tasks and equivalence to real task trainingErgonomics200043449451110801083
  • ReggenteNEssoeJKAghajanZMet alEnhancing the ecological validity of fMRI memory research using virtual realityFront Neurosci20181240829962932
  • JuvrudJGredebäckGÅhsFet alThe immersive virtual reality lab: possibilities for remote experimental manipulations of autonomic activity on a large scaleFront Neurosci20181230529867318
  • VelingWPot-KolderRCounotteJvan OsJvan der GaagMEnvironmental social stress, paranoia and psychosis liability: a virtual reality studySchizophr Bull20164261363137127038469
  • ValmaggiaLRFreemanDGreenCet alVirtual reality and paranoid ideations in people with an ‘at-risk mental state’ for psychosisBJPsych2007191S51s63s68
  • FreemanDEvansNListerRAntleyADunnGSlaterMHeight, social comparison, and paranoia: an immersive virtual reality experimental studyPsychiatry Res2014218334835224924485
  • WieserMJPauliPGrosseiblMMolzowIMühlbergerAVirtual social interactions in social anxiety—the impact of sex, gaze, and interpersonal distanceCyberpsychol Behav Soc Netw201013554755420950179
  • PanXHamiltonAFCWhy and how to use virtual reality to study human social interaction: the challenges of exploring a new research landscapeBr J Psych20181093395417
  • FreemanDGaretyPABebbingtonPet alThe psychology of persecutory ideation II: a virtual reality experimental studyJ Nerv Ment Dis2005193530931515870614
  • ValmaggiaLRDayFRus-CalafellMUsing virtual reality to investigate psychological processes and mechanisms associated with the onset and maintenance of psychosis: a systematic reviewSoc Psychiatry Psychiatr Epidemiol201651792193627262562
  • RizzoASDifedeJRothbaumBOet alDevelopment and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat-related PTSDAnn N Y Acad Sci20101208111412520955333
  • SturmeyPFunctional Analysis in Clinical Treatment2011 Elsevier
  • GattiEMassariRSacchelliCRivaGVirtual reality protocol: an instrument to assess alcohol-dependent individualsAnnu Rev Cyberther Telemed20075563
  • TorousJKiangMVLormeJOnnelaJPNew tools for new research in psychiatry: a scalable and customizable platform to empower data driven smartphone researchJMIR Ment Health201632e1627150677
  • Ben-ZeevDBrianRWangRet alCrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapsePsychiatr Rehabil J201740326627528368138
  • WangKVarmaDSProsperiMA systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disordersJ Psychiatr Res2018107737830347316
  • BellIHLimMHRossellSLThomasNEcological momentary assessment and intervention in the treatment of psychotic disorders: a systematic reviewPsychiatr Serv201768111172118128669284
  • TorousJStaplesPBarnettISandovalLRKeshavanMOnnelaJPCharacterizing the clinical relevance of digital phenotyping data quality with applications to a cohort with schizophreniaNPJ Digit Med201811531304300
  • TrillenbergPLencerRHeideWEye movements and psychiatric diseaseCurr Opin Neurol2004171434715090876
  • MühlbergerASperberMWieserMJPauliPA virtual reality behavior avoidance test (VR-BAT) for the assessment of spider phobiaJ Cyber Ther Rehabil200812147158
  • GaggioliACipressoPSerinoSet alA decision support system for real-time stress detection during virtual reality exposureStud Health Technol Inform201419611412024732491
  • PallaviciniFAlgeriDRepettoCGoriniARivaGBiofeedback virtual reality and mobile phones in the treatment of generalized anxiety disorder (GAD): a phase-2 controlled clinical trialJ Cyber Ther Rehabil20092315327
  • FioravantiMBianchiVCintiMECognitive deficits in schizophrenia: an updated metanalysis of the scientific evidenceBMC Psychiatry20121216422715980
  • BellIHAlvarez-JimenezMDigital technology to enhance clinical care of early psychosisCurr Treat Options Psychiatry201963256270
  • HodgeJBalaamMHastingsSMorrisseyKExploring the design of tailored virtual reality experiences for people with dementia Proceedings of CHI Conference on Human Factors in Computing Systems2018 Montreal, QC, Canada Association for Computing Machinery’s Special Interest Group on Computer Human Interaction (SIGCHI) Paper 514 https://doi.org/10.1145/3173574.3174088
  • PollakYWeissPLRizzoAAet alThe utility of a continuous performance test embedded in virtual reality in measuring ADHD-related deficitsJ Dev Behav Pediatr20093012619194324
  • LombardMDittonTAt the heart of it all: the concept of presenceJ Comput Mediat Commun199732JCMC321https://doi.org/10.1111/j.1083-6101.1997.tb00072.x
  • RetauxXPresence in the environment: theories, methodologies and applications to video gamesPsychNology J200313283309
  • ChoiSHKuJHanKet alDeficits in eye gaze during negative social interactions in patients with schizophreniaJ Nerv Ment Dis20101981182983521048475
  • VelingWBrinkmanWPDorrestijnEVan Der GaagMVirtual reality experiments linking social environment and psychosis: a pilot studyCyberpsychol Behav Soc Netw201417319119523992475
  • SlaterMLottoBArnoldMMSánchez-VivesMVHow we experience immersive virtual environments: the concept of presence and its measurementAnuario de Psicología200940193210
  • MoriMMacDormanKFKagekiNThe uncanny valley [from the field]IEEE Robot Autom Mag201219298100
  • DeterdingSDixonDKhaledRNackeLFrom game design elements to gamefulness: defining “gamification.” Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments2011 Tampere, Finland Association for Computing Machinery https://doi.org/10.1145/2181037.2181040
  • SorkinAWeinshallDModaiIPeledAImproving the accuracy of the diagnosis of schizophrenia by means of virtual realityAm J Psychiatry2006163351252016513875
  • KuJChoWKimJJet alA virtual environment for investigating schizophrenic patients’ characteristics: assessment of cognitive and navigation abilityCyberpsychol Behav20036439740414511452
  • SpiekerEAAsturRSWestJTGriegoJARowlandLMSpatial memory deficits in a virtual reality eight-arm radial maze in schizophreniaSchizophr Res20121351-3848922154760
  • NeguțAJurmaAMDavidDVirtual-reality-based attention assessment of ADHD: ClinicaVR: Classroom-CPT versus a traditional continuous performance testChild Neuropsychol201723669271227258210
  • FreemanDStudying and treating schizophrenia using virtual reality: a new paradigmSchizophr Bull200834460561018375568
  • FreemanDPughKVorontsovaNAntleyASlaterMTesting the continuum of delusional beliefs: an experimental study using virtual realityJ Abnorm Psychol20101191839220141245
  • StinsonKValmaggiaLAntleyASlaterMFreemanDCognitive triggers of auditory hallucinations: an experimental investigationJ Behav Ther Exp Psychiatry201041317918420207344
  • Ferrer-GarciaMGutierrez-MaldonadoJTreasureJVilalta-AbellaFCraving for food in virtual reality scenarios in non-clinical sample: analysis of its relationship with body mass index and eating disorder symptomsEur Eat Disord Rev201523537137826095041
  • DetezLGreenwoodLMSegraveRet alA psychophysiological and behavioural study of slot machine near-misses using immersive virtual realityJ Gambl Stud201935392994430684139
  • Oculus Quest 2019 Oculus Available at:
  • LindnerPMiloffAZetterlundEReuterskiöldLAnderssonGCarlbringPAttitudes toward and familiarity with virtual reality therapy among practicing cognitive behavior therapists: a cross-sectional survey study in the era of consumer VR platformsFront Psychol20191017630800086
  • CipressoPChicchi GiglioliIAAlcañiz RayaMRivaGThe past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literatureFront Psychol20189208630459681
  • RivaGVirtual reality: an experiential tool for clinical psychologyBr J Guid Counc2009373337345
  • MohrDCLyonARLattieEGReddyMSchuellerSMAccelerating digital mental health research from early design and creation to successful implementation and sustainmentJ Med Internet Res2017195e15328490417
  • MohrDCRiperHSchuellerSMA solution-focused research approach to achieve an implementable revolution in digital mental healthJAMA Psychiatry201875211311429238805
  • MorrisZSWoodingSGrantJThe answer is 17 years, what is the question: understanding time lags in translational researchJ R Soc Med20111041251052022179294
  • LarsenMEHuckvaleKNicholasJet alUsing science to sell apps: evaluation of mental health app store quality claimsNPJ Digit Med201921831304366
  • FirthJTorousJNicholasJet alThe efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trialsWorld Psychiatry201716328729828941113
  • LindnerPMiloffAHamiltonWet alCreating state of the art, next-generation Virtual Reality exposure therapies for anxiety disorders using consumer hardware platforms: design considerations and future directionsCogn Behav Ther201746540442028270059
  • de BorstAWde GelderBIs it the real deal? Perception of virtual characters versus humans: an affective cognitive neuroscience perspectiveFront Psychol2015657626029133
  • D’AlfonsoSPhillipsJValentineLGleesonJAlvarez-JimenezMModerated online social therapy: viewpoint on the ethics and design principles of a web-based therapy systemJMIR Ment Health2019612e1486631799937
  • TorousJRobertsLWThe ethical use of mobile health technology in clinical psychiatryJ Nerv Ment Dis201720514828005647
  • WykesTLipshitzJSchuellerSMTowards the design of ethical standards related to digital mental health and all its applicationsCurr Treat Options Psychiatry201963232242
  • MadaryMMetzingerTKReal virtuality: a code of ethical conduct. Recommendations for good scientific practice and the consumers of VR-technologyFront Robot AI201633https://doi.org/10.3389/frobt.2016.00003
  • RizzoASchultheisMTBushSSDrexlerMLEthical issues for the use of virtual reality in the psychological sciences Studies on Neuropsychology, Development, and Cognition, vol 4. Ethical Issues in Clinical Neuropsychology2002 Lisse, Netherlands Swets & Zeitlinger 243277