Interacting with virtual characters, objects and environments: investigating immersive virtual reality in rehabilitation

Abstract

Purpose

This pilot study aimed to (a) investigate opportunities for immersive Virtual Reality (VR) technology in communication, physical, and visual rehabilitation by examining the interaction of people without disabilities in a range of structured virtual environments; and (b) validate research protocols that might be used to evaluate the physical, visual, and verbal interaction of users in virtual worlds, and their safety while using the technology.

Methods

Thirteen adults identifying as people without disability were exposed to VR via a head-mounted display. A video-review method was used to qualitatively code and analyse each participant’s communication, movement, orientation, and support needs.

Results

All participants oriented to their virtual environments sufficiently to use applications. Their spoken language was effective for interaction, although unconventional social behaviours were also observed. Two participants reported minor adverse reactions consistent with mild cybersickness.

Conclusion

The results provide insight into the types of environments and characters that support the greatest communicative, physical, and visual interaction in immersive VR. The tested protocols are useful to assess safety when using VR, and to observe communicative, physical, and visual interaction with virtual environments and characters. Implications for future research and use of VR with people with communication, physical and visual disability are discussed.

IMPLICATIONS FOR REHABILITATION

• Safe use of virtual reality in rehabilitation requires careful assessment. Comprehensive observational protocols were sufficient to manage safety concerns.

• Rehabilitation applications need rapid responsivity in verbal interactions to ensure users are immersed in interactions.

• Non-verbal cues from virtual avatars best direct user attention in the environment.

• Novice virtual reality users largely controlled technology using their dominant hand. They could control VR successfully with one hand.

Keywords:

• Virtual reality
• rehabilitation
• communication
• disability
• physical therapy
• vision

Advances in technology continue to open new avenues for assessment and rehabilitation across medical and allied health fields [1,2]. Virtual reality (VR) is increasingly being used for these purposes, enabling gamification and generating environments in which repetitious tasks can be more enjoyable [3]. As such, potential users of VR in therapy could include people with disability associated with a range of physical, visual, cognitive and communication impairments, activity limitations and participation restrictions.

In this study, we define VR as any computer technology that creates an interactive three-dimensional environment around a user, with which they can interact, and which they can explore from within [4]. We also differentiate non-immersive and immersive VR based on the characteristics of the technology, in that (a) non-immersive VR allows the user to interact with the virtual environment through a flat screen and standard computer interface (e.g., commercial game consoles or a laptop computer); whereas (b) immersive VR is accessed using a head mounted display (HMD) with built-in specialised spatial audio headphones. Its use situates the user fully within a virtual synthetic environment that surrounds them. The simulated environment changes according to the user’s head motion and orientation to mimic the viewing and hearing of the natural world.

Current research reflects a growing trend towards greater use of immersive VR for rehabilitation purposes [3,5]. Indeed, more recent research indicates superior outcomes in the recovery of motor function when patients utilised immersive VR in their rehabilitation, when compared to non-immersive VR [6]. A key feature of immersive VR is that it provides the level of presence needed for skill transference or generalisation [7]. With such immersion, users can feel situated in their virtual surroundings and may be able to create, learn, and apply skills [8] and transfer these into the physical world.

A VR user’s level of immersion is often observable in their visual interactions with the virtual environment; specifically, the area of the environment they observe, and the movements they make to visually assess and attend to their surroundings [9]. To make the virtual world more realistic, the user must be able to interact with virtual people and objects. VR technology allows users to interact via hand controllers which sense their hands and replicate hand movements in the virtual space. Users may also receive tactile feedback through the hand controllers. Some environments increase the level of interaction through voice recognition, permitting spoken communication with virtual characters.

Literature review

The emergence of VR systems in the fields of disability and health should be inclusive and support people with communication, physical, cognitive, sensory, and psychosocial disability receiving rehabilitation [2]. Researchers in physical, cognitive and communication rehabilitation have used a range of VR modalities, including both non-immersive applications, and semi- or fully immersive interventions. However, results have been varied, with recent systematic reviews showing mixed results when VR is used in the treatment of communication [10], cognition [5], and physical function [6].

For immersive VR technology to be usable within rehabilitation settings, effective sociotechnical interactions between users (i.e., clients and health professionals) and the technology are required to achieve an immersive experience that can be used for therapeutic value. Specifically, users must be able to adapt and respond in VR to visually orient to their environment, move within that environment in ways that mimic natural movement, interact with objects, respond to cues and instructions, and communicate with virtual avatars – all without experiencing adverse events such as cybersickness (i.e., a form of motion sickness induced by exposure to extended realities such as VR [11]). The complex relationships between people and technology to achieve optimal task performance can be viewed through the lens of sociotechnical systems theory [12]. This theory acknowledges that people, unlike technical systems such as VR, do not always act in predictable or consistent ways [12]. In rehabilitation, this may affect the effectiveness of intended treatment programs.

Studies involving the use of VR by people without disabilities have noted unpredictable interactions in the virtual environment that could impact the effectiveness of rehabilitation. For example, an analysis of social norms in virtual games highlighted that game users may adhere to patterns of social behaviour unique to the game yet unacceptable in physical real-world interactions [13]. In interviews with 30 adult users of social VR, Freeman et al. [14] identified that “there seems to be no consensus on whether and to what degree social norms in the physical world should also be applied to social VR” (p. 21). As such, interventions targeting communication rehabilitation could face difficulties in rehabilitating appropriate use of language in real social situations.

A range of socio-technical interactions may also impact attention to target details in rehabilitative VR applications. An assessment of 24 novice users of VR found that users favoured visual cues over other methods used to direct their attention and were more likely to miss audio or vibrotactile cues [15]. Extending from this, Liu et al. [16] identified that attentional cues were needed to enhance learning performance in human interactions with immersive VR systems. Lack of attention to performance cues could therefore impede rehabilitation outcomes.

In order to create a successful interaction between people and technology that is fit for rehabilitation purposes, both the performance of the people and the technical system and the interaction of the people with the system need to be optimised [12]. Successful use of immersive VR in health and rehabilitation will rely on (a) “goodness of fit” and usability of the hardware and software applications for the virtual environments, and (b) the ways that people interact with these elements. In taking a sociotechnical approach, this study recognises the importance of these sociotechnical forces when using VR successfully [12].

Study aims

In VR environments, observation protocols are needed to understand complex sociotechnical interactions to ensure that the quality of interactions provide a foundation for rehabilitation. In the absence of prior research examining the use of VR in the field of disability, and to inform future research including people with disability, these protocols, and an understanding of sociotechnical interactions between VR systems and novice users without disability, are particularly important. Therefore, this pilot study had twofold aims:

1. To determine the potential uses for VR in rehabilitation by identifying how novice users of immersive VR technology:

   1. experience cybersickness or any adverse events in VR,

   2. visually immerse and orient to the virtual environment,

   3. interact with characters, objects and the visual environments when immersed in the virtual world, and

   4. need support to engage with the virtual environments using an immersive VR system; and

2. To validate research protocols that might be used to evaluate the physical, visual, and verbal interaction of users in virtual worlds, and their safety while using the technology.

Focusing this study on novice users without disability was important to be able to identify any safety issues, risks, or adverse outcomes associated with interactions with characters and environments in immersive VR, prior to its use with people with disability due to their perceived increased vulnerability in human research ethics protocols. By identifying these concerns, they could be addressed and managed in more vulnerable populations with disability (e.g., people with stroke or developmental disability [2]).

Methods

A mixed-methods design was used to examine the behaviour and interactions of participants immersed in VR, the difficulties they experienced, and the support and instructions needed to facilitate their use of immersive VR. This research was reviewed and approved by the Human Research Ethics Committee of the University of Technology Sydney, Australia (ETH19-3648).

Participants

Convenience and purposive sampling were used to recruit adults (aged 16 years or older) who self-identified as having no neurological conditions or disabilities and as being novice users of VR. Potential participants expressed interest in the research via a webform (n = 22); 13 (59%) provided written informed consent and took part. This sample size enabled the collection of a variety of experiences in terms of VR interactions, provided sufficient observational data for the purpose of a pilot study, and in-depth interview data for an inductive qualitative content analysis with data saturation.

Prior to immersion in VR, participants were screened using a safety checklist to minimise the risk of adverse events occurring during VR immersion. Participants were excluded if they had (i) a history of epilepsy or seizures, (ii) any chronic neurological impairments, or (iii) any significant visual or hearing impairments that were not corrected by eyeglasses or hearing aids. Participation would have been delayed if any participants reported (i) a severe head cold, (ii) any eye or ear infections, (iii) a hangover, or (iv) pregnancy beyond 5 months, however no participants reported these conditions. Ultimately, none of the participants who had given informed consent were excluded from the study.

Setting

All data collection was undertaken at the University of Technology Sydney, Australia, between October 2019 and March 2020.

Procedure

Prior to immersion in VR, participants completed a short demographic questionnaire about themselves and their prior experience with immersive VR and other computer technologies. They received a general introduction and orientation to the research space and the immersive VR equipment, including the HMD, controllers, and safety system that would warn of proximity to walls while in VR.

Participants were screened to determine their risk of experiencing cybersickness. The Motion Sickness Susceptibility Questionnaire (MSSQ; [17]) was used to screen for risk of motion sickness symptoms, with participants’ risk of experiencing illness classified as low risk (below the 50th percentile), average risk (50th to 75th percentile) or high risk (above the 75th percentile). Participants were informed of their level of risk following screening and given the opportunity to withdraw from the study. Participants were also screened after completing each application using the Simulator Sickness Questionnaire (SSQ; [18]) to determine if they were experiencing any symptoms of cybersickness.

Individual participants were immersed in five VR applications and experiences using the Meta (formerly Oculus) Rift® HMD. Meta Touch® controllers allowed participants to interact with virtual environments, objects, and characters. The Meta Rift® HMD was selected over other devices owing to its availability, affordability, and accessibility, heightening opportunities for implementation in rehabilitation facilities. Additionally, all target applications could be accessed using the device. All sessions were conducted with the first author directing the session. Each participant completed one session, lasting approximately 2.5 h. All participants used the same five VR applications. The order of presentation of the applications remained consistent as these provided a gradual progression of the skills required to complete tasks within virtual worlds. Participants used each application for between three and 15 min depending on the time needed to complete the required gameplay tasks. The applications are shown in order of presentation in Table 1.

Table 1. Description of the five VR applications (in order) in which participants were immersed.

Application type	Description	Purpose	Observations
Introductory instructional game	A series of directed activities to introduce controls and VR space	Orient to object and character interaction	Orientation to 360-degree VR space. Grasping, lifting, manipulating objects with one or two hands. Non-verbal communication (e.g., waving).
Cafe simulation [19]	Simulation requiring conversation to order a drink/food. Avatar turns-at-talk controlled by the first author using a linked iPad (a “Wizard-of-Oz” protocol; [20].	Simulated conversational interaction	Orientation to 360-degree VR space. Grasping, passing objects with one hand. Verbal communication.
Short film	A non-interactive animated short film in the full 360-degree VR space, requiring users to track the story.	Attention to narrative	Orientation to 360-degree VR space. Response to visual and audio directional cues.
Interactive short film	An animated interactive film where the user directs parts of the narrative by responding to the main character (a talking seagull)	Participation in computer-led verbal interaction. Attention to narrative.	Orientation to 360-degree VR space. Grasping, manipulating objects with one hand. Non-verbal and verbal communication.
Interactive VR game	Participants took the role of a convenience store clerk in a simulation designed to represent the occupation in a humorous way.	Integrating attention, character and object interaction, problem-solving	Orientation to 360-degree VR space. Grasping, lifting, passing, manipulating objects with two hands. Non-verbal and verbal communication.

During their VR immersion, all participants were observed by the first author who took field notes of key observations. The researcher observed the participants as they interacted with the virtual environment and the interaction was recorded with a video camera in the room. The participant view within the VR HMD was mirrored to a desktop video display, allowing the researcher to observe the participant’s actions and intent within the virtual environment. This mirror view was also recorded. Both recordings were later reviewed, coded, and analysed.

A coding guide/observation protocol was used to define behaviours during observation (see Supplementary File 1). This observation protocol was developed prior to any participants being immersed in the applications based on the aims of the study, discussion between investigators on key factors that would impact sociotechnical VR interactions in rehabilitation settings, and investigator analysis of the applications. It was drafted by the first and final authors and discussed with all members of the research team to arrive at consensus on and refine any of the items prior to use for coding in the study. The protocol included items relating to:

1. Experience of any adverse events in VR space that would limit safety in the context of rehabilitation.

2. Visual immersion and orientation: whether the participant moved their head and body to view the full 360-degree virtual environment.

3. Interaction with objects: grasping, lifting, passing, and manipulating objects, using their left or right hand, and in what locations relevant to their physical body, whether objects were used in conventional ways.

4. Interaction with characters: verbal and non-verbal interaction with virtual avatars.

5. Interaction with the environment: response to visual and audio directional cues.

6. The level and type of support novice users needed to engage with the virtual environments, including the number and type of requests for assistance.

Additional behaviours that were observed but not included in the coding guide were documented in field notes (e.g., words spoken, gestures used).

Following immersion in each application, participants removed the headset and were provided a short break of approximately 10 min to reduce fatigue and minimise the risk of cybersickness symptoms developing from prolonged exposure to VR. All breaks were mandatory and provided equally to all participants. During breaks, participants completed the SSQ, the User Engagement Scale – Short form (UES [21]), a valid and reliable tool for the measurement of user experience in digital domains, and answered open-ended interview questions about their experiences, likes and dislikes of each application. Participants were again interviewed at the conclusion of the immersive VR experience about their impressions, attitudes, and facilitators and barriers to future use of such technology. The findings of the UES and these interviews will be reported separately.

Outcomes and analyses

Data were analysed both qualitatively and quantitatively to provide a thorough understanding of participant interaction with immersive virtual environments. Cybersickness measures were analysed according to the procedures defined in each tool. The MSSQ was scored according to symptom frequency (where “never” = 0 and “frequently sick” = 3). Childhood and adulthood sickness scores were summed to create a raw MSSQ score, and a polynomial equation was applied to identify percentile rank (see [17], for details). The SSQ [18] was analysed quantitatively by counting the number of instances with which each cybersickness symptom was reported per application.

Deductive content analysis [22] was applied using a video review methodology and the pre-determined observational protocol. All video recordings of participant’s immersive VR experiences were reviewed by the first author and coded according to the protocol. Additions to the protocol were made where any observed behaviour by any participant did not fit the existing protocol. These were confirmed with reference to field notes and further discussed and verified with all co-authors. Field notes were reviewed and coded alongside the observation protocol and triangulated with the video analysis to verify its accuracy. To ensure reliability of coding, 10% of the video recordings of immersive VR experiences were reanalysed by a second reviewer. Inter-rater reliability of the coding guide was calculated using Cohen’s Kappa. The interrater reliability was k = 0.77, indicating moderate agreement [23].

The observation protocols for all participants were collated, and interpretations of meaning in the observational data was discussed with all researchers to reduce bias in the analysis. Counts of discrete, observed behaviours were analysed quantitatively using descriptive statistics (counts and percentages). In reporting the observation results, specific examples observed and qualitatively coded are provided to enhance transparency and plausibility in the analysis.

Results

Participant characteristics

Participants were aged between 22 and 49 years (mean = 25 years), and nine identified as female. No participant reported history of epilepsy or seizures, or any hearing impairments. Six participants reported having glasses or contact lenses to correct minor visual impairments, and four chose to wear their glasses or contacts while immersed in VR with the aid of a spacer inserted into the HMD.

All participants reported often using computer technologies such as smart phone applications and social media. Online gaming and gaming consoles were each used by eight participants “rarely” or “sometimes”. Two participants each reported one prior experience with immersive VR, and one reported a prior experience using an augmented reality headset. No participants reported extensive experience with the technology used in this study. As such, all were classified as novice immersive VR users.

Risk and experience of cybersickness

The risk of motion sickness was classified as low for seven participants, average for one participant, and high for five participants. All participants were informed of their risk, and none chose to withdraw from the study. Of the seven participants that were identified as having a low risk of motion sickness, one reported a feeling of fullness/fluid within one ear during immersion in VR, though did not associate this with sickness. No other participant identified as low risk reported any symptoms of cybersickness on the SSQ during the study.

Five of the six participants who scored average or high risk on the MSSQ reported symptoms of cybersickness on the SSQ after immersion. Symptoms were experienced across four of the five applications and included fullness in the head, eyestrain, dizziness/vertigo, general discomfort, and burping. Participants reported between one and five symptoms (of a possible 13 on the SSQ), with the greatest number of symptoms reported after the non-interactive VR short film experience. Most symptoms (87%) were reported as “slight”, with only two participants reporting moderate symptoms of sweating or dizziness/vertigo after high levels of physical exertion during one of the VR experiences. Where symptoms were reported, participants were urged to take a short break until these resolved. Required breaks did not exceed five minutes.

Orientation to the virtual space

Participants were observed to see if they turned to view the 360 degrees around them, and if they lifted and lowered their head to view the space between the ceiling and the floor (see Figure 1). No participant fully oriented to every virtual space in which they were immersed. In all applications, every participant viewed the spaces directly in front of them (front left and front right), and the space to their right. They also viewed the virtual environment at eye-level and table-level in every application. For the 13 participants across five experiences each (N = 65 experiences total), the spaces below (n = 57, 87.69%) behind (n = 53, 81.54%) and above (n = 41, 63.08%) were viewed with lowest frequency.

Figure 1. Observed directions that participants viewed to orient within the virtual space.

A graphic representation of regions around the body in the horizontal and vertical planes. Percentages are given as to what proportion of participants oriented to each space.

All participants rotated from the neck to gain a view of their virtual environment in every application, and all rotated using their hips and feet in at least two of their VR experiences. Rotation of the body from the hips was observed in 95.38% (n = 62) of experiences, with 11 (84.62%) participants doing so in all experiences. Rotation of the body from the feet was observed in 48 (73.85%) experiences, with only four (30.77%) participants rotating the body from the feet in every application.

Limited orientation to the virtual space was observed in two applications, with only two (15.38%) participants viewing the full space in the second (cafe simulation) and final (interactive VR game) applications. Each of these applications involved interaction with virtual characters that were directly in front of the participant. The complete virtual space was viewed by ten (76.92%) participants in the third application (non-interactive short film), and by nine (69.23%) in the first (instructional game) and fourth (interactive short film) applications.

Object interaction and manipulation

The ability of participants to reach for, grasp, and interact with virtual objects within their environment was evaluated in four applications (N = 52 interactive virtual experiences). All participants demonstrated an ability to use the immersive VR system controllers to open and close their hands to grasp objects, with such behaviour observed in 50 (96.15%) experiences. All participants were observed to use both hands to grasp virtual objects at least once during their immersion in virtual environments, though clear hand-dominance was noted in qualitative observations. All participants were observed to drop and throw objects in at least one virtual experience. Participants dropped objects, either intentionally or unintentionally, in 39 (75%) experiences, with the number of participants dropping objects increasing with each subsequent virtual experience. The more complex object interaction of passing a virtual object from one hand to the other was observed in seven (13.46%) experiences and achieved by six (46.15%) participants. Additionally, eight (61.54%) participants used objects in relation to themselves (e.g., bringing a cup to their mouth) in twelve (23.08%) of virtual experiences.

In all 52 interactive virtual experiences, participants reached for objects at a table-level and directly in front of them in the front left (n = 47, 90.38%) and front right (n = 49, 94.23%) regions (see Figure 2). Less frequent reaching was observed for objects located above (n = 13, 25%) or behind (n = 10, 19.23%) the participant. During three of the applications, participants could not reach the floor region without bending over. In the fourth application, participants were able to reach down and touch the virtual “ground” without bending due to the positioning of their virtual character in the virtual space. In the 39 virtual experiences where participants were required to bend, there were 20 (51.28%) instances where participants reached into the floor space. Of those 20 instances, in ten (50%) participants bent from the hips and in ten (50%) participants bent from the waist. While bending and reaching, only one participant was observed to lose balance in one application (1.92% of experiences), making a small stumble before regaining balance.

Figure 2. Observed regions where participants reached for and/or grasped virtual objects.

A graphic representation of regions around the body in the horizontal and vertical planes. Percentages are given as to what proportion of participants reached for and/or grasped objects within each space.

While reaching for objects, eight (61.54%) participants were observed to make depth judgment-related errors (see Table 2). Each of these participants made errors in no more than two of the four experiences allowing object interaction. Under-reach errors were most common, making up two thirds (n = 12, 66.67%) of depth errors. Under- and over-reach errors were never made within the same virtual experience.

Table 2. Proportion of participants and virtual experiences where depth perception errors were observed in reaching behaviours.

Errors observed	Participants (N = 13) performing behaviour % (n)	Experiences (N = 52) where behaviour observed % (n)
Over-reach errors	7.69% (1)	15.38% (8)
Under-reach errors	30.77% (4)	7.69% (4)
Both under- and over-reach	23.08% (3)	0%

Interaction and communication with characters

Each participant engaged in three virtual experiences (total N = 39) where virtual characters delivered instructions or engaged in conversation verbally. In 35 (89.74%) of these experiences, participants turned to locate the character who had spoken to them, with four (30.77%) participants failing to acknowledge the source of communication in one of their experiences.

In virtual experiences where participants were required to speak (N = 26, two experiences for each participant), all used verbal communication to engage in the interaction. However, four (30.77%) participants required a prompt from the researcher to use spoken communication at the beginning of an experience, and three (23.08%) required a prompt to use non-verbal or gestural communication (i.e., nodding or shaking their head) to elicit a response from the virtual character. Where non-verbal communication was not required, eight (61.54%) participants still used gestures to supplement their spoken communication. Even when participants were not required to use any expressive communication, seven (53.85%) still used spoken (n = 6, 46.15%) or gestural communication (n = 1, 7.69%). All unnecessary verbal and gestural expression used by participants was rated as being situationally appropriate (i.e., relevant to the context of the interaction) by the observing researcher during qualitative coding, and verified by co-authors. In contexts where communicative expression was not necessary to the gameplay and virtual characters did not respond, only one participant (7.69%) was observed to continue communicating. All other participants ceased using verbal communication with avatars in the game when it became apparent that they were not responding.

The virtual experiences requiring communicative interaction took two forms. The first form of interaction used the Wizard-of-Oz protocol [20] to control the conversational turns of the virtual avatar. In this interaction, participants used an average of 1.25 utterances per conversational turn (SD = 0.27, range = 1–1.91) and engaged in the interaction for an average of 8.69 conversational turns (SD = 2.36, range 4–12). No participant was observed to repeat themselves and the interaction appeared to flow in a conventional manner. In the second form of interaction with an automated virtual character, ten (76.92%) participants repeated responses between two and five times during conversational turns before the virtual avatar partner answered, resulting in all exhibiting signs of frustration. The application failed to play in full for four participants due to the automated character failing to recognise or respond to communicative turns.

In addition to the interaction required in gameplay, nine (69.23%) participants used objects to interact with characters. Their interactions included attempting to hand objects to a character, showing objects to a character to reinforce their verbal responses, and waving or throwing an object to gain attention or elicit a reaction. Such interactive behaviours were observed a total of 30 times across 20 (38.46%) of the 52 experiences. In all 31 experiences where participants needed to use objects to interact (i.e., giving objects to or taking objects from a character), all participants did so successfully.

While interacting with characters, seven (53.85%) of the participants engaged in unconventional behaviours in a total of 16 (30.77%) virtual experiences. Behaviours were categorised into five types (see Table 3). The behaviour observed most frequently was use of an object to strike a virtual character by throwing, hitting, or shooting. Of the seven participants who engaged in these unconventional behaviours, they performed them in one (n = 1/7, 14.29%), two (n = 4/7, 57.14%,), three (n = 1/7, 14.29%) or all four (n = 1/7, 14.29%) interactive applications.

Table 3. Instances of unconventional behaviour observed with characters in virtual interactions.

Unconventional Behaviour	Times observed	Participants (N = 13) performing behaviour % (n)	Experiences (N = 52) where behaviour observed % (n)
Throwing/ shooting/ hitting character	21	46.15% (6)	25.00% (13)
Giving a false name	4	15.38% (2)	3.85% (2)
Taking clothing accessories from character	3	7.69% (1)	1.92% (1)
Offering food as payment	2	15.38% (2)	3.85% (2)
Yelling at character	1	7.69% (1)	1.92% (1)

Visual and spatial attention

Participants each engaged in one virtual short film experience where the researcher observed visual and spatial attention including attention to directional sound, visual attention, and object tracking. Within the virtual experience, there were three instances where directional sound signalled the need for participants to look for a virtual character. Only one participant (7.69%) attended to the sound in all cases; six (46.15%) reacted in two cases, and one participant (7.69%) did not attend to any directional sound. Participants were also presented with four opportunities to attend to directional gaze (i.e., a character within the interaction looking to the left to indicate the user needed to do the same). All participants reacted to at least 50% of these opportunities, and eight (61.54%) participants attended at every opportunity.

Within the virtual film, five objects moved through the virtual space that required participants to track that movement by rotating from the head, waist, or feet. All participants tracked at least three of the moving objects, with four participants (30.77%) tracking four objects and four (30.77%) tracking all moving objects. While all participants were able to track moving objects, seven participants (53.85%) interrupted their tracking to glance back at the main character avatar, but then resumed tracking. After tracking an object movement to its completion, most participants (n = 10, 76.92%) always and immediately returned their attention to the main film event.

Support requirements during immersion

Most participants (84.6%) maintained some connection to the physical world while engaged in a virtual experience by asking the researcher for support, indicating a sustained awareness that someone was present outside of the virtual experience. Participants who made these requests did so in one (36.36%), two (36.36%), or three (27.27%) of the five applications, with the most requests coming in each person’s final virtual experience.

Of the 13 participants, 11 (84.6%) were observed to make task-related comments while engaged in the virtual world, in one (63.64%,), two (18.18%), or three (18.18%) of the five applications. Participants were observed to make such comments more often during the final application, with almost half (72.72%) vocalising in some way during the virtual game. Task-related comments were classified into four distinct types: exclamations (e.g., “wow”, “oh no”), self-talk (e.g., “I didn’t mean to do that”, “I’ll pick that up”), statements related to what they were observing (e.g., “I’m a rabbit”, “It’s loading”), and repetitions of instructions provided in the application (e.g., after a character asked “can I have a hotdog” the participant repeated “can I have a hotdog” while looking for the item). While more participants were observed to make exclamations (72.72%), self-talk was the most frequent comment type with 38 instances observed in total.

All participants required at least one instance of support during their immersion experience, although the number of times support was required varied greatly (mean = 6.15, SD = 6.66, range = 1–27). Support was required in 34 (53.85%) of the virtual experiences, with participants needing support in up to four interactive applications. The type of support required by participants was analysed and is reported in Table 4. Of the total 79 instances of support provided to participants, 56 (70.89%) related to gameplay, while the remaining instance of support related to use of the VR system.

Table 4. Type of support required while immersed in virtual reality.

Type of support needed	Times observed	Participants (N = 13) performing behaviour % (n)	Experiences (N = 65) where behaviour observed % (n)
Gameplay support
Locate needed item	14	53.85% (7)	12.31% (8)
How/When to communicate	7	38.46% (5)	10.77% (7)
What to do next	30	61.54% (8)	21.54% (14)
Reinforcement of action	5	23.08% (3)	4.62% (3)
System support
Control use	11	61.54% (8)	12.31% (8)
Physical positioning	11	53.85% (7)	15.38% (10)
Headset adjustment	1	7.69% (1)	1.54% (1)

In addition to required support, all participants experienced a technological challenge while using between one and four applications. A total of 33 technical issues were identified in 28 (43.08%) virtual experiences (see Figure 3). The most frequent technical challenge, and the one experienced by the most participants, related to interference with the tether cord as five (38.46%) of the participants became twisted in the cord that connected the VR HMD to the computer. All were able to remove the obstacle on their own and it did not result in any adverse events. Technical issues were evenly distributed between hardware issues (51.52%,) and software issues (48.48%).

Figure 3. Technological challenges experienced by users while using immersive virtual reality.

A bar graph depicting the different technological challenges faced by participants. Interference of the tether cord is the most frequently reported challenge, followed by accidentally pausing or exiting the game.

Discussion

Considering the complex sociotechnical interactions required for immersive VR to be used effectively in rehabilitation activities and settings, the results of this study provide some important insights into the ways that novice users approach and interact with immersive VR. The observational protocol provided a comprehensive tool to analyse the performance of participants and understand their interactions within immersive VR. All novice users were able to interact with the technical system in a meaningful way, even without training. The novice users were able to engage with virtual characters, objects, and environments to complete a range of tasks including verbal and non-verbal communicative interactions, physical action sequences, and visual tracking of events within the virtual environment. The research protocols were adequately detailed to capture these features of communicative, physical, and visual interaction with characters, objects, and environments while immersed in VR, with only minor additions required. This study contributes new insights and knowledge in the areas outlined in the aims of the study and discussed below.

Experience of cybersickness

The cybersickness screening procedures used in this research were an acceptable indicator of individuals who may experience cybersickness in immersive VR. This is consistent with the findings of prior research (e.g., [11]). While participants in the current study who had a higher risk of cybersickness as per the MSSQ were more likely to report symptoms, few symptoms of illness were reported overall. The low rate of symptoms may be explained by the applications used, where all body movements in VR were a direct mirror of the participant’s movement in the physical world. There was no additional movement of the user’s body created by the environment, joysticks, or anyone other than the participant; features of immersive VR that have been identified as increasing risk of cybersickness [11]. The immersive short film experience was slightly different in that the user’s virtual body did not move (i.e., could rotate but was anchored to one spot), even if they did move in the physical world. The virtual characters within the environment moved around them. This may explain the higher rate of symptoms, although mild, that were reported by participants after experiencing this application.

While mild symptoms of cybersickness were observed, these had no overt impact on user performance in VR. This finding stands in contrast with prior research that demonstrates that cognitive performance declines in the presence of cybersickness symptoms [24] and disorientation significantly increases [25]. It is possible that the mild nature of the cybersickness symptoms observed amongst users in this study did not highlight previously documented impacts of cybersickness; which might reduce the usability of immersive VR in rehabilitation.

Visual immersion and orientation to the visual environment

While adult novice VR users in this study did not always fully orient themselves to the virtual space, they were able to look in all directions that required their attention to proceed in the application. This observation is consistent with prior research that visual attention is directed by tasks in virtual environments [26]. It is therefore unlikely that the observed orientation behaviours will have any impact on the use of immersive VR for a range of purposes in rehabilitation. Observations and rehabilitative actions in immersive VR can also proceed with the expectation that visual attention will be linked to planned tasks.

Interaction with characters, objects, and the virtual environment

In this study, novice VR users without disability were able to attend and orient to the visual environment with minimal movement of their feet, instead turning their head or twisting from the hips. These findings have implications for the use of immersive VR systems by people with physical or visual disabilities that may increase their risk of or loss of balance or falls. When considering the use of immersive VR in rehabilitation, there is a need for caution due to reported high rates of falls amongst individuals in such settings [27], and other risks to people with disabilities [2]. The use of a VR system while seated may facilitate use by individuals with disability. However, the impacts of use in a seated position will need to be carefully considered, with prior research showing that experts perceive a trade-off between greater safety and accessibility in seated use of VR, with lesser engagement and precision in locomotion [28]. Some refinement of research protocols will be necessary if users are engaging in immersive VR while seated.

The primary use of a dominant hand by most novice users observed in this study suggests that users may be able to adequately engage with immersive VR environments using only one hand. However, such uses would need to consider that single-handed operation may negatively affect perceptions of usability and naturalness in VR interactions [29]. Where single-hand operation is not possible, people with unilateral or bilateral physical impairments (e.g., related to cerebral palsy, motor neurone disease, Parkinson’s Disease) would also need assessment for a range of alternative access methods to enable their inclusion in immersive VR interventions. For users without disability, access through hand-tracking and eye-tracking have enabled successful system interactions. However, Johnson et al. [29] reported significant increases in physical and mental demand when using such access methods. The involvement of an interdisciplinary team, including occupational therapists and physiotherapists, could help to ensure adapted access is provided to VR users in rehabilitation contexts with physical limitations affecting their use of controls.

The novice users in this study showed that they were able to use spoken communication within the context of an immersive virtual environment, and many naturally supplemented their speech with body language and gesture. However, the responsiveness of the immersive VR programs to spoken language is a key factor for success. The café simulation relied on the researcher to initiate responses, and so permitted a natural and immediate response time. However, delayed responses and long lag times in conversations with virtual characters in off-the-shelf applications appeared to impede the required sense of presence, with participants demonstrating obvious frustration and having to repeat themselves multiple times. As such, immersive VR experiences that utilise automatic voice recognition may not yet be of a high enough quality for communication rehabilitation. A user’s communication disability (e.g., speech or language impairments) could further limit the accuracy of software in interpreting the user’s communication attempts [30]. Considering the need for continuous flow of conversation, the Wizard-of-Oz protocol [20] may be more efficient in generating an immersive conversation in virtual environments while also facilitating integration of the rehabilitation professional into the virtual world for added therapeutic value [31].

Differences in social norms and convention in the physical and virtual worlds could also potentially factor into rehabilitative success, particularly where character interaction is considered. Bylieva and Nam [13] identified that social behaviours classified as “deviant” (p.369) in the physical world may be attractive, or indeed required, in virtual games. Examples of such behaviours were observed in this study with participants using unconventional and injurious social behaviours when interacting with avatars. These responses were not initially included in the observation protocols, though were added from field notes, and following discussion with all co-authors. Altered social conventions may have implications for therapies aiming to increase a person’s skills in conventional interactions. Researchers need to be aware of the possibility that such social behaviours may appear in immersive VR studies, necessitating the inclusion of relevant items in research protocols.

In attending to events in the virtual environment, novice users were observed to attend well to cues from characters such as directional gaze and pointing, though directional sound was a less reliable attention cue. These findings were consistent with an analysis of directional cues by Chen et al. [15] who found that visual cues elicited the best performance by novice users in immersive VR. Visual cues appeared to make immersive VR gameplay more seamless, with users in this study requiring fewer instances of gameplay-related support when in-app visual cues and directions were provided. As gameplay support was frequently required throughout the study, these findings suggest that the integration of visual cues may be a necessary component in immersive VR applications designed specifically for rehabilitation purposes particularly if users are to achieve successful independent use of immersive VR, for example, to engage in home practice.

Support requirements during immersion

Some technical challenges were experienced during this study, relating both to the participants’ use of immersive VR hardware and software. These challenges required the support of a researcher in many cases. Our results suggest that a rehabilitation professional trained and experienced in using immersive VR technologies and applications may need to be available to assist or to mentor others at least during early interactions with immersive VR software, to ensure that both researchers and participants are able to appropirately use the software without difficulty [31]. In addition, our findings highlighted a need for careful equipment choice, particularly using wireless immersive VR systems where possible, to minimise technical problems and eliminate potential safety risks for participants. Although many participants experienced minor difficulties when becoming entangled with (or even just aware of) the cord that tethered the HMD to the PC, none experienced any adverse events (e.g., tripping or falling). The feature of the headset being tethered to the computer is not shared by all commercially available immersive VR HMDs so could be avoided with the choice of an alternate device.

Limitations

All participants in this study were self-reported novice users of immersive VR. While the findings may not reflect the use or support needs of more experienced users of VR or other extended reality technologies (e.g., augmented reality), the findings are likely to reflect the technological skill of users who might engage with immersive VR for rehabilitation. As such, the patterns of use and the support requirements identified in this study could inform the design of studies testing the implementation of immersive VR in general health and rehabilitation settings. Future studies will need to look more specifically at diverse populations with a wide range of health conditions and associated disabilities to better understand how immersive VR technology and virtual environments could be used in their rehabilitation.

Each participant in this research experienced five different immersive VR applications. This allowed the researchers to observe communicative, physical and visual interactions with charaters, objects and the virtual space. The five applications were presented in a standard order for all participants, and were not counterbalanced. The lack of counterbalancing meant that performance in the later applications was likely influenced by sequence and order effects, whereby participants had learned expected responses from prior experiences [32]. However, as noted in the methods, this allowed the researchers to observe some aspects of skill development and progression across applications with increasing interactivity requirements. This could help to understand how practice and exposure may influence skill in novice users of immersive VR. Fatigue may have also affected performance in later applications, though all participants were given mandatory breaks to recover between each limited exposure and were immersed in VR for no more than 15-minutes at a time. The changes in performance observed in this study have implications for how VR might be introduced to novice users in rehabilitation settings, and how skill and interaction can be influenced through graded exposure.

Most participants who expressed interest and participated in this study were in their early to mid-20s, and as such its findings might not apply to older adults in the general population. The younger demographic was consistent with participation in other interventions using immersive VR for rehabilitation purposes. For example, in a systematic review of VR-based neurocognitive interventions, the average age of 132 participants across 13 studies was 36.1 years [33]. The consistently younger participant demographic in these VR studies suggests that younger individuals may have a greater interest in such technologically-focused approaches to rehabilitation. Future research might need to more carefully consider recruitment strategies to attract an older demographic, to consider needs across the lifespan.

Conclusion

The growth in immersive VR systems in recent years has led to novel adaptations in health and medical fields, particularly for rehabilitation purposes. However, the often-unpredictable relationships and interactions between humans and technology can impact the usability of these devices. The findings of this study could inform how VR systems might be tested for rehabilitation for people with communication, physical and visual disability, though careful consideration is needed if used to inform the design of applications for such purposes. Novice immersive VR users in this study were able to adapt to the technology and engage in a range of virtual scenarios; however showed some minor difficulties in visual tracking, physical use of hand controls, and the following of social conventions in communication with avatars. Nonetheless, users without disability were able to orient to the virtual environment sufficiently for the tasks they needed to complete. They largely relied on visual cues to direct attention, although some users also reacted to auditory cues. Only minor cybersickness symptoms were reported for two participants whom the screening protocol identified as at high risk. Importantly, the developed observational protocol captured the vital elements of the user’s experience and interaction with the immersive VR system.

The results of this study, and the observational protocol developed and tested, can be used to inform future research aiming to include adults with physical, intellectual, sensory, communication, and social disabilities so that immersive VR applications can be designed that are inclusive of adults in need of rehabilitation. Factors to be considered in the design of future studies include the range of movement needed, the verbal cues to be provided for users to make full use of all parts of the immersive environment, the availability of support from a facilitator to troubleshoot technical hitches and provide instructional prompts, the use of wireless over tethered head-mounted devices, and the use of any adapted controllers or assistive technologies. Controlled trials of immersive VR in rehabilitation that take account of these factors will provide clarity in outcomes to determine if immersive VR is an effective platform for rehabilitation. The observational and cybersickness screening protocols appear to have good validity in assessing the interaction of users with no reported disability with the immersive VR system and should now be tested with user populations with disability to ensure validity across populations.

Disclosure statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability statement

The data associated with this research is not available due to ethical constraints.

Additional information

Funding

This research was funded by a Seed Grant to all authors from the University of Technology Sydney Graduate School of Health, NSW Australia.