A multiplayer VR showdown game for people with visual impairment

ABSTRACT

Researchers have developed virtual reality (VR) systems for people with visual impairment using various audio feedback techniques. However, few studies have investigated collaborative VR systems where people with and without visual impairment can participate together. We developed a VR Showdown, similar to a real showdown game, where players with and without visual impairment can play together in a remote virtual environment. We incorporated auditory distance perception using a head-related transfer function based on the spatial position of the Showdown ball in VR. We developed two modes of VR Showdown. One is the player vs. AI agent (PvA) mode, where visually impaired people can play alone. The other is the player vs. player (PvP) mode, in which visually impaired people can play with another player in a network. The user study results showed that people with and without visual impairment enjoyed playing VR Showdown games that were well-matched.

KEYWORDS:

• Virtual Reality (VR)
• Assistive Technology
• Serious Games
• Showdown
• People with Visual Impairment
• Spatial Audio

1. Introduction

Virtual reality (VR) can support people with visual impairment, even though these systems rely mainly on visual feedback. One possibility is to play audio sports like “Showdown” virtually without the bulky table, racket, and ball. Showdown is a fast-moving sports game, similar to tabletop air hockey, originally designed for people with visual impairment, but anyone can play. Researchers are investigating using the virtual reality Showdown game for people with visual impairment: Wedoff et al. (2019) developed the Virtual Showdown, where players hit the incoming ball based on verbal/vibration scaffolds. In their game, the system detects the player’s gestures and provides spatial audio and verbal feedback to the player to detect the ball’s movement. They used a head-related transfer function (HRTF) with a Microsoft audio spatializer to present a localization sound cue of a moving ball to people with visual impairment. In addition, the system used a Microsoft Kinect camera and a Nintendo Joy-Con controller to track the player’s body position and hand movement and provide vibrations via Joy-Con’s HD Rumble to detect collisions between the Joy-Con and virtual ball. However, their Virtual Showdown was a single-player game in which one player hit the one-off incoming ball; moreover, the rally was not supported, unlike a real showdown.

Collaborative virtual reality (VR) applications exist in several fields. In education, VR teachers can remotely teach students how to assemble aircraft engine combustion chambers (Oda et al., 2015). In real estate, a Dollhouse VR system (Ibayashi et al., 2015) facilitates collaboration between designers and consumers. Designers can arrange furniture in a VR environment using a top-down view, whereas consumers can explore virtual homes from a first-person perspective. In the medical field, therapists use collaborative VR systems to help children with neurodevelopmental disabilities improve their emotional recognition and communication skills (Loiacono et al., 2018).

We developed a multiplayer VR Showdown game using 3D spatial audio and haptic feedback to explore VR opportunities for the visually impaired. There are two modes for the VR Showdown game: a single-player mode called PvA (Player vs. artificial intelligence (AI) Agent) and a multiplayer mode called PvP (Player vs. Player). The PvA mode allows people with visual impairment to play independently by competing against an agent. In contrast, the PvP mode allows people with visual impairment to play remotely with people without visual impairment through photon unity networking (PUN) for multiplayer games. In PvP mode, two remotely located players share audiovisual scenes of the Showdown game in real time using head-mounted displays (HMDs) through the PUN.

The research goal of this study is to design a VR game that allows people with and without visual impairment to play together and verify that the game is enjoyable for all participants. We conducted two user studies with 27 visually impaired individuals. In the first user study, the Showdown game was between an AI agent and visually impaired individuals. In the second user study, the Showdown game was between people with and without visual impairment and played remotely in real-time. The contributions of this study are as follows:

• We developed a multiplayer VR game that people with and without visual impairment could play together.

• We demonstrated that people with visual impairment could competitively play a state-of-the-art VR game against an AI Agent.

• We demonstrated that people with visual impairment and those without visual impairment (but blindfolded) could play well-matched VR Showdown games together.

2. Related works

This section introduces related work on the themes of VR for people with visual impairment, body tracking in VR games for people with visual impairment, and multiplayer serious games in VR.

2.1. VR for people with visual impairment

Researchers have developed various VR systems for people with visual impairment. One of the main features of these VR systems is the provision of spatial audio, such as HRTF. HRTF describes how sound waves interact with the listener’s torso, shoulders, head, and, particularly, the outer ear (LaViola et al., 2017). One of the main properties of HRTF is that they can contain considerable spatial information in real-world listening environments. This technical characteristic has facilitated the development of various VR applications for people with visual impairment. Applications can be divided into three categories depending on the sound generated by HRTF.

In the first, sound is generated at a position where the system responds to the user’s behavior. For example, in a VR walking system for people with visual impairment, a virtual cane sound is generated at the spatial point where the cane collides with another virtual object (Siu et al., 2020; Zhao et al., 2018). In the second, sound is generated at a particular location regardless of the user’s behavior. The sound indicates the location at which the user needs to arrive (Kreimeier & Götzelmann, 2019) or the location of the object they want to find (Eckert et al., 2018). HRTF gives a sense of a specific location via sound; however, the sound reflected differed depending on the surrounding environment.

In the third, the location where the sound is generated changes in real-time with a moving object. Thevin et al. (2020) applied the HRTF to the sound of passing cars on the streets so that visually impaired people could recognize a passing car in a virtual environment just as they would in a real street environment). Simões and Cavaco (2014) developed an audio game where people with visual impairment detected moving “aliens.” In a recent study, researchers developed VR applications whereby people with visual impairment could visualize bi-molecular structural models. When rendering the molecular structure of a protein in sound, the HRTF generates the sounds of atoms along with the spatial position of the protein’s backbone (Arce & McMullen, 2017). Using this method, researchers have noted that people with visual impairment can identify the spatial position of each atom and create a mental map of the bi-molecular structure. These studies show that many VR systems have been developed for people with visual impairment. However, few studies have been conducted on VR systems that allow people with and without visual impairment to play together. In this paper, we present a multiplayer VR game that enables people with visual impairment play remotely in real-time with people without visual impairment.

2.2. Body tracking in VR games for people with visual impairment

Tracking devices, such as the Nintendo Wii^TM or Microsoft Kinect^TM have been used in VR games to support people with visual impairment. In VI-Bowling and VI-Tennis, the player rolls (Morelli, Foley, & Folmer, 2010) and smashes the ball (Morelli, Foley, Columna, et al., 2010), respectively, by swinging a Wii controller. In Pet-N-Punch, the player wields a Wii controller from top to bottom, mimicking hitting a mole with a hammer (Morelli et al., 2011). In virtual track and field games, the Kinect camera captures the running motions of a player and maps them to the athlete avatar’s running (Morelli & Folmer, 2011).

In a recent study, which is similar to our work, Wedoff et al. (2019) used a Kinect camera to track the bodies of people with visual impairment and used the Joy-con^TM (the improved version of a Wii controller) to track the x- and z-axis position of their hand. However, it did not fully reflect the swing of the bat, the real-time position and motion of the wrist, or the hand-introduced jitter; thus, the virtual racket was only implemented to turn discretely at 45° intervals.

Overall, there is a lack of VR games using immersive VR devices, such as the HTC Vive^TM or Oculus Rift^TM, for people with visual impairment, although there are studies of people without visual impairment (Koulouris et al., 2020; Michael & Lutteroth, 2020; Yoo et al., 2017, 2020). In our study, we developed an immersive VR game for people with visual impairment using an HMD and controllers to track the directional sound of a ball based on the user’s head position and provide haptic feedback based on the user’s hand position. We expected that people with visual impairment would have a more immersive and kinetic experience in our study than that reported in related studies based on the Nintendo Wii ^TM and Microsoft Kinect ^TM (Wedoff et al., 2019).

2.3. Multiplayer VR serious games

Several researchers have developed multiplayer VR serious games; Wegner et al. (2017) developed a game in which one player played the role of a senior paramedic helping a junior paramedic using a virtual tablet PC, and the other player played the role of a junior paramedic treating the patient using various virtual tools at his waist. Ha et al. (2016) developed a game in which several players put out a fire that blocked their way to escape from the building. Social MatchUp is a smartphone VR game in which children with neurodevelopmental disorders play activities with a therapist to improve their ability to recognize and verbalize emotions (Loiacono et al., 2018).

Although research has been conducted on multiplayer VR games for people with disabilities (Hong et al., 2017; Loiacono et al., 2018; Millen et al., 2014), research on multiplayer VR interactions for people with visual impairment is lacking. One benefit of multiplayer VR games for people with disabilities is that they can communicate in a virtual environment and learn to engage with others (Loiacono et al., 2018). According to Grabski et al. (2016), multiplayer VR games between people with and without visual impairment are expected to help people with visual impairment be more widely included in society. However, the asymmetrical roles of people with and without visual impairment present challenges in the awareness of their counterparts (Gonçalves et al., 2021). In this paper, we present a user study involving people with and without visual impairment playing symmetrical roles through a newly developed multiplayer virtual reality game, VR Showdown.

3. VR showdown implementation

This section explains the implementation of the VR Showdown with respect to body tracking, racket positioning, spatial sounds, haptic feedback, and game design.

3.1. Body tracking and racket positioning

We used the HTC Vive Pro^TM because it provides accurate head tracking and creates more precise and realistic spatial sounds through the HMD’s headphone than previous work with Kinect or Nintendo Wii (Wedoff et al., 2019). The Vive Pro^TM HMD can play lossless audio in a high-quality sound format. In addition, the Vive Pro^TM HMD can detect the user’s head position, angle, and rotation to match the movement of the user’s head with that of the virtual head in the VR environment. The virtual head transformation consists of the position, orientation, and scale. This transformation is used to calculate the direction and distance between the user’s head position and the origin of the sound to produce 3D spatial sounds. A Vive Pro^TM controller was used to track the user’s hand movements to represent the user’s swing in a virtual environment precisely. Figure 1 shows the VR Showdown system used in the experiments.

Figure 1. Overall view of the VR Showdown.

3.2. Spatial sound system

We used a head-related transfer function (HRTF) to provide players with spatial audio relating to the the Showdown ball. Auditory distance recognition is caused by binaural signals such as acoustic parallax (Zahorik et al., 2005). Acoustic parallax detects significant differences in the relative angles of each ear to the virtual audio source object. As an alternative to HRTF, to create spatial audio, a stereo panning system adjusts the sound level on the left and right sides. However, stereo audio panning systems cannot provide users with sufficient acoustic parallax. Larsen et al. (2013) compared a panning system with the HRTF to localize a virtual audio source object. They found that HRTF was superior to the panning system in terms of audio source location recognition accuracy and search speed. Therefore, in our VR Showdown, we used an HRTF to provide spatial audio feedback so that players could detect the precise position of the Showdown ball.

Various Software Development Kits (SDKs) are available for implementing HRTF. These include Oculus Spatializer 1.22.0, Google’s Resonance Audio 1.2.0, and Steam Audio 2.0 Beta 13. We used Google’s Resonance Audio SDK because it delivers accurate spatial sounds to identify the direction and location of nearby sound sources in real time (Gould, 2018). Its features include a reflection function that calculates sound waves bouncing through the air, a direct sound function that decreases the sound with increased distance, and a function that simulates environmental occlusion effects by low- and high-frequency components. These functions allow us to develop a system that enables players to precisely locate the ball by listening to sounds.

To properly migrate the Showdown into the virtual environment, a resonance sound listener component was assigned to the object corresponding to the player’s head position. For various conditions, such as the rolling ball sound, sound of the ball crashing into the wall, and sound of the racket colliding with the ball, we recorded audio bites from the original Showdown game and combined them with the HRTF. When the speed of ball movement dropped below 2.8 m/s, the sound size was reduced to 9/10 of the speed of 0.28 m/s, and when the ball stopped, no sound was heard from the ball. In addition, all the start and end sound wavelengths were continuously aligned to ensure that natural sounds were produced from repeated audio recordings.

3.3. Haptic system

We implemented the vibrations on Vive Pro^TM controllers when the racket hits the ball and the player uses the holding function. A detailed explanation of the holding function is provided in Section 3.4.2. The Vive Pro^TM controller has an internal function called “Pulse,” which momentarily vibrates the controller. We used “Pulse” to generate vibration frequencies of 70 Hz per second for strong vibrations and 30 Hz per second for weak vibrations. A short, strong vibration is generated when the racket hits the ball. In contrast, in the holding situation (when the racket drags the ball), a weak, continuous vibration is generated.

3.4. VR showdown game design

Our system replicated the original Showdown sports as closely as possible, as described below.

3.4.1. Basic rules for VR showdown

We designed the basic rules of VR Showdown to match the real-world “Showdown” sport in Paralympic games, as follows (IBSA, 2022).

• Before the game begins, all the players are blindfolded.

• The game begins with serving the ball: The system automatically serves the ball by rolling it from the center of the table to one of the players.

We chose this service method because it is difficult for the player to smash a stopped ball, which does not produce a sound. The sound of the ball rolling from the center of the Showdown table toward the player signals the start of the game. Each player receives a service twice for each turn.

• Scoring: A player scores two points if they put the ball into the opponent’s goal pocket. Upon scoring, the game system announces (in the moderator’s voice) that the player has scored two points. The game system starts the next service sequence 5 s after the goal announcement.

• Dead ball: When the ball lies “dead” (motionless and soundless), the serve sequence is re-executed by the last player to serve and with no demerits.

• Victory: The player wins the game if they score 12 points first. The player wins the match if they win two of the three games.

3.4.2. New functions of VR showdown

A new function, “holding” has been added to the VR Showdown gameplay. In the original Showdown game, players use the racket to “hold” the ball by gently pushing the ball to the right or the left sidewall. Once the ball stops, the player can shoot it accurately. In reality, friction can stop the ball’s movement by pushing it against a wall using a racket. However, in the Unity engine, collisions occur between objects, walls, and rackets during such actions. If the collision occurs continuously in a short time, the limit of the physics engine causes the ball to pass through the racket or amplify the collision reaction, causing it to bounce at an abnormal speed. “holding” has been implemented in our game to address this problem. If the ball is in contact with the racket and the player simultaneously pushes down the trigger button of the Vive Pro^TM controller, holding is activated, and the racket drags the ball. The ball’s coordinates are calculated every 1/60th of a second, and the ball’s position is changed to follow the racket at a fixed distance from the end of the racket while the trigger button is pressed. During holding, the controller provides vibration feedback. The player can release the button at his/her preferred spot and moment and then strike the ball toward the opponent.

3.4.3. PvA game mode

For the PvA game mode, we implemented an AI Agent as the opponent, as shown in Figure 2. The AI Agent counter-hits with the following success rates.

Figure 2. PvA (Player vs. AI Agent) play. The left image shows the participant playing the VR Showdown, using a green screen. The right image shows the AI Agent and participant’s virtual racket.

1. The agent returns the ball in the first approach with a 100% success rate.

2. After hitting the ball once, the probability of the agent counter-hitting the ball is reduced to 70%.

3. After hitting the ball twice, the probability of the agent counter-hitting the ball is reduced to 40%.

4. After hitting the ball thrice, the agent randomly moves to either the right or left area of the goal pocket and counter-hits the ball.

The average number of rallies between the visually impaired player and AI agent before a point was scored was 4.03, with the longest rally being 13. The AI Agent had difficulty blocking the players’ strong shots. The approximate criterion for strong shots is 11, which is indexed to the ball’s velocity. A value of 11 represents the threshold for shots greater than 10 m/s in the real world.

3.4.4. PvP game mode

The multiplayer mode between people with and without visual impairment was developed by implementing network communication through the photon engine from the Unity Asset (Unity3D, 2020), as shown in Figure 3. We developed our program so that each user could estimate the distance and direction of the ball with the same but mirrored spatial sound information based on each user’s position. The ball information (e.g., position, speed, and direction) was transmitted to each player in real time through the network server through auditory feedback. Each player strikes the ball in the opponent’s goal pocket to score. The players could have vocal conversations while playing games.

Figure 3. PvP (Player vs. Player) play. The upper player (a) hits the ball on his left (b). The lower player (c) anticipates the ball on the other player’s side (d) by hearing. Each player estimates the distance and direction to the ball with the same but mirrored spatial sound information based on each user’s position.

4. Preliminary study: evaluation of HRTF system in VR showdown

Prior to the main user experiments, we investigated whether participants with and without visual impairment could navigate the Showdown ball using HRTF sounds without visual cues, as described by Wedoff et al. (2019).

4.1. Participants and apparatus

Twenty-six participants with and without visual impairment were recruited for this study. However, one participant with visual impairment was excluded because of being deaf, and another without visual impairment was excluded because of a data recording error. Thus, the HRTF system was evaluated in the context of a VR Showdown involving 24 people with and without visual impairment. The mean age of the 12 participants with and without visual impairment was 59.67 (47 to 74 years) and 22.41 (20 to 27 years), respectively. Nine participants were male and three were female in both cases.

4.2. Tasks and measurement

For the main task, the participants were asked about the start and end positions of the moving ball in the Showdown. The ball was placed in three areas in front of the participant (left, middle, and right) and in two areas far from the participant (left and right). Thus, there were six departure routes from the three areas on the near side to the two on the opponent’s side. In addition, the six arrival routes correlated oppositely with the departure routes. Each route was played thrice by each participant. Each participant was asked 36 questions regarding the routes of ball movement (i.e., (six departure and six arrival routes) × three repetitions). The trajectories of the ball movements were randomized using a Latin square design. For the quantitative evaluation, we measured the accuracy of the participants in detecting the correct route.

4.3. Procedure

Each participant provided written informed consent and completed a demographic questionnaire. Subsequently, they participated in a training session. The participants were made to stand at the Showdown table, as shown in Figure 4 and made to hear the sound of the Showdown ball movement. During training, each route was announced in advance, and the six departure and six arrival routes of the ball were repeatedly played until they became familiar with the sound of the ball’s downward movement. For the main post-training tasks, all participants wore eye patches covering both eyes. Participants answered thirty-six questions about the ball’s movement based solely on auditory cues. Participants were allowed a 5-min break after answering 18 questions.

Figure 4. Setting for HRTF Evaluation.

4.4. Results for preliminary study

We measured the accuracy of the participants in detecting the correct route for moving balls. The results show over 90% average accuracy in detecting the correct trajectory of the moving ball for participants with and without visual impairment (92.7% and 91.9%, respectively). Table 1 summarizes the accuracy rates for detecting the various trajectories of outgoing and incoming balls.

Table 1. Results of the preliminary study.

Outgoing ball direction detection accuracy rates (%)			Incoming ball direction detection accuracy rates (%)
Direction	visual impairment	people without visual impairment	Direction	visual impairment	people without visual impairment
Left->Left	97.44	89.74	Left->Left	84.62	94.87
Left->Right	100	89.74	Left->Middle	84.62	97.44
Middle->Left	89.74	84.62	Left->Right	94.87	100
Middle->Right	84.62	84.62	Right->Left	87.18	92.31
Right->Left	92.31	100	Right->Middle	79.49	79.49
Right->Right	94.87	94.87	Right->Right	94.87	97.44

5. Experiment 1: PvA mode

To evaluate the VR Showdown, we conducted two user experiments. The first experiment tested the PvA mode, specifically designed to assess whether visually impaired people can competently play a game against an AI agent. Participants in this experiment were involved in a best-of-three series of 12-point games, in which the goal was to detect the position of the Showdown ball and successfully hit the opponent’s goal.

5.1. Participants and apparatus

Experiment 1 was conducted with the same group of participants as in the preliminary study. All participants had previous experience in VR environments but no previous experience with Showdown. The PvA mode, detailed in Section 3.4.3, was used in the experiment. Details of the participants with visual impairment are presented in Table 2.

Table 2. Demographic information of participants with visual impairment in the preliminary study and Experiment 1.

ID	Age/Sex	Description of the Visual condition	Experience of Showdown	Experience of VR
PVI 1–1	63/M	Low vision for 25 years (with light perception)	Yes	No
PVI 1–2	47/M	Blind for 3 years	Yes	No
PVI 1–3	58/M	Low vision for almost 30 years (with shape perception)	Yes	No
PVI 1–4	74/M	Low vision for 43 years (with light perception)	No	No
PVI 1–5	72/M	Blind for 60 years	No	No
PVI 1–6	54/M	Low vision for 34 years (with shape perception)	Yes	No
PVI 1–7	58/F	Blind for 29 years	Yes	No
PVI 1–8	54/F	Low vision from birth (with shape perception)	Yes	No
PVI 1–9	50/F	Low vision from birth (with shape perception)	Yes	No
PVI 1–10	60/M	Low vision from birth (with shape perception), no vision perception on the left eye	Yes	No
PVI 1–11	59/F	Blind for 15 years	No	No
PVI 1–12	67/F	Low vision for 18 years (with light perception)	Yes	No

5.2. Tasks and measurement

Participants played three sets of games against an AI Agent. We motivated the participants to do their best and win the game in two rounds by paying them a cash reward for winning both games. During gameplay, we systemically gathered game log data (see Table 3). Unity’s trigger feature collects log data and automatically records them in a CSV file for subsequent analysis. We analyzed these log data to compare the game performance of participants with and without visual impairment.

Table 3. Systematically measured game data automatically logged by the system in Experiments 1 and 2.

Collected data	Definition
Game time	Number of seconds a participant played the game
Rally	Number of iterations that a ball moved from the participant’s area to the opponent’s area.
Match Win	Number of participants who won two games over the opponent (AI in experiment 1 or partner in experiment 2)
Game Win	Number of games in which participants got 12 points
Shots on Target	Number of balls that reached near the opponent’s goal
Hit	Number of hits on the ball by a participant
Left Hit	Number of hits on the ball in the left area of the table by a participant
Middle Hit	Number of hits on the ball in the middle area of the table by a participant
Right Hit	Number of hits on the ball in the right area of the table by a participant
Match Win Rate	Number of matches participants won/Total number of matches
Game Win Rate	Number of games participants won/Total number of games
Shots on Target Rate	Number of Shots on Target/Total number of Hits
Hit Rate	Number of Hits/(Number of Hits + Number of Misses)
Left Hit Rate	Number of Left Hits/(Number of Left Hits + Number of Left Misses)
Middle Hit Rate	Number of Middle Hits/(Number of Middle Hits + Number of Middle Misses)
Right Hit Rate	Number of Right Hits/(Number of Right Hits + Number of Right Misses)

5.3. Procedure

All participants began the study by signing a consent form and completing a demographic survey. Participants with visual impairment had an assistant sign the consent form. Survey questions were read aloud to visually impaired participants, and an assistant or research facilitator recorded their responses.

All participants were required to wear blindfolds while playing the VR Showdown game. Additionally, all participants underwent a training session to familiarize themselves with the game. During training, participants played a participants played a 3-min practice game against an AI agent to familiarize themselves with the game rules.

After training, participants played three 12-point games against an AI opponent. The third game was not played if a participant or AI agent won the first two. The experimental procedure is illustrated in Figure 5.

Figure 5. Experimental procedure.

6. Results for experiment 1

For the statistical analysis of the game data, we used a chi-squared test for the Match and Game Win data and an independent t-test for the other data. Overall, nine out of 12 people with visual impairment and eight out of 12 people without visual impairment won the match against the AI agent in the PvA mode. The cross-tabulation of the Match and Game Win data are presented in Tables 4 and 5, respectively. Participants with visual impairment have a significantly higher rate of “shots on target” than people without visual impairment (U = 10, p < .001). The remaining measures show no significant differences between the two groups. Figure 6 shows the game data for the two groups in Experiment 1.

Figure 6. Results of game data of the two participant groups in Experiment 1.(*p < .001).

Table 4. Cross table of Match Win in Experiment 1(X²(1, N = 24) = 0.202, p = .653).

Game Win	People with Visual Impairment	People without visual impairment
Win	19	19
Lose	9	8

Table 5. Cross table of Game Win in Experiment 1(X²(1, N = 55) = 0.041, p = .840).

Match Win	People with visual impairment	People without visual impairment
Win	9	8
Lose	3	4

In the interview, participants with visual impairment mentioned having a similar experience to a real showdown game by experiencing the spatial collision and movement of the virtual ball in VR. Participants who liked the game also mentioned having fun and wanting to recommend it to others with visual impairment. Additionally, they preferred a holding function that makes it easy to control the ball. However, some participants had difficulty playing the game because of difficulty recognizing the size of the virtual rackets. As the size of the virtual racket is different from that of the controller, the participants have to estimate the length of the racket.

7. Experiment 2: the PvP mode

In the second experiment, we tested the PvP mode to determine whether visually impaired people could play a Showdown game with people without visual impairment. In the PvP mode, ball movements are less predictable and significantly faster than those in the PvA mode. All studies were approved by the ethics committee, and a special COVID-19 protocol was applied to eliminate any health risks to the participants.

7.1. Participants and apparatus

Nineteen visually impaired participants were recruited for Experiment 2, none of whom had participated in Experiment 1. Four participants with real-life Showdown experiences were excluded from the analysis for a fair comparison. Participants with visual impairment ranged from 35–68 years (mean: 56.28). Further information is presented in Table 6.

Table 6. Demographic information of participants with visual impairment in Experiment 2.

ID	Age/Sex	Description of the Visual condition	Experience of Showdown
PVI 2–1	62/M	Blind for 5 years	None
PVI 2–2	64/M	Blind for 45 years	None
PVI 2–3	62/M	Blind for 3 years	None
PVI 2–4	54/F	Blind for 8 years	None
PVI 2–5	56/F	Blind for 40 years	None
PVI 2–6	59/M	Blind for 20 years	None
PVI 2–7	62/M	Left eye blind for 7 years	None
PVI 2–8	47/F	Blind from birth	None
PVI 2–9	66/F	Low vision for 20 years (with light perception)	None
PVI 2–10	47/F	Low vision from birth (with light perception and shape perception)	None
PVI 2–11	55/F	Low vision for 5 years (with shape perception)	None
PVI 2–12	35/M	Low vision for 4 years (with light and shape perception)	None
PVI 2–13	56/M	Blind for 4 years	None
PVI 2–14	68/M	Blind for 30 years	None
PVI 2–15	57/M	Low vision for 20 years (with light perception)	None

Nineteen participants without visual impairment were recruited for Experiment 2. Four were excluded from the analysis because they played the same group as the visually impaired participants. The participants without visual impairment ranged from 37 to 72 years (mean: 56.05) and consisted of three males and 12 females.

All participants wore blindfolds during the experimental tasks regardless of their vision status. All participants were inexperienced with Showdown or VR. For a detailed explanation of the PvP mode, please refer to Section 3.4.4.

7.2. Tasks and measurement

Participants aimed to win two games against their opponents. The first participant to win two games in each set received an extra cash reward of approximately four dollars. We measured the same game data as in Experiment 1. In addition, we used the SSQ and SUS questionnaires (Kennedy et al., 1993; Usoh et al., 2000). SSQ was used to assess participants’ motion sickness. In contrast, SUS evaluated the degree of immersion and sense of presence within the virtual Showdown game, measuring their sense of being in another person’s company. Interviews were conducted with each participant after completing the questionnaires. The interview questions were open-ended to encourage the sharing of a wide range of opinions about the participants’ VR experiences.

7.3. Procedure

The procedure used in this experiment was the same as that in Experiment 1. After training, all participants wore eye patches on both eyes while playing the game to focus only on the sound, regardless of their visual condition (Imbriani et al., 2018). For the experiment, participants played a best-of-three series of 12-point games. All participants played three games except for one who won the first two games. Players were encouraged to communicate with each other while playing the games. Researchers used filler phrases, such as “Wow, good job!” to encourage conversation if it did not start. After playing, the participants completed questionnaires and were interviewed by an experimenter. Figure 5 shows a flowchart of the experimental procedure.

8. Results for experiment 2

For the statistical analysis of the game data, we used a chi-square test for both Match and Game Win data and an independent t-test for the other data. Overall, ten of 15 people with visual impairment won matches against their opponents. For each game, people with visual impairment won 20 of 35 games, a 57.1% win rate. The “Match Win Rate” and “Game Win Rate” have no statistically significant differences between the two groups of people with visual impairment and people without visual impairment. Tables 7 and 8 show the cross tables of Match and Game Wins, respectively.

Table 7. Cross table of Match Wins (X²(1, N = 30) = 3.333, p = .068).

Match Win	People with visual impairment	People without visual impairment
Win	10	5
Lose	5	10

Table 8. Cross table of Game Wins (X²(1, N = 70) = 1.429, p = .232).

Game Win	People with visual impairment	People without visual impairment
Win	20	15
Lose	15	20

A comparison of other game data between the two participating groups revealed significant differences in all hit rates. Participants with visual impairment have a significantly higher Hit Rate than those without (t (15.798) = 5.308, p < .001), Left Hit Rate (t (14.375) = 4.121, p = .001), and Right Hit Rate (t (19.080) = 4.993, p < .001). In contrast, participants without visual impairment have a significantly higher Middle Hit Rate than participants with visual impairment (t (28) = −7.024, p < .001). The remaining measurements show no significant differences between the two groups. Figure 7 shows the game data for the two groups in Experiment 2.

Figure 7. Results of game data of the two participant groups in Experiment 2. (*p < .01, **p < .001).

Regarding the results of the questionnaires, we used the t-test for the SUS and the Mann-Whitney U test for the SSQ. For SSQ, we followed the formula introduced by Kennedy et al. (1993): We multiplied 9.54 for nausea (N), 7.58 for oculomotor disturbance (O), 13.92 for disorientation (D), and 3.74 for total simulator sickness (TS). We found a significant difference in the SSQ-TS scores between the groups (U = 12.5, p < .05). We also found significant differences in the SSQ-N (U = 31.5, p < .05) and SSQ-O (U = 22, p < .01) scores using the Mann-Whitney U test (see Table 9). We did not find any significant differences for the other questionnaires. People with visual impairment experience less motion sickness in our VR game than those without. A likely reason is that people with visual impairment are accustomed to an environment without vision, whereas people without visual impairment are less familiar with this environment and experience more motion sickness.

Table 9. Mean (and standard error) of scales from questionnaires between the two groups in Experiment 2 (*p < .05, **p < .01). SUS uses a 7-point Likert scale and SSQ value is 0–3. All values of SSQ are weighted based on Bimberg et al. (2020).

Questionnaires	People with visual impairment	People without visual impairment
SUS (Presence Questionnaires)	5.85 (0.31)	5.089 (0.41)
SSQ-Total sickness*	12.88 (5.95)	32 (6.42)
SSQ-Nausea*	15.11 (5.31)	34.19 (6.59)
SSQ-Oculomotor disturbance**	9.47 (3.75)	27.79 (5.22)
SSQ-Disorientation	9.28 (4.64)	30.16 (10.26)

Conversation data were collected by recording videos during the experiment. We also conducted conversation analysis between people with and without visual impairment to examine their emotional reactions to the game. The main objective was to understand the different emotions experienced during conversations while playing. After the experiment, we transcribed all statements made by the participants based on the recorded videos. Two researchers rated the emotions of the sentences in the transcription based on the valence-arousal dimensions originally presented by Russell (Camras, 1980; Russell, 1980). A representative emotion was selected from each category expressed in the interviews. Cohen’s Kappa was used to determine inter-rater reliability. Table 10 presents the corresponding agreement and Cohen’s Kappa for each category. Cohen’s Kappa for each category was above 0.7, indicating good agreement between the ratings of the two researchers for each category. After rating, the researchers agreed on the code. Based on this analysis, all participants reported experiencing enjoyment. In addition, the high-arousal, positive-valence category, which includes the emotion of enjoyment, had the highest proportion of all categories except neutral. Figure 8 illustrates the proportion of each emotion observed in the conversation analysis.

Figure 8. Percentage of coding results for each emotion.

Table 10. Table of code category titles, emotions included in the category, corresponding agreements (percentages), and Cohen’s Kappa.

Category	Emotions included in the category	Corresponding agreement	Cohen’s Kappa
Enjoy (High-Arousal, Positive-Valence)	Enjoy, Happy, Delighted, Glad, Excited	99%	0.96
Tension (High-Arousal, Negative-Valence)	Tense, Frustrated, Angry, Afraid, Annoyed	91%	0.82
Relaxed (Low-Arousal, Positive-Valence)	Relaxed, Calm, Sleepy, Tired, At ease, Serene	85%	0.73
Bored (Low-Arousal, Negative-Valence)	Droopy, Bored, Gloomy, Depressed	95%	0.86
Neutral	Informatic, Objective, Emotionless	94%	0.91

This outcome indicates that people with and without visual impairment enjoyed the game.

8.1. Findings from experiment 2

Through conversation analysis, we evaluated participants’ reactions to the game based on valence-arousal dimensions (Russell, 1980).

• Enjoy (High-arousal, Positive-valence): All participants mentioned that the game was enjoyable.

• Tension (High-arousal, Negative-valence): Nine of 15 with visual impairment and eight of 15 without visual impairment talked about tension during the game, mainly when they anticipated that the ball would approach them.

• Relaxed (Low-arousal, Positive-valence): Three of 15 with visual impairment and one of 15 people without visual impairment reported being calm while playing the game.

• Bored (Low-arousal, Negative-valence): Five out of 15 with visual impairment and five out of 15 without visual impairment were bored, especially when the ball stopped in the middle of the table while playing the game.

During the postgame interview, we asked the participants follow-up questions regarding the meaning of their comments. All participants with visual impairment reported enjoying the VR Showdown. The common response was “fun” for the sound feedback of the Showdown ball and for the haptic response when they hit the ball. PVI 2–12 said noticing a rolling ball hit by sound and vibration was fun. PVI 2–9 said the system’s audio narration of saying “reset” when the ball has moved to its original position and stating the score when one of the participants has scored was interesting because it is like a judge’s announcement in the original Showdown game. PVI 2–14 felt excited because the narration updated the score as it increased after hitting the ball. With this narration, PVI 2–11 gained confidence as it became clear whether the ball was hit well.

Most people without visual impairment also reported that they enjoyed the VR Showdown because it was a new experience for them to play the VR game while blindfolded. They initially felt frustrated by the visionless environment, which helped them to focus on the sound of the ball. PNV 2–5 mentioned that he initially felt cramped by the visionless environment, but this feeling disappeared after he noticed that people with visual impairment could play the game competitively in such an environment. Furthermore, PNV 2–11, 10, and 12 mentioned that they could understand the visionless environments of people with visual impairment by covering their eyes. In addition, PNV 2–12 mentioned the need to understand and consider the difficulties of people with visual impairment, as well as participants who played the game together. Many participants concentrated on learning the sense of distance from the ball. PNV 2–7 focused on the sound of the ball to become accustomed to the distance of the ball. PVI 2–8 and PNV 2–8 mentioned that they could learn the sense of distance from the ball through vibrations in the controller. The participants recognized the relationship between the sound and position of the ball through the controller’s vibration when they hit the ball.

PNV (People without visual impairment) 2–7: It is necessary to get the sense of the distance of the ball. So, I was focused on how much the volume of the ball’s sound got louder when hitting the ball.

PVI (People with visual impairment) 2-8: I keep moving the controller to see whether it vibrates. After hitting the ball, I felt the vibration and got a sense of the distance of the ball.

From the VR Showdown, people with and without visual impairment became aware and could communicate with each other. PNV 2–11 and PNV 2–5 recognized that the opponent was hitting the ball through awareness of the sound from the opponent’s area. PVI 2–3 noticed that the opponent strongly desired to win through the scratching sound of the Showdown table as he moved the controller to hit the ball.

PNV 2-11: I recognized the opponent hitting the ball by the sound of hitting the ball from far away.

PVI 2-5: I recognized the opponent starting to hit the ball, and I noticed the sound of the ball movement was different from the first. This allowed me to know that the opponent was playing hard in the game.

PVI2-3: I kept hearing the scratching sound from the table because the opponent had a strong desire to win.

People without visual impairment thought positively about people with visual impairment who eagerly participated in games. PNV 2–6 noted that the VR Showdown game would be helpful for other people with visual impairment.

PNV 2-6: There is nothing much to put energy into for people with visual impairment, so I liked seeing my partner player pour her energy into this game. From my experience, people with visual impairment are usually passive in playing a game, and I think it would be great to have more activities (like this game) to put their energy into.

Many people with visual impairment unconsciously made a sound like “wak” when they tried to hit the ball with force. These sounds made participants without visual impairment feel that those with visual impairment were enjoying the game.

PVI 2-14: The reason for the sound was to hit the ball and score a goal with a strong hit.

PVI 2-2: In the original Showdown game, players should be quiet and not make a loud sound to hear the ball sound. But since it doesn’t interfere with the sound of the ball, we can make a sound in this version of the game.

PNV 2-2: I noticed that people with visual impairment who made a noise were enjoying the game. This sound comes out involuntarily. It is only triggered when they are engaged in the game.

Participants encouraged each other by saying, “Let’s play hard!.” Participants also said, “Hit it!” when the opponent was supposed to hit the ball. In one case, they cheered each other’s scoring.

PNV 2-1: (On the timing to hit for PVI) “Hit it!”

PVI 2-1: (When PNV scored a score) “Oh! You got the goal!”

PVI 2-1: (When PNV scored a goal) “You got the goal in one shot!”

PVI 2-8: “Let’s play hard!”/PNV2-8: “OK!”

After the game, participants with and without visual impairment had conversations about their game experience. PVI 2–9 and PNV 2–9 discussed holding the controller during the game. After the first match, PVI 2–10 and PNV 2–10 discussed how to hit the ball.

PVI 2-9: Did you have difficulty holding the controller?

PNV 2-9: I needed to hold a controller horizontally not vertically. That’s why I couldn’t block your ball. I only thought of scoring a goal, not of defending the ball.

PVI 2-10: Should we hit the ball when it comes closer?

PNV 2-10: As the ball comes closer, you will feel the vibration.

PVI 2-10: When do I feel the vibration?

PNV 2-10: When the ball comes, you feel the vibration.

PVI10: I got it.

Most participants agreed that there were limited activities that people with and without visual impairment could play together. PVI 2–15 appreciated the opportunity to participate in inclusive games.

PVI 2-15: It’s great for people with visual impairment and those with normal vision.

9. Discussion

We introduced a new opportunity for people with visual impairment by developing a collaborative VR game and conducting experiments involving interactions between people with and without visual impairment. Our VR Showdown used HMD to track the position and orientation of the player’s head and provided proper spatial audio feedback using HRTF technology. Compared with previous single-event modes of Virtual Showdown, which did not allow rallies (Wedoff et al., 2019), our PvA mode is a full-rally game capable with an AI Agent. We also demonstrated that people with and without visual impairment could play the VR Showdown together and could have rewarding social interactions. This is an important difference from previous studies that focused on the individual interactions of people with visual impairment (Siu et al., 2020; Wedoff et al., 2019; Zhao et al., 2018). We found that visually impaired individuals expressed and communicated their emotions to other players during games. Even after playing the game, participants with visual impairment had several conversations with other players. In previous studies, playing games with others evoked meta-gaming (Mueller et al., 2009) and provided rich resources for talking with other players (Segal, 1994). Likewise, in our case, the experience of playing the VR Showdown allowed people with and without visual impairment to talk and engage in rewarding social interactions.

People with and without visual impairment enjoyed the VR Showdown remotely through network connections. The participants impacted each other’s actions in real time by sending the shared object, the Showdown ball, to the other player’s area and making them block the ball. In our experiment, participants developed a strategy to score a goal by determining the other person’s movement. One strategy was sending the ball to an empty space the opponent had not defended. Participants focused on the sounds made by the opponent while playing the game. If the opponent made a scratching sound on the table with the racket, even with the ball in front of the player, the player kept moving their racket randomly, regardless of the ball’s position. Thus, it might be an opportunity for the player to score a goal, as mentioned in the interview.

This study provided insights into VR systems for people with visual impairment.

- Non-isomorphic VR interactions expand the VR experience of people with visual impairment. Two approaches have been considered for designing VR interfaces: isomorphic and nonisomorphic interactions (Knight, 1987). Isomorphic interactions are almost the same as real-life interactions; thus, they support natural interactions. However, this is limited by the input devices and the human itself (LaViola et al., 2017). For example, if the input device has a certain tracking area, users cannot move far from the tracking area. In addition, humans cannot select objects that are located away from arm’s length. A non-isomorphic interaction method was developed to overcome these limitations. Non-isomorphic interactions are not natural but often permit more usability and performance than isomorphic interactions (Bowman & Hodges, 1997; Poupyrev et al., 1998). For example, the Go-Go technique allows users to grasp a distant object by stretching their virtual hand without walking toward it (Poupyrev et al., 1996). For the VR Showdown, we developed a non-isomorphic interaction (i.e., holding) that is unavailable in a real-world Showdown. The holding feature allows the player to attach the ball to the racket and drag it as intended while holding the controller’s trigger button. During the interviews, the participants mentioned that the holding function of the VR Showdown was easy to understand and efficient in controlling the ball, allowing them to use various strategies to win the game. This non-isomorphic feature provides people with visual impairment with a satisfactory VR experience.

- Collaborative systems should give players a sense of synchronicity. De Kort and Ijsselsteijn (2008) indicated that social presence is built based on awareness of others through synchronicity. During the VR Showdown, the ball positions were presented synchronously to both players in real-time. The system also provides a narrative to share the current status of the game. This synchronicity allows people with and without visual impairment to experience co-existing in the same time and space, even though their real spaces are separated. A collaborative game should give participants a feeling of synchronicity by sharing game information in real-time.

- By sharing the VR experience of people with visual impairment, people without visual impairment get to understand people with visual impairment. Previous collaborative games for people with visual impairment have led players to focus on their abilities. Kinaptics allows people with visual impairment to play a collaborative game using multimodal feedback (e.g., sound, haptic, and wind), while people without visual impairment play using their vision (Grabski et al., 2016). Gonçalves et al. (2021) created a collaborative game in which people with and without visual impairment played asymmetric roles. In their collaborative game, people with visual impairment relied on auditory feedback, whereas people without visual impairment used their visual capability. However, the researchers mentioned that it was difficult for participants to understand each other empirically because they could not experience each other’s tasks (Gonçalves et al., 2021). By contrast, our study provided the same virtual environment for both types of participants. People without visual impairment could experience an environment where sight was lost, and had to depend on auditory information. In the interviews, many people without visual impairment mentioned that the VR Showdown helped them understand the sightless, sound-only environment that people with visual impairment experience. They mentioned they could connect better with people with visual impairment by playing together in a virtual environment where sight was excluded.

10. Limitations

In Experiment 1, we found age differences between participants with and without visual impairment, which could affect their motor skills and be a confounding variable. To mitigate this, we matched the age groups used in Experiment 2 for a more controlled comparison. Additionally, we could not assess the users’ sense of presence and motion sickness in Experiment 1 because we did not conduct related questionnaires. We administered the SUS and SSQ in Experiment 2 to address this issue.

In Experiment 2, we blindfolded all participants, following the rules of a real-world Showdown game. However, participants with visual impairment had a wide spectrum of impairment and different prior visual experiences. Having them play the VR Showdown without blindfolds would have provided more accurate data by matching their experiences of playing other VR games. In our future work, an alternative design will be considered to provide well-matched daily experiences for people with and without visual impairment rather than blindfolding all participants.

11. Conclusions and future work

We developed a collaborative VR Showdown program accessible to the visually impaired. We used HRTF so that people with visual impairment could detect a rapidly moving Showdown ball using auditory feedback. In addition, we used HTC Vive Pro^TM to track the participants’ head and hand positions and movements in real time. We tested the system on individuals with and without visual impairment. People with visual impairment could detect the position of a Showdown ball with an accuracy of over 90%. Two-thirds of the participants with visual impairment were matched with people without visual impairment. In the PvP mode, all participants reported having fun and recommending the game to other users. This study provides a potential avenue for further research on VR applications accessible to people with visual impairment. Future research should focus on enriching the interactive experience and expanding the range of e-sports for people with visual impairment by extending it to other paralympic games.

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (No. 2018R1D1A1A02085645); This work was also supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI22C0619).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Korea Creative Content Agency [R2021040093]; Korea Medical Device Development Fund [NTIS9991006786, KMDF_PR_20200901_0113]; Ministry of Culture, Sports and Tourism and Korea Creative Content Agency [R2021040093]; Korea Health Industry Development Institute (KHIDI) [HI22C0619]; National Research Foundation of Korea (NRF) [2018R1D1A1A02085645].

Notes on contributors

Hojun Aan

Hojun Aan is currently a PhD candidate in the Department of Human-Computer Interaction at Hanyang University in Seoul, South Korea. His research interests include human-computer interaction (HCI), virtual, augmented, and mixed reality (VR/AR/MR), and gamification.

Sangsun Han

Sangsun Han is currently the post-doc researcher in bionics research center of Korea Institutes of Science and Technology. He is interested in digital therapies based on XR (eXtended Reality).

Kibum Kim

Kibum Kim is a full professor in the Department of Human-Computer Interaction at Hanyang University, South Korea. He received a bachelor’s degree from Korea University, a master’s degree from the University of Illinois at Urbana–Champaign, and a Ph.D. from Virginia Tech, all in computer science. His research focuses on HCI, CSCW, CSCL, VR/AR/MR, and computer education.