Effect of Virtual Intervention Technology in Virtual Vocational Training for People with Intellectual Disabilities: Connecting Instructor in the Real World and Trainee in the Virtual World

Abstract

There are various social support services to help people with intellectual disabilities (ID) find jobs for their independent social and economic activities in life. With the advancement of virtual reality (VR) technologies, vocational training can take place in a virtual environment (VE) without temporal and spatial limitations. Therefore, opportunities for people with ID to receive professional vocational training in a VE continue to expand. Accordingly, this study proposes that virtual intervention (VI) technology can improve the effects of virtual training and learning transfer from the virtual to the real world. We defined VI as a supportive activity that assists trainees in a VE according to their individual ability and training situation through an instructor’s control in the real world. We designed the virtual intervention content (VIC) as intervening virtual objects and applied them to a virtual barista job training program. We developed the virtual intervention interface (VII) for smoothly supporting the interaction between the trainee in the VE and the instructor in the real world, especially for people with ID or beginner trainees. In this study, we derived statistically significant findings that demonstrated the effects of VI technology through two experiments conducted with 39 participants with ID. Firstly, VIC showed an effective intervention function that helped the participants solve problems during virtual vocational training. Also, we observed differences in the intervention effect according to the sensory perception characteristics of the participants. Secondly, trainees’ barista job performance rates improved by 37.43% after receiving virtual training. Moreover, the effects of training were largely maintained in the job performance assessment after three weeks in an actual cafe situation. This suggests that the effects of VI-based virtual training are meaningful not only within the VE but also in the real world for the transfer of learning. Furthermore, the VI method was significantly more effective and efficient than the conventional method using the human instructor’s verbal and physical intervention in terms of time, frequency, and success rate.

1. Introduction

Generally speaking, people with intellectual disabilities (ID) refer to those who are cognitively restricted in performing daily or social activities due to lacking or underdeveloped intellectual capabilities caused by a delay in mental development (American Psychiatric Association, 2021; Räty et al., 2016). IDs as such continue throughout one’s lifetime and require more care and attention by a caretaker compared to other disabilities. Moreover, the number of people with ID has increased globally due to changes in the diagnostic criteria, social awareness, and environmental factors (Zablotsky et al., 2019).

Recently, attitudes toward social support for people with ID have shifted from caregiving to self-reliance. Accordingly, job placement programs, vocational training, etc. are offered at the regional and national levels. Institutions such as vocational training centers and specialized schools for ID have built physical mock-up spaces identical to the actual workplace environment to provide vocational training. However, there are limits to providing job training or experience in various occupations because it is necessary to simulate the same actual environment and tools as a physical workplace.

Meanwhile, virtual reality (VR) has gained renewed attention as a medium that overcomes the limits of conventional methods by providing opportunities for new experiences without time and space constraints, while ensuring a safe learning environment. The term “VR” first emerged in 1989 when Jarrow Lanier used it to describe “an immersive visual experience created by a computer.” VR is a technology that allows the virtual representation of real or imaginary objects in 3 D computer-generated realms to provide immersive and realistic experiences (Cobb et al., 2000). VR can be divided into immersive, semi-immersive, or non-immersive VR, depending on the level of immersion and systems used (Bamodu & Ye, 2013). Among these, immersive VR completely engages the user inside a computer-generated realm. Through a head-mounted display (HMD) worn on the head of a user and a tracking device, immersive VR technology presents a 360-degree 3 D virtual scene to create immersive and interactive experiences. Owing to the recent release of new devices that allow precise motion recognition using acceleration and gyro sensors, as well as the advancement of HMD technologies, immersive VR is heralded as a way to generate a new platform that could bring innovations to training and learning.

What is also notable is that VR-based job training not only convincingly replaces real environments, but also provides an immersive learning experience tailored to the individual needs of trainees (Huang et al., 2010; Petrakou, 2010). As such, VR technology could be particularly useful to train those with limited cognitive skills (Rose et al., 2000; Standen & Brown, 2005; Wilson et al., 1997).

Particularly, people with ID have a higher probability of getting a job through training and learning than those with other developmental disabilities because they can learn and engage in social activities by training (Hazmi & Ahmad, 2018). However, people with ID are generally known to have a low attention span, low motivation to learn, and difficulty with iterative learning (Smith et al., 2014). At the same time, sensitivity, reactivity, and cognitive characteristics for personal abilities and stimuli may be manifested differently depending on the degree of disability, cause, or existence of multiple disabilities (Buntinx & Schalock, 2010). Fortunately, training in a virtual environment (VE) can be an easier, better solution than training in a real environment because it can consider such individual characteristics of disabilities and provide personalized support by enabling stimulation and procedural operations (Stichter et al., 2014).

However, there is a relative lack of VR technologies designed for training and learning of people with ID. The main reason may be attributed to the size and profitability of the market since most virtual training contents are developed for non-disabled people. Such general training content is difficult for people with ID to use in terms of the level of difficulty, method of use, and time of concentration. In addition, many existing studies for those with intellectual or developmental disabilities focused on improving social skills, daily life skills, and emotional skills for children and adolescents rather than acquiring skills through vocational training (Glaser & Schmidt, 2022; Mesa-Gresa et al., 2018).

With this backdrop, we carefully considered what kind of technological support is needed to effectively conduct VE-based vocational training for those with ID and for beginning trainees who are not familiar with the VE. In this study, we propose a virtual intervention (VI) technology that facilitates VE-based training by delivering seamless support and guidance in accordance with a trainee’s cognitive and performance skills.

2. Literature review

2.1. Prior studies on VR technology: From training to learning

Education and training are the areas in which VR technology is most commonly used. In particular, VR technology has been widely used for training in medical, military, aerospace, and industrial domains, attributing to its stability, accessibility, and economic efficiency. Stone (2001) has introduced cases for the successful application of VR-based training in the above-mentioned areas.

VR technology has been widely used for job training and experience in various occupations (Lee et al., 2010; Li et al., 2017; Yang et al., 2010). Recently, VR technology has been expanded and utilized in more popular fields for various purposes such as practical tools for education (Hu-Au & Lee, 2017), psychological stability (Wilson & Soranzo, 2015), emotional skills (Yuan & Ip, 2018), health (Chan et al., 2011), new experiences in leisure time (Weiss et al., 2003), etc. Given that job training would require more repetitive and focused sessions, there have been studies on the mental and physical impacts of consistent exposures to the VE on human subjects (e.g., Chang et al., 2020; McCauley & Sharkey, 1992).

Recently, there have also been evaluations on virtual training and its learning effect. Based on the various literature, Abich et al. (2021) suggested that VR training shows evidence on positive effects on psychomotor performance, knowledge acquisition, and spatial ability. In addition, Renganayagalu et al. (2021) demonstrated that VR-based training is useful for improving cognitive skills such as memory, learning and remembering procedures, and psychomotor skills. They suggested that VR solutions could be utilized for job training performed in relatively unsafe or unique environments. Bohne et al. (2021) argued that evidence-based principles have to be applied to VE-based training programs for improving the performance of trainees from an educational perspective. These prior studies show that VE-based training signifies more than the possibility to train in a VE that mimics the real world.

2.2. Prior studies on VR technology for people with intellectual disabilities

Virtual training for people with ID needs to be based on training methods customized to the individual needs of trainees with different cognitive features (Rose et al., 2000; Standen & Brown, 2005; Stichter et al., 2014). We referred to the prior studies that investigated people with ID served as the subject for VE. Among them, Kim et al. (2021) analyzed the attention and cognitive features of people with ID in terms of the sensory stimuli provided in a VE. The study indicated the collective cognitive features of people with ID, and pointed out that exogenous-cues such as movements or flashes were more helpful for their task performances than endogenous-cues such as symbolic signs (Corbetta & Shulman, 2002). Meanwhile, Jo (2021) suggested a method to visualize the eye-tracking information of those with developmental disabilities. Hong et al. (2021) proposed a method to analyze the visual attention of people with ID in virtual training. The above-mentioned studies served as a suitable reference to infer a virtual training situation for people with ID who tend to face more challenges in performing tasks, and to identify moments when they need help.

Glaser and Schmidt (2022) explained essential design elements of the VE training system such as realistic expressions and interaction to achieve learning benefits based on the literature review of studies in which VR technology was applied to people with autism spectrum disorder (ASD). We need to fully utilize the technical advantages of VR, such as immersion, representational fidelity, and interaction, to obtain the virtual learning benefits based on a sufficient understanding of the evolving VR technology (Dalgarno & Lee, 2010; Fowler, 2015; Glaser & Schmidt, 2022). In addition, some studies analyzed the VE experiences of those with ID, ASD, and developmental disorders, as well as those related to the training of social skills required for them (Cobb et al., 2002; De Luca et al., 2021; Ip et al., 2013; Parsons & Mitchell, 2002).

2.3. Prior studies on intervention-based teaching methods in special education

There are various studies in the field of special education concerning teaching methods for those with ID in the real environment (Hazmi & Ahmad, 2018; Räty et al., 2016). Among them, we focused on the research studies that introduced intervention-based teaching methods (Wong et al., 2015). In fact, intervention-based teaching methods are commonly used in the education of those with developmental disabilities. Also, its efficacy has been empirically demonstrated, with the increasing emphasis placed on evidence-based practices (Kim & Park, 2017; Wong et al., 2015).

In the field of special education, the term “intervention” is defined as a series of supportive activities to help trainees or subjects achieve a task goal. In other words, intervention can be explained as an instructor’s supportive activities according to the trainee’s abilities and training circumstances by responding to issues caused by a gap in the trainee’s cognitive skills. For such an intervention, an instructor could use a wide range of scaffolding tools including video clips, visual cues (i.e., text, pictures, objects, or signs), and software programs. Major intervention methods include video modelling (i.e., a video-recorded demonstration of the targeted behavior that has to be mimicked by the learner), prompting (i.e., verbal or physical assistance given to learners to support their training participation or acquisition of the desired skill), and visual support (i.e., a visual resource that supports the learner to engage in a desired behavior or skill) (Douglas et al., 2011; Hammond et al., 2010; LeGrice & Blampied, 1994). In addition, there have also been studies on which intervention methods are effective for a particular situation or whether a combination of two different methods can be more effective (Kim, 2013).

While interventions are expected to be effective for both actual training and virtual training, related studies are lacking—which is the case for virtual training for both people with and without disabilities. At present, the trainees control the training process themselves during virtual training or receive feedback after virtual training based on the data accumulated within the training system. That is, in the current virtual learning environment, experts and instructors have limited opportunities to directly intervene in the training process. This indicates that there is a lack of the interface that allows a trainee in a VE to interact and communicate with an instructor in the real world. Brown et al. (2016) also stressed the necessity of an interface that enables an instructor in the real world to give information to a trainee in the VE. Such an interface is particularly necessary for trainees who need the support of an external instructor due to deficient cognitive skills or for children or seniors who are novices of a virtual training system.

3. Design and implementation

3.1. Definition of virtual intervention

We defined the following concepts to promote a novel interface through which a human instructor can guide the training of a trainee in the VE and apply intervention-based VR training for trainees with ID:

Firstly, applying the intervention-based teaching method in special education to a VE, we defined “virtual intervention (VI)” as a supportive activity that a human instructor offers to a trainee in the VE.

Secondly, we defined virtual objects offered for VI as “virtual intervention content (VIC).” Here, virtual objects refer to a virtual representation of intervention media used in real-life circumstances, such as visual clues, checklists, and video clips. As a trainee acts in a VE, an instructor intervenes in virtual training using a new intervention medium—VIC, which links the trainee in the VE with the instructor in the real one. This serves as a tool for a human instructor to directly intervene in virtual training. As seen in Figure 1, such VIC should be provided promptly by the respective training situation and methods, as is the case for face-to-face training. We focused on providing an interface that would help human instructors seamlessly reflect on their judgements and controlling activities in the VE.

Figure 1. Conceptual diagram of the intervention in real and virtual environments.

Thirdly, we developed a “virtual intervention interface (VII)” where the instructor in the real world can visualize easily the desired information in the VE using devices such as a remote control or keyboard. We additionally developed a function that allows trainees to activate VIC on their own in the VE.

3.2. Design and implementation of virtual training program for baristas

Baristas have recently become one of the jobs preferred by many people with disabilities or their guardians. Many regionally and nationally funded vocational training centers offer training courses for barista jobs to people with ID. To evaluate the effects of the proposed VI method, we developed the virtual vocational training content for barista jobs. Given that a barista job requires the acquisition of complex procedural skills (Abich et al., 2021; Renganayagalu et al., 2021), we found that VR-based learning was appropriate for the target trainees.

The virtual vocational training content for baristas was developed using the unreal engine (www.unrealengine.com). A first-person view of a trainee was visualized in a virtual space to increase immersion. Virtual objects used during barista training reflected physical phenomena such as gravity and collision. Primarily, the sense of reality was improved using sound and vibration effects, with the virtual cafe environment developed to be as similar to a real cafe as possible. All physical actions and phenomena such as holding, touching, and dropping virtual objects were reproduced to be the same as in the real world, so that interaction with objects in the VE occurs naturally. In addition, the difficulty of using controllers was minimized by using only one button on the controller and creating various hand models in a VE according to the motion of hands and the type of virtual objects (Dalgarno & Lee, 2010; Glaser & Schmidt, 2022).

Our developed VR content was composed of four core training procedures: (a) to learn how to operate the controllers for virtual training (controller operation training), (b) to learn terminology for tools used by baristas (terminology of tools training), (c) to learn how to extract espresso (espresso extraction training), and (d) to learn how to make steamed milk (milk-steaming training). The level of difficulty in virtual training was set as high to train both people with and without ID. Each main procedure was divided into sub-procedures to allow for gradual progress and iterative training. All main and sub-procedures were designed in consultation with professional baristas (Table 1).

Table 1. Composition of procedures & sub-procedures of the VR barista training.

Name of procedure	Description of training procedure	No. of sub-procedure	Example of sub-procedure training	Example of display
Controller operation training	Training to learn how to use the controllers	3	“In this step, you will learn how to pick up & touch objects. Watch the video & follow along.”
Terminology of tools training	Training to learn the terminology for tools used by baristas	15	“Find a portafilter & touch it with your hands.”
Espresso extraction training	Training on the espresso extraction process	21	“Discard the grounds remaining in the portafilter into the knock box.”
Milk-steaming training	Training on the milk-steaming process	17	“Immerse the nozzle in the steam milk pitcher.”

3.3. Design and implementation of virtual intervention

Next, we designed and developed VIC, which is to serve as an intervening object to materialize the proposed VI technology. When designing VIC, we referenced the intervention-based teaching methods used in real-world education for people with ID (e.g., Bjerrum et al., 2013; Björn et al., 2009; Cannella-Malone et al., 2013; Renkl, 2014; Waldman-Levi et al., 2019). Among them, we implemented cues and modelling as interventions necessary and easy to use in the VE, as shown in Figure 2. Shapes and types of VIC were diversified to reduce iterative expressions.

Figure 2. Examples of virtual interventions.

Considering that the training content is for the barista job, the cue intervention was developed as positional and numerical cues (i.e., time, length, frequency, temperature, etc.). The modelling intervention was implemented as hand motions and video models.

VIC in virtual training needs to be featured in a way to catch the trainee’s attention and facilitate the trainee’s perception and cognition of information. Since the VE provides different intensities of sensory stimulation and mechanism of information delivery from the real environment, it is necessary to understand the perceptual and cognitive characteristics of trainees with ID in the VE. Referring to the prior studies (Kim et al., 2021), we aimed to achieve the following goals:

• All VIC were implemented to express visual, auditory, and haptic for exogenous expressions. Specifically, movement (up, down, left, right, rotation, tilt, and scale) and color (glitter and transparency) attributes were used as exogenous visual expressions. Sound source files were connected to enable exogenous auditory expressions. The intensity and duration attributes of both controllers were used and adjustable for exogenous haptic expressions.

• Given that the attention and perception of trainees with ID could vary depending on the sensory medium that delivers the information, VIC was designed in different versions (visual, auditory, and haptic) that provide the same information. As shown in Figure 3, a visual medium is used as an object for VIC if a trainee tends to react well to visual cues. Also, an auditory medium is used for delivering verbal information if the trainee reacts well to auditory cues.

Figure 3. Example of virtual intervention content for each sensory type.

• Attributes of all visual, auditory, and haptic expressions could be changed in real-time. Diverse types of VIC were implemented, as shown in Figure 4, so that an instructor can select among various expressed objects that have the same intervention function to avoid repeated expressions.

Figure 4. Example of diverse types to express visual position cues.

• VIC was implemented independently from the training content for the connection with existing various contents. VIC could be easily added to the existing virtual content as a plugin.

• As shown in Figure 5, VII was used by an instructor to control the VIC easily in the real world with a keyboard or remote controller. Specific buttons on a keyboard or remote controller were used to activate or deactivate specific VIC while the instructor monitors the status of a trainee’s virtual training.

Figure 5. Virtual intervention content controlled by an instructor.

4. Experimental study I on the effects of virtual intervention

4.1. Experimental design and procedure

In the first experiment, we investigated whether the VIC developed in the above-mentioned ways could assist the training of those with ID. In particular, we examined whether the designed virtual content performs its intervention role in a virtual training environment. The experiment was conducted with the IRB approval (IRB approval number: 2020_100_HR).

Before receiving the virtual training, the participants were given instructions on how to use an HMD and controllers. Among the procedures in the virtual barista training content, our experiment focused on the terminology of tools training and the espresso extraction training procedures. The participants were instructed to ask for help if they faced any difficulty in performing a sub-procedural task. As seen in Figure 6, all trainees were provided with the same video modelling intervention. While visual cues may be sufficient for the terminology training procedure, the espresso extraction training procedure required additional information due to the complex nature of training procedures. Thus, we chose video modelling as the intervention method in our experiment as it is suitable for both terminology training and espresso extraction training (Douglas et al., 2011; Hammond et al., 2010).

Figure 6. Example of visual cue and video modelling.

The subjects comprised 21 people (15 men, 6 women) with mild to moderate ID aged 18 above and below 50 (mean age = 30.6 years old). They enrolled in the vocational training center specifically established for people with developmental disabilities. They did not have problems with visual, auditory, and haptic sensory functions and had no previous experiences in training for barista jobs.

After receiving instructions on the methods and procedures for performing the main task, the subjects participated in the experiment by wearing the VIVE Pro Eye HMD and holding the HMD controllers in both hands. The stimuli presented in the experiment consisted of the visual and audio outputs in the HMD. Once the subjects completed the sub-procedural tasks by voice-enabled instructions, they received instructions for the next sub-procedure task, such as “Turn on the coffee grinder machine” and “Pull the lever of the coffee grinder machine.” A video instance was presented if the subjects had difficulty performing the task. The presented video modelling was only delivered by the movement of visual expressions without auditory or haptic feedback.

As shown in Figure 7, we recorded the data of subjects’ eye-gaze, performed tasks, provided VIC, and the subject’s motion for each performed task. To increase the data accuracy, we reconfirmed the automatically recorded data based on the eye-gaze and recorded video data to exclude meaningless or erroneous inputs by trainees from the experimental data. From the collected data, we analyzed whether the subjects successfully performed the task after receiving the virtual intervention. We also measured how differently the video modelling intervention of exogenous visual expressions affected to the subjects with different cognitive characteristics.

Figure 7. Experimental environment & data collection screen.

4.2. Results and consideration

Each subject had to perform 36 sub-procedures during the experiment. In total, 144 video modelling intervention instances were provided to the subjects. The task success rate of the subject’s operation after receiving the video model instance was 88.46% in the terminology training procedure, while the success rate in the espresso extraction training procedure was 95.45%.

We also examined whether the effects of visual stimuli of the VIC vary by the cognitive and perceptual characteristics of people with ID in the VE, as discussed in Kim et al. (2021). We first classified the data of the seven subjects whose data on cognitive and perceptual information were available. We then analyzed the correlation between the inverse efficiency (IE)¹ of visual and auditory cues and the inverse number of tasks performed successfully by providing video modelling.

As presented in Figure 8, Spearman’s correlation analysis showed a positive correlation where the success rate of the video intervention increased with the increasing the efficiency of exogenous visual cues for movements (r_s = .68, p=.09). The effect of the color cue and the auditory cue showed a moderately positive correlation (r_s = .50) while the effect of the auditory cue showed a moderately negative correlation (r_s = −.53). However, the correlations were not statistically significant. These results indicate that when trainees have the characteristics of high attention and perceptual ability on the visual movement cue, they tend to perform more successfully by utilizing the video intervention expressed as visual movement cues rather than auditory and tactile cues.

Figure 8. Correlation between the efficiency of cues and success rate of video intervention.

5. Experimental study II on the effects of virtual intervention

5.1. Experimental design and procedure

After confirming the positive effect of video intervention in Experiment I, we then examined the effect of training and learning transfer using the VIC and VII in the virtual barista content. We conducted the second experiment to analyze the effect of our VI method through a comparison between the proposed VI method and the traditional instructor’s intervention method (IRB approval number: 02110-0002-01). An instructor intervention (II) refers to a teaching method in which an instructor in the real world uses verbal and physical promptings to teach a trainee in the VE. The method includes both physical facilitation and verbal facilitation. In the physical facilitation, the instructor physically controls a trainee’s hand by holding a controller or the trainee’s head wearing an HMD. For the verbal facilitation, the instructor gives verbal assistance to the trainee.

To examine whether the effect of virtual training merely stems from the novelty effect such as temporal interests in a new form of stimulus mentioned by Parong and Mayer (2018), we also measured the level of learning transfer. Notably, previous studies of the effect of virtual training methods were mostly conducted by comparing the performances before and after the subjects’ participation in virtual training, or by comparing groups participating in virtual training with those that did not. This indicates that we need additional evidence on whether skills obtained through virtual training last long enough to be transferable to a real environment.

Despite the importance of the assessment of learning transfer effects, related studies have been insufficient so far. The potential reasons are attributed to the difficulty of tracking whether trainees continue to apply the acquired skills in a real-life environment. Also, it is challenging to accurately measure learning transfer effects due to the lack of generalized measurement methods.

This study used observations and interviews among the six methods for measuring learning transfer effects suggested by Caravaglia (1993). Observations were examined based on recorded video data as to whether the subjects performed the expected behavior, while interviews were conducted only with selected subjects with no restrictions on verbally communicating their opinions and learning experiences.

The experiment was conducted on 18 people with ID at the vocational training center for developmental disabilities. The subjects were recruited when there was no problem with visual, auditory, and tactile sensory functions without any vocational training experience in barista jobs. While we initially recruited 25 subjects with mild to moderate ID, two were excluded due to their autism and severity of disabilities. Two trainees were also excluded because they could not continuously participate in the long-term experiment. We excluded three participants who showed excellent barista job performance skills in the pre-measurement stage. By the random matched group design, nine subjects (6 men, 3 women at a mean age of 24.56 years) were assigned to the experimental group for the VI method, whereas nine subjects (4 men, 4 women at a mean age of 21.60 years) were assigned into the control group for the II method.

As visualized in Figure 9, this experiment was conducted in four steps, namely pre-measurement, intervention-measurement, post-measurement, and transfer-measurement.

Figure 9. Experiment II procedure with four measurement steps.

1. In the pre-measurement step, we measured the subject’s ability to perform sub-procedures of espresso extraction training and milk-steaming training in an actual cafe environment.

2. In the intervention-measurement step, the subjects in both groups performed a total of three virtual training sessions once a week. During virtual training, VI and II were provided respectively. We observed whether the subjects performed the sub-procedure successfully according to the VI and II.

3. In the post-measurement step, the barista job performance ability of each subject was measured in the same way as for the pre-measurement.

4. The transfer-measurement step was carried out three weeks after the post-measurement. The respective performance level of subjects was measured in an actual cafe setting for the espresso extraction and milk-steaming procedures. To evaluate the ability to apply the learned content to a new situation, we added a new task, making a cappuccino, as an indication of learning transfer.

The information expressed by the VIC was converted into a language terminology and physical guide to balancing the amount of information between the virtual and instructor’s interventions. Also, the instructor’s intervention was presented according to the pre-defined scripts to minimize the instructor’s subjective perspective or opinion. The experimental group was provided with VIC consisting of audio instructions and video models as the virtual intervention. All visual and auditory information was included in the video to minimize the difference in attention and cognitive ability according to the sensory characteristics of those with ID.

As seen in Figure 10, subjects in the experimental group were asked to activate the VIC on their own to obtain necessary information without the instructor’s intervention. Figure 11 shows that subjects in the control group were guided to ask the instructor for help if they had difficulty achieving the sub-procedure and reaching goals. For the request of the subject, the instructor helped the subject with verbal and physical guides.

Figure 10. Virtual intervention content for the experimental group.

Figure 11. Instructor’s intervention for the control group.

5.2. Results and consideration

We used the Mann-Whitney test (p<.05) for the between-group analysis while the within-group analysis was conducted using the Wilcoxon signed-rank test (p < .05). For the analysis of the virtual training effect of the two groups, we compared job performance rates measured in the pre-, post-, and transfer-measurement steps. The job performance rate was calculated by dividing the sum of the points given according to success or failure of the sub-procedures by the total score. For all sub-procedures, we gave two points for success after one attempt, one point for success after two or more attempts, and 0 points for a failure. For example, the procedure of espresso extraction training includes a total of 21 sub-procedures scored from 0 to 2 points. Hence, the range of the total scores that each subject could receive was from 0 to 42 points. Three procedures were evaluated in the transfer-measurement step as we added the cappuccino-making procedure as a learning transfer task. The results of the job performance rates are shown in Table 2 and Figure 12.

Figure 12. Comparison of job performance rates of virtual intervention and instructor intervention in pre-, post-, and transfer-measurement steps.

Table 2. Comparison of job performance rates in pre-, post-, and transfer-measurement steps.

Intervention type	Pre-measurement step		Post-measurement step		Transfer-measurement step
	Espresso extraction	Milk- steaming	Espresso extraction	Milk- steaming	Espresso extraction	Milk- steaming	cappuccino
Virtual intervention (n = 9)	48.39% (36.78/76)		85.82% (65.22/76)		82.11% (67.33/82)
	55.03% (23.11/42)	40.20% (13.67/34)	87.30% (36.67/42)	83.99% (28.56/34)	86.24% (36.22/42)	74.51% (25.33/34)	96.30% (5.78/6)
Instructor intervention (n = 9)	54.68% (41.56/76)		65.64% (49.89/76)		68.70% (56.33/82)
	56.08% (23.56/42)	52.94% (18.00/34)	70.90% (29.78/42)	59.15% (20.11/34)	73.28% (30.78/42)	62.09% (21.11/34)	74.04% (4.44/6)

Firstly, based on the job performance success rates of the VI group, (see Table 3), the average job performance rates were 48.39% in the pre-measurement step and 85.82% in the post-measurement step. This indicates a 37.43% improvement in the job performance success rate, which was statistically significant. In the transfer-measurement step measured after three weeks of the post-measurement, the average performance rate was 82.11%, indicating a 3.71% decrease. There was no statistically significant difference between the post-measurement and the transfer-measurement. Average job performance rates improved statistically significantly from 55.03% to 87.30% for espresso extraction learning and from 40.20% to 83.99% for milk-steaming training procedures. In the espresso extraction, the difference in job performance rates between the post-measurement (87.30%) and transfer-measurement (86.24%) was not significant, while job performance rate decreased significantly for the milk-steaming procedure.

Table 3. Statistical results in job performance rates.

Intervention type	Espresso extraction			Milk-steaming
	Pre: Post	Post: Transfer	Pre: Transfer	Pre: Post	Post: Transfer	Pre: Transfer
Virtual intervention	p = .008*	p = .625	p = .004*	p = .004*	p = .016*	p = .004*
Instructor intervention	p = .008*	p = .992	p = .031*	p = .402	p = .477	p = .262

(*p < .05).

Secondly, in the case of the II group, the average job performance rates were 54.68% in the pre-measurement step, 65.64% in the post-measurement step, and 68.70% in the transfer-measurement step. There was a significant difference between the values measured in the pre-measurement (56.08%) and post-measurement (70.90%) only for the espresso extraction training procedure.

For the comparison, job performance success rates of the VI and II groups in the pre-measurement step were 48.39% and 54.68% respectively. However, there was no statistically significant difference (p = .258), meaning no difference in the initial job performance ability between these groups. The results of comparing job performance success rates between the two groups showed statistically significant differences in the transfer-measurement step for the espresso extraction.

Next, we compared the frequency and duration of using two types of interventions between the experimental and control groups based on the data recorded in the intervention-measurement step. The intervention efficiency rates, that is, the rate of successful performance with the intervention in the VI and II groups were 84% and 31% on average respectively. As shown in Table 4, the ratio of requesting additional interventions after the failure in the first attempt was significantly lower in the VI group than in the II group.

Table 4. Ratio of those requesting additional intervention due to a failure of the first intervention (No. of additional interventions/No. of first interventions).

Intervention type	Percentage of additional interventions in the 1^st measurement (No. of additional provided/No. of first provided)		Percentage of additional interventions in the 3^rd measurement (No. of additional provided/ No. of first provided)
	Espresso extraction	Milk-steaming	Espresso extraction	Milk-steaming
Virtual intervention	6.67% (1/15)	28.57% (2/7)	22.22% (2/9)	0.00% (0/2)
Instructor intervention	89.70% (61/68)	63.24% (25/41)	60.98% (43/68)	66.67% (26/39)

Table 5 shows the average execution times of the virtual and instructor interventions. The average execution time of each intervention was calculated as the average execution time of the intervention provided during the two main procedures. The execution times of the two groups decreased as the sessions progressed. Also, the execution time of the VI group was shorter than that of the II group.

Table 5. Comparison of the average execution time of intervention.

Intervention type	1st intervention measurement	2nd intervention measurement	3rd intervention measurement
Virtual intervention	587 s	526 s	388 s
Instructor intervention	1,010 s	578 s	434 s

As summarized in Table 6, 10 subjects (6 for VI, 4 for II) participated in the interview. Among six subjects in the VI group, five subjects (83%) said that it was helpful to watch videos or to hear an auditory guide using the intervention button whenever they wanted. One person (17%) was rather negative, responding that she did not notice the clear effects of the VIC because she could perform may procedures without the assistance of the intervention information. Among four participants in the II group, two answered that the instructor’s explanation was helpful. the other two shared that it was more effective to watch the video than to listen to the instructor’s explanation because they could not listen well while wearing the HMD.

Table 6. Interview results about the effect of virtual barista training.

Intervention type	No. of subjects	Is the intervention during virtual training effective?	Frequency (%)
Virtual intervention	6	Yes	5 (83)
		No	1 (17)
Instructor intervention	4	Yes	2 (50)
		No	2 (50)

Regarding the VR device experiences, eight out of 10 people said they had no difficulty or inconvenience. One person mentioned the difficulty of operating the controller, especially the movement of gripping and placing virtual tools. One participant reported that she experienced dizziness while wearing the HMD because of difficulty to focus accurately as wearing both glasses and HMD together.

6. Discussions

6.1. Experiment I on functions of virtual intervention

The first experiment demonstrated that video modelling VIC is useful for the virtual training of those with ID. The method was more effective for espresso extraction training. This may indicate that video modelling VIC is more effective for procedural learning than cognitive learning such as learning terminologies. The result is consistent with the previous research that indicated that virtual training is more effective for procedural learning (Abich et al., 2021; Hoareau et al., 2017; Renganayagalu et al., 2021).

In the additional analysis conducted for some subjects, we found that those characterized with better attention spans and cognition for virtual visual information (i.e., movements) tend to gain more benefits from video modelling information during training. While it is difficult to generalize this result due to the small sample size, we believe that the analysis is still meaningful to emphasize the need to provide VIC tailored to the individual characteristics of each person with ID. The personalized VIC is also essential, given the wide difference in cognitive capabilities and sensory characteristics among people with ID (Kim et al., 2021).

6.2. Experiment II on effects of virtual intervention

In the second experiment, we evaluated and observed the job performance success rates over 7 weeks to verify the learning and transfer effects of our VI method. In the II group, an instructor provided direct intervention while the VI group received virtual intervention without the help of any human instructor. The two groups showed almost no difference in terms of job performance abilities before virtual training. As for the II group, a significant improvement before and after virtual training was witnessed only for espresso extraction training, with transfer effects remaining unverifiable. In contrast, for the VI group, performance rates improved after virtual training for both espresso extraction and milk-steaming training, with the transfer effects verified for the espresso extraction. This comparison suggests that virtual training based on the VI method is more effective than the II method for improving job performance skills and transferring learned skills.

In the case of milk-steaming training, the VI group improved their job performance in the post-measurement step, whereas the II group did not. However, in the transfer-measurement step, the success rates of job performance in the VI group decreased slightly, indicating that there was no transfer effect. In the transfer-measurement step, we observed that some subjects gave up their task out of fear of the steam noise and heat rather than the lack of ability to perform the procedure. This is an area of improvement in the future design of virtual training content.

Through analysing the patterns of how virtual and instructor interventions were used in virtual training, we identified the unique effects of each intervention type. VI method allowed subjects to perform the task successfully on the first attempt without additional interventions compared to an II method. While an equal amount of information was provided to the two groups, there was likely a difference in trainees’ acceptance of the information due to the variances in information delivery methods and interfaces. This implies that the information provided through VIC and VII in a virtual training environment is better utilized than that offered through the II method.

The frequency of requesting intervention was higher in the II group. This result, however, does not indicate a gap in ability between the groups since II naturally provided an environment where trainees could easily request help. There was no difference between the two groups’ abilities in the pre-measurement step. While II is an easy and familiar way for people with ID to ask for help, II is more difficult than VI for people with ID to apply and utilize information in a VE. From the interviews with the subjects, we could infer that they could not listen well and understand the instructor’s information when being immersed in a VE with the HMD.

As the number of VI performed by a subject was repeated, a change in the level of task performance of the subject was evident. This suggests that repetitive learning using VIC can play an important role in enhancing task performance of people with ID. This also indicates that the guide provided by VIC has a positive effect on virtual training for the barista job.

While the overall duration of our experiments may be relatively short, we believe that our study is meaningful in that it tracked learning transfer effects beyond the training. In particular, this study makes it clear that virtual training using the VI method represents a meaningful training approach that facilitates the transfer of learned skills rather than the temporal arousal of the interest of those with ID (Bossard et al., 2008; Ganier et al., 2014; Hoareau et al., 2017).

6.3. Feedback from the field

The VI-based barista training content featured in this study was used for several months at the actual vocational training centers for people with ID (see Figure 13). Training sessions were conducted regularly once a week, with each session attended by around 10 trainees. The instructors used a remote controller to control the VIC by the trainee’s ability and training situation. In the interview with the instructors after five months of virtual training, they commented that trainees showed enhanced attention and stronger job performance skills. The instructors also said that virtual training was useful as they could easily intervene in the training using a remote controller, without having to give intervention in a loud voice or use physical assistance. The trainees responded that the virtual training approach was interesting and helped them to become familiar with difficult terminologies and the barista procedures.

Figure 13. Barista virtual training session at the national vocational training center.

7. Conclusion

In this study, we defined a new concept of VI to make virtual training more effective. We developed VIC and VII and applied them to the virtual barista training content. The effect and significance of VI technology were demonstrated through two rounds of experiments for people with ID. Notably, the proposed VIC led to the greater improvement of job performance skills as opposed to the traditional II method. It was also found that the VI method is effective for learning transfer, which means that trainees applied the skills gained in the virtual training to the actual cafe environment. Taken together, the VI method served as an interface that allows for effective communication between a trainee in the virtual world and an instructor in the real world.

In addition, our VI method was developed separately from the virtual training content to be applied to existing various training contents. The VI method can be applied to a wider range of training content, thus providing more opportunities for people with ID or beginning trainees to receive effective and customized virtual vocational training.

Based on the feedback from actual users, we have improving virtual barista training contents. We have classified training into either procedural learning-based or knowledge learning-based training to develop VIC tailored to each training category. We have also conducting in-depth experiments on the effects of different types of VICs on different types of learning such as procedural learning and knowledge-based learning.

Going forward, we plan to develop technologies that automatically offer VIC without the control of an instructor to allow for an independent and remote training environment. With reference to the research by Jo (2021) and Hong et al. (2021), which analyzed the visual attention levels of those with developmental disabilities in virtual training, we plan to improve our VII to automatically offer VIC based on a trainee’s attention span level and training situation. Our main research goal is to enable an automatic provision of suitable VIC based on the system’s assessment, so that multiple trainees can simultaneously engage in virtual training in a remote, contact-free environment.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by Korean government [22ZH1200, The research of the basic media·contents technologies].

Notes on contributors

Heesook Shin

Heesook Shin is a principal researcher in Communication & Media Research Laboratory, Electronics and Telecommunications Research Institute, South Korea. She received her M.S. degrees in computer engineering from Pohang University of Science and Technology (POSTECH) in Korea. Her research interests include human-computer interaction, accessibility, and virtual training.

Sungjin Hong

Sungjin Hong received the B.S. and M.S. degrees in computer science engineering from Inha University, South Korea. He has been a senior researcher in Communication & Media Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, South Korea. His research interests include computer vision, deep learning, and VR/AR/MR.

Hyo-Jeong So

Hyo-Jeong So is a professor in the Department of Educational Technology, Ewha Womans University in Korea. She received her Ph.D. degree from Instructional Systems Technology, Indiana University. She is particularly interested in examining how to integrate emerging technologies for teaching and learning from collaborative knowledge building perspectives.

Seong-Min Baek

Seong-Min Baek received MS in Computer Science (virtual reality) from Pohang University of Science and Technology (POSTECH), KOREA in 2001. He is working as senior member of engineering staff at Contents Research Department of ETRI with interest in Digital Contents, Animation, and Physics.

Cho-Rong Yu

Cho-Rong Yu works in intelligence knowledge content research section at the ETRI. She received a bachelor’s degree and master’s degree in computer engineering from Chungnam National University in Daejeon, Korea. She is interested in virtual environment for training and new interfaces for AR/VR/XR environment including metaverse.

Youn-Hee Gil

Youn-Hee Gil received the B.S. and M.S. degrees in computer engineering from Pusan National University, South Korea, respectively. She has been a principal researcher in Communication & Media Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, South Korea. Her research interests include NUI/NUX, VR/AR, and Accessibility.

Notes

1 The inverse efficiency (IE) value was used to analyze the combined effects shown by the reaction time and accuracy. The IE value is the reaction time divided by accuracy, and shows the eclectic reaction results in a situation that requires fast reaction time and high accuracy (Townsend & Ashby, 1983).