Ready, trainer … one*! discovering the entanglement of adaptive learning with virtual reality in industrial training: a case study

ABSTRACT

In response to the industrial world becoming more complex and the recent social distancing caused by the COVID-19 outbreak, emerging technologies such as Virtual Reality (VR) are gaining importance for industrial training. The novelty, however, does not lie in the sole use of VR-based technology but rather in its adaptive and flexible nature. This paper explores the synergy between VR environments and adaptive-based learning and tests the proposed approach in the railway sector. An iterative co-design approach is adopted to develop a VR-based training system for educating train conductors. The developed system is validated with ten conductors, where their learning preference, learning outcome, and learning performance were assessed. The results show an increase in individual motivation and perceived usefulness for the developed VR-based training system, demonstrating a higher learning preference, in comparison to the currently used physical simulator. Similarly, the results outline a knowledge gain in two out of six key knowledge subjects for conductor education and indicate that time improvements per exercise are needed to meet organizational performance goals. Consequently, this research presents an attractive alternative to costly and product-specific physical simulators and offers three strategic managerial implications opening new opportunities for using VR in developing learning training programmes.

KEYWORDS:

• Adaptive learning
• virtual reality
• railways
• motivation

1. Introduction

Industrial training requires the continuous development of new solutions to meet the ever-changing work environment and provide employees with the necessary twenty-first-century skills that the digitalization of the working environments demands. Such skills include among other critical thinking, creativity, and problem solving (van Laar et al., 2020). Researchers and practitioners face challenges on multiple fronts in proposing effective training solutions while balancing between the financial interests of firms and promoting sustainable growth of the industrial sector. New technologies such as blockchain (Abbas et al., 2020a), digital twins (Kritzinger et al., 2018), augmented reality, and virtual reality (VR) (Gavish et al., 2015; Santamaría-Bonfil et al., 2020; Shamsuzzoha et al., 2019) provide new unique opportunities to replace or complement mainstream industrial training solutions. Next to technological developments, novel educational approaches are readily available to further enhance industrial training. Recently the educational technological research community highlighted technology-enhanced adaptive learning to be a key learning paradigm (Kenny & Pahl, 2009; Wang et al., 2020; Xie et al., 2019) as it offers the opportunity to first determine user strengths and weaknesses and consequently adapt the amount of instruction the user needs to receive (Martinetti et al., 2020).

VR-based training systems are now being proposed for the maintenance training of line workers, providing an affordable, versatile, and safe alternative for training in high-risk environments (Santamaría-Bonfil et al., 2020). However, ensuring the robustness, providing quality training, and demonstrating a business case for implementing such solutions in comparison to mainstream solutions is no easy task (Wu & Lin, 2012). Among other things, mainstream training and mentorship programs have evolved with time to transfer not just consolidated explicit knowledge, but also at times inarticulable tacit knowledge. Embedding VR-based training systems in an organisational setting, where consolidated organisational knowledge exists in both, tacit and explicit forms within the organisational members (Nonaka & von Krogh, 2009), brings new challenges for the VR-based training systems. Although the role of Information and Communication Technology (ICT) in managing tacit knowledge lacks consensus in the academic community (Al-Qdah & Salim, 2013), Bokhorst et al. (2014) and Wang et al. (2011) state that by using knowledge maps (realised from collected assessments) and using patterns that contain problem-solution pairs as an established method, tacit knowledge can be accumulated, visualised and exchanged. Similarly, Mitri (2003) points out that technology databases, internet architecture, artificial intelligence, and decision support techniques facilitate tacit assessment management. Endorsing the viewpoint that ICT based solutions can be used to manage tacit knowledge, this research investigates the potential of VR-based adaptive learning for industrial training. Consequently, the following research question is investigated in depth in this study.

How can we develop virtual reality based industrial training to (re)instruct operators in an adaptive and flexible manner?

More specifically, the presented research is conducted within the Netherlands Railways (the principal railway passenger operator in the Netherlands) hereon referred to as Railway company. Conductor education remains the primary focus for the Railway company to deliver the promised performance to its customers. A comprehensive conductor education program has been developed by the Railway company of which “departing safely and punctually” is a key learning module (NS Learning Centre, 2019). Among the e-learning courses, this module also consists of mandatory practical sessions with conductors where they practice multiple departure procedures and possible irregularities through a physical door simulator shown in Figure 1.

Figure 1. Images of the current physical door simulator.

Besides being costly and product-specific the door simulator is only available at four locations in the Netherlands for over three thousand conductors. This results in the simulator being highly occupied and limited versatility in building conductor education programs. This calls for a creative solution to address the shortcomings of the current setup. Thanks to its adaptive nature and its increased use in medical, industrial, and commercial training (Vaughan et al., 2016; Zahabi & Abdul Razak, 2020), VR-based educational training can be considered as a suitable candidate for educating conductors on the departure procedures and possible irregularities. Moreover, the demonstrated cost-effectiveness of VR application in the medical field (Pot-Kolder et al., 2020) paves the way for its adaption and pilot testing in conductor education. However, developing a virtual reality experience for practicing departure procedures and possible irregularities with a similar or higher learning preference than the physical door simulator is a key challenge. Previous investigations done with the railway personnel showed that they are willing to use new technologies in their educational programs (Martinetti et al., 2018). Consequently, in this study, a VR-based conductor training system is developed as an alternative to the existing physical door simulator.

The rest of the paper is structured such that the literature review on adaptive learning, adaptive learning in virtual environments, and challenges in designing virtual learning environments is presented in section 2. Section 3 outlines the followed methodology. Section 4 provides a detailed description of the Railway company case by presenting the underlining research problem, identifying the objectives of the desired solution, and describing the corresponding design and development part of the solution. Section 5 presents the results of the pilot evaluation of the developed VR solution and section 6 discusses the presented results, the overall design approach, and the performance of the developed solution. Section 7 outlines the managerial implications of the study. Lastly, section 8 provides the conclusion of the conducted research and offers some key recommendations for future research.

2. Literature review

This section lays out a detailed account of relevant theoretical concepts for this study such as adaptive learning, adaptive learning in virtual environments, and challenges in designing virtual learning environments.

2.1. Adaptive learning

Adaptive learning, also known as adaptive teaching, is an educational method to customise resources and learning activities to address the unique needs of each learner. Adaptive learning systems aim to transform the learner from a passive receptor of information into a collaborative interactor in the educational process.

It should be emphasised that “personalised” and “adaptive” learning are not interchangeable terms (Kerr, 2016). Fully personalised learning experiences also focus on tailoring learning methods to user preferences and content towards user interests, variables that could not be changed in this research, whereas adaptive learning focuses on tailoring the content to what students need to know and takes into account their progress (Martinetti et al., 2020). However, according to the above definitions, the similarity between “personalized learning” and “adaptive learning” is that technology essentially aims to cater to the diverse learning needs of learners. In particular, the boundary between “personalized learning” and “adaptive learning” becomes vague if they are limited to the scope of “technology-enhanced learning” as they have been used as two interchangeable terms in many extant studies (Aroyo et al., 2006; Gómez et al., 2014; Göbel et al., 2010; Lin et al., 2013).

Adaptive learning methods keep learners inside a “flow channel” Göbel and Wendel (2016) defined as a perfect balance between anxiety and boredom in relation to skill and challenge. In this zone, the exercises are not too difficult to make learners anxious, but also not too simple that they become bored. Chen and Jang (2010), proposed an effective view on how this flow can be realised within game applications by underlining three conditions that need to be met: the system needs to be intrinsically motivating, offering the right amount of challenge as mentioned earlier, and providing the player the sense of being in control. According to Paramythis and Loidl-reisinger (2004), there are roughly four categories that define the boundaries of the adaptive learning environments, in which it is possible to act. Figure 2 shows the interaction of these elements and their position within the “flow channel” proposed by Göbel and Wendel (2016), namely:

• content discovery: adaptive techniques are used to present individual learners with personalised content during a course, possibly from other sources.

• interaction level: learning content and the course are not adapted, but the user interface and the interaction with it are tailored towards user preferences.

• course delivery: the way the course is set up, is tailored towards user needs.

• adaptive collaboration: learning processes requiring communication or collaboration is supported.

Figure 2. Adaptive learning boundaries embed in the “flow channel” (adapted from Göbel and Wendel (2016) and Paramythis and Loidl-reisinger (2004)).

Because every student learns in different ways, even when these are classified into broad groups (e.g. visual, spatial, logical, social, etc.) students are going to have different learning outcomes (Abbas et al., 2020b). Not everyone is going to absorb the received knowledge from a teacher or a training system in the same way. Therefore, instead of offering a single package of learning, education content creators and providers need to tailor learning packages around a variety of situations by determining the learning styles present in a class (Chang et al., 2009; Huang et al., 2012; Klašnja-Milićević et al., 2011).

Despite the mentioned benefit, it needs to be said that adaptive learning may not always connect easily with every course, discipline, and subject area. Instructors must always make a judgment call, partly factoring in the needs and styles of the students they are expecting in a new intake, along with the demands of the course and anticipated learning outcomes.

2.2. Adaptive learning in virtual environments

The use of VR systems for training complex and highly demanding tasks is gaining attention. However, Gavish et al. (2015) report that there is a need for empirical evaluation of the efficiency and effectiveness of VR systems compared to traditional methods. This research addresses this gap by developing an adaptive VR-based training system for training conductors on departure procedures and potential irregularities and conducting a pilot evaluation. The importance of the interplay between adaptive learning and learning analytics for learning-centred education is well understood with the educational technological research community (Mavroudi et al., 2018). Learning in an adaptive virtual learning environment (VLE) is to a great extent dependent on the amount of immersion that it can provide, as learners should feel physically present in the virtual environment (Dalgarno & Lee, 2010). Additionally, Rienties et al. (2016) demonstrate that perceived usefulness of the VLE was not related to (perceived) task performance and that perceived ease of use is rather related to intentions and actual behaviour in the VLE. Immersion is an important feature generated by two factors that are unique to VLEs: representational fidelity (RF) and learner interaction (LI) (Dalgarno & Lee, 2010). RF is defined as the degree to which the virtual world is realistic (ibid). It is influenced by factors such as the display of the environment, the smoothness of the object motion and changes in view, the consistency of object behaviour, the quality of the user representation, spatial audio, and force feedback. LI describes the richness of the interactions, such as embodied interactions and -object manipulation, but also the construction of new objects and the ability to modify object behaviour (ibid). It is worth mentioning that immersion is not merely a consequence of technology. There are also pedagogical requirements that facilitate the sense of immersion. Fowler (2015) distinguishes three types of pedagogical immersion correspondent to the three stages of learning which are as follows:

• Conceptual immersion: This is all about demonstrating what needs to be learned. Introducing a new concept, phenomenon, or process.

• Task immersion: The concept needs to be translated to the knowledge or skill of the participants. This is usually done by practicing things introduced in the previous stage.

• Social immersion: If a skill is then learned, it should then be put in a social context. This way, participants test their understanding of the concept or skill and its consequences.

In this research, task immersion is of particular importance considering the practical nature of the conductor’s tasks and skillset needed to operate effectively under time pressure. Fowler (2015) lays out three guidelines to ensure that trainees are immersed in the tasks they are practicing. First, he points to the need for generating deep levels of interaction. This stimulates active experimenting and thus contributes to experiential learning. Manipulating the virtual environment can also help to grasp relationships between concepts or objects that would otherwise be unclear. Secondly, he stresses that the focus should be on knowledge construction instead of reproduction (ibid). Trainees should be able to experiment, try out what they have learned before, and make mistakes so that they can experience the consequences of their actions. Finally, he states that for trainees to become pedagogically engaged with the tasks, they should be authentic (ibid). Important to state here is that authenticity in this context is not the same as realism. A task or environment can be authentic without full realism, and a very realistic environment can be completely unauthentic. The main objective is to make practice scenario as close as possible to the real-world situations in terms of events and actions.

2.3. Challenges of designing virtual learning environments

Using a virtual instead of a regular learning environment has its advantages, but also poses challenges from a technological, pedagogical, and organisational perspective. The main technological challenge is to shape an authentic and immersive virtual experience necessary for effective learning (Doumanis et al., 2019). Among other things, this means that the accuracy of the VR applications should be improved, and the delay of the visuals should be reduced. Currently, this causes some users to become nauseous or feel dizzy, comparable to motion sickness (Feng et al., 2018; Kartiko et al., 2010; Soltani & Morice, 2020). This implies that VR systems need to provide natural and intuitive interactions to minimize such experiences. Technologies that have helped address these issues are among others voice control, gesture control, and haptic feedback (González-González & Blanco-Izquierdo, 2012; Vaughan et al., 2016). Similarly, from a pedagogical point of view, one of the biggest challenge is to maintain learning performances in face of ongoing introductions of new and unfamiliar technologies in the learning systems (Barari et al., 2020).

In addition, it is a challenge to address the cognitive overload which commonly occurs when people receive too much information interfering with their short-term memory, and distracting them from their original thought process (Bolisani et al., 2018). Especially in VR environments, where multisensory information is used to increase the sense of presence. Additionally, for many companies, it can be an organisational challenge to start using and maintaining VR applications (Martinetti et al., 2018; Tyre & von Hippel, 1997). They might not have the necessary infrastructure in place to properly embrace the VR-based training. However, it is worth highlighting that despite technological, pedagogical, and organisational challenges, VR based training has gained a keen interest in both educational and industrial communities (Chien et al., 2020; Dubovi et al., 2017; Gao et al., 2019; Virvou & Katsionis, 2008) and several studies have proposed solutions to effectively tackle these challenges (Cherni et al., 2020; Mikropoulos & Natsis, 2011; Qin et al., 2010). From a VR system design standpoint, the relationship between learning content and choice of design elements plays a central role in supporting VR-based learning (Radianti et al., 2020), providing a detailed overview of suitable design elements, such as moving around and realistic surrounding to train for procedural practical knowledge. Similarly, the integration of peer assessment (Chang et al., 2020) and iterative feedback from users and experts (Wang et al., 2019) in the VR system design have also been proposed as ways to improve VR-learning systems. Consequently, this research takes guidance from the presented literature to develop an immersive and versatile VR industrial training system.

3. Methodology

The Design Science Research Methodology (DSMR) by Peffers et al. (2007), originally developed for conducting design science research in information systems, is used as an overarching approach to investigate the presented research question. The methodology was chosen mainly for its general acceptance in the educational technological research community (Park et al., 2019) and prior use in designing VR-based training systems (Metzger et al., 2017). More specifically, to better align the research question with the presented methodology, an appropriate case within the railway sector was identified for further investigation. Additionally, the problem-centred approach of the DSMR methodology was selected as the starting point of the conducted research. Figure 3 presents the followed design process in this research.

Figure 3. Followed design process adapted from Peffers et al. (2007) with numbers indicating respective sections and subsections of the paper.

As shown, multiple iterations of concept detailing (paper prototype, the first version, and the final version) were performed before conducting the final user test with the conductors. The conducted pilot evaluation aimed at identifying the usefulness of the VR training system in comparison to the currently used physical door simulator in an industrial context. In total, twenty-six people participated in the evaluation of the current door simulator by responding to an evaluation questionnaire. In comparison, ten people participated in the evaluation of the developed VR training system, where each participant was given a pre-exercise questionnaire to realise his/her prior VR knowledge and get zero measurements of the learning outcome. The pre-exercise questionnaire was aimed at understanding the group characteristics of the participants. Table 1 presents the group characteristics of the individuals who participated in the VR training system evaluation.

Table 1. Group characteristics.

Participant	1	2	3	4	5	6	7	8	9	10
Age  (years)	29	64	43	44	63	63	51	50	48	49
Experience in the Railway Industry  (years)	2	30	22	20	18	19	8	4	5	2
VR Experience  (instances of previous use)	1	1	1	0	0	1	0	0	0	0

The next section provides an insight into conducted case study within the Railway company.

4. The Railway company case study

Considering the central role human behaviour plays in the operation of railways (European Commission, 2018), developing state of the art industrial training programs is not just a matter of optimizing organisational performance but also of public safety and prosperity for the railway organisations. Coupling this with high workforce turnover resulting in potential loss of tacit knowledge (Levallet & Chan, 2019), investigating VR-based adaptive learning within the railway sector is of paramount importance for the educational technological research and knowledge management research communities. The next section describes the identified research problem in further detail and outlines the objectives of the solution.

4.1. Problem description and objectives of the solutions

As explained in the introduction, conductor education remains a key priority for the Railway company to ensure deliverance of quality service and achieve desired system performance levels. Although a physical door simulator helps in learning required skills and gaining critical knowledge, it has limitations in terms of its availability and flexibility, and it requires a lot of investment and maintenance costs. This triggered the desire to develop a state-of-the-art VR training system as an alternative to a physical simulator. It is important to highlight that a simulator’s main goal is to increase the awareness of personal competencies (Dieckmann et al., 2012). This implies that in the case of the railway company, the train conductors can benefit from getting increased awareness of their competencies (i.e. scenarios they can easily solve, and which require more practice). To realise the accurate design for the VR training system, Table 2 outlines the formulated objectives of the desired solution and their corresponding requirements. It is worth mentioning that objectives, in this case, are defined as measures that determine the success of addressing the research question and requirements to achieve those measures.

Table 2. Solution objectives and requirements.

Objective	Requirements
The virtual simulator achieves at least the same scores on learning preference and learning outcome as the current door simulator	The simulator features high levels of physical, semantic, and phenomenological realism The simulator has added value to the conductors The simulator limits anxiety The simulator is focused on obtaining the correct declarative and procedural knowledge
The virtual simulator provides an adaptive learning experience	The simulator has multiple entrance points The simulator measures skill level and progress The simulator adapts its recommended learning steps towards the skill level of the student The application is a static intrinsic game The application has a storyline
The virtual simulator provides logistical benefits over the current simulator	All costs associated with setting up the virtual door simulator in one location are less than 80.000 The virtual simulator requires maintenance or support less than four times a year Conductors can use the simulator without an instructor present
The virtual simulator allows for experimentation and collaboration to acquire new knowledge	Conductors can work together if that is required in some departure scenarios. The VR solution allows participants to observe the performance of others The solution limits the cognitive load to a minimum Interaction with VR should be natural and not in the way of learning. The virtual simulator includes multiple train types to practice on
The virtual simulator provides an adequate amount of technological immersion	Users are adequately represented in the VLE The simulator features spatial audio The visuals are physically and semantically realistic The objects in the VLE behave consistently Interactions in the VLE are embodied
The virtual simulator provides a high amount of task immersion and an adequate amount of social immersion	The tasks should be simulated as authentic as possible The simulation features deep levels of manipulation The VR-simulator incorporates the bigger picture of the departure processes: Travellers Platforms Time pressure Safety hazards

The next section provides a detailed insight into the design and development of the solution.

4.2. Design and development of the solution

The section is divided into three key subsections. First, an insight into the conceptual exploration of the hardware and software concepts is presented. This is followed by concept finalisation in a two-step approach where first conductor workshops were organised to identify preferred hardware and software concepts. Afterwards, a multicriteria analysis for the hardware concepts was conducted together with the learning department of the Railway company to realise the organisational readiness for the identified concepts. Last, a detailed account of the design and development of the chosen concept is presented. Figure 4 presents a holistic overview of the design and development process.

Figure 4. Holistic overview of the design and development process.

4.2.1. Conceptual exploration

Designing and developing a VR training system for conductor education requires conceptual exploration of suitable choices both on the hardware and software front. Exploration of software concepts is needed to identify and describe system functionalities and user interaction with the system, while hardware concepts focus on identifying and describing the suitable equipment to deliver that functionality. The authors identified three suitable hardware concepts namely standalone, mobile, and Kinect. Specifications for each of the identified hardware concepts are mentioned in Appendix A. Similarly, to explore the functionality of the desired program, four use case scenarios were developed: basic practice, multiplayer, adaptive learning, and serious gaming. These scenarios were designed to complement each other and had their foundations in the overall conductor training program of the Railway company. Additionally, a small user-story for each scenario was outlined to clarify their functionality. Similarly, activity diagrams and functional block diagrams for each scenario were developed to display the interactions of a user with the system and the background actions of the system, respectively. The designed activity diagrams and functional block diagrams for each of these concepts are displayed in Appendix B. A co-design approach was followed to design and develop the VR training system. The preliminary investigation showed that the conductors expected the VR-based learning to benefit them in terms of practicing more realistic scenarios, not having to go to the training points, and having more flexible learning content as shown in Appendix C.

4.2.2. Concept finalization with conductors

To understand the conductors’ preference for presented hardware and software concepts two workshop sessions of ninety minutes each were organised with three and two conductors, respectively. The discussion held during these sessions also assisted in co-designing the desired functionalities for the chosen concepts. Both sessions began by introducing the conductors to the research project and an explanation of both hardware and software concepts. Afterwards, the goal of workshop sessions was explained to the participants and they were presented with opportunities, unique selling points, ways of interaction, and specifications of each hardware concept on an A3 poster.

In these sessions, the participants were given five minutes to familiarise themselves with every concept. Afterward, they were encouraged to try out different hardware concepts by playing a small videogame, whose tasks were like ones of the physical door simulator. Finally, the participants were asked to document the strong and weak points on every concept within the context of its use for conductor education and state their preferred hardware concept. Table 3 presents the results of the two conductor workshop sessions for the hardware concept.

Table 3. Results of the conductor workshop on the hardware concepts.

Concept	Strong points	Weak points	Preference
Standalone	Use both hands Most possibilities in terms of interactions	Complicated interactions Hygiene	2
Mobile	Simple controller	Just one controller Unpractical during conductor training, switching phones, etc. Hygiene	1
Kinect	Natural interaction without controllers Fun experience	Always facing screen No real environment The sensor does not easily pick-up a new player after switching	2

The software concepts were evaluated in a similar manner where first the four software concepts were introduced to the participants by describing the use case scenarios. They were then given five minutes to study the four concepts and encouraged to identify the functions that they would like to be embedded to further improve and contextualise each of the four software concepts. Lastly, after identifying the potential improvement functionalities the participants were asked to indicate their preferred software concept. Table 4 presents the results of the two conductor workshop sessions for the software concept.

Table 4. Results of the conductor workshop on the software concepts.

Concept	Functionality improvements	Comments	Preference
Basic	Record practice so that replays can be watched Including the smartwatch	Looks like a well-founded basis Think about the used terminology. It should match the slangs conductors use on the job	5
Multiplayer	Include communication via a transceiver	Seems essential because it is also possible in the current simulator	5
Adaptive learning	Do not use a star system	Conductors do not like to be assessed during practice	2
Serious gaming	Also, practice other parts of the job	I would like to use it at home to show my friends and family	2

To investigate the organisational readiness and compatibility for the identified hardware concepts, a multi-criteria analysis approach was developed based on solution objectives together with the learning department of the Railway company. Each hardware concept was evaluated based on eight criteria points as described in Table 5 and a scoring schema was developed to evaluate each of these criteria points as shown in Appendix D. The outcome of the multi-criteria analysis performed together with the experts from the learning department of the Railway company, is presented in Table 5.

Table 5. Results of the multi-criteria analysis.

Criteria	Sub criteria	Definition	Weighting factor	Concept standalone	Concept mobile	Concept Kinect	Current simulator
Costs	Development costs, Hardware costs, maintenance costs	Costs of hardware only, not development time	4	8	9	7	1
Learning curve/complexity	null	How difficult is it learning to use the hardware in such a way that you can practice properly	3	3	6	5	10
Use setting flexibility	Learning content, Location,	In how many contexts or locations can the hardware be used	5	7	9	9	2
Digital quality of the simulation	Visual, Audio, Feeling	How well does the simulation physically represent reality	4	7	4	3	4
True to nature virtual object manipulation	Null	Is interacting with virtual objects the same as you would do with those objects in the real world?	5	7	2	9	10
True to nature navigation	Null	Do you navigate through the virtual world the same way as you would in the real world	4	10	7	4	10
Comfort of experience	Motion sickness, physical comfort, eye tiring	How long can you pleasantly use the hardware for practice	3	3	7	8	10
Maintenance & support	Repair, Replace, Preventive maintenance	How well is the hardware supported by the manufacturer and how difficult is it to maintain in case of malfunctions	4	7	7	1	7
TOTAL				216	202	189	208

As Table 3 shows, the conductor workshop sessions resulted in a tie between the concept “Standalone” and the concept “Kinect”. The multi-criteria analysis assisted in addressing this point (Table 5) showing a clear preference for the “Standalone” concept. Therefore, it was decided to choose the “Standalone” concept for further development. Similarly, as Table 4 shows, the conductor workshop revealed that some conductors were concerned about being assessed in the context of the adaptive learning software concept. Therefore, it was decided to address the conductors’ concern by following an iterative co-design approach for prototype concept detailing. Additionally, discussion with the learning department showed that they were quite supportive of the adaptive learning concept as it matched their learning vision. The next section presents the further development and integration of the chosen concepts into an overall design.

4.2.3. Further detailed concept

This section describes the development process of the final VR training system which was realised in three iterations. The first iteration focused on getting the right functionalities and features for the desired solution. Feedback received during the concept finalization phase was used to revise the activity and functional block diagrams as shown in Figures 5 and 6, respectively.

Figure 5. Activity block diagram of the application.

Figure 6. Functional block diagram of the application.

For instance, the distinction between different train types before the base level test for adaptive learning and replay function was incorporated in the prototype concept as shown in Figure 5. A quick practice mode was also incorporated which included the possibility of adjusting certain parameters such as changing weather conditions, crowdedness of train and platform, and choosing between day or night-time in the VR concept. These additions helped in incorporating more real-life circumstances and making the training more realistic which in the case of the physical door simulator were difficult to achieve. Besides this, to clarify the connection between the activity and functional block diagram the same colour schemes for related blocks were used. Activities such as watching replay coloured orange part in Figure 5 was also coloured orange for its corresponding function of showing replay in Figure 6.

To ensure that all the desired functionalities will be built into the VR training system, a paper prototype that represented a sketched-out version of the intended user interface was created as shown in Figure 7.

Figure 7. Paper prototype.

Paper prototype assisted in investigating whether the intended flow, task sequence, content of menus, and overall layout were in accordance with the conductor’s expectation. Additionally, it proved to be an effective way of gaining feedback from the conductors and addressing problems early in the design process without addressing technological variables such as bugs in the program that can cause problems but have nothing to do with the interface itself (Snyder, 2004). A more detailed insight into received feedback on iteration 1 is provided in Appendix E.

The second iteration focused on developing the first version of the VR training system such that the menu structure and three practice scenarios were realised in the VR-environment. The version was aimed at enabling conductors to gain experience with the developed VR-environment and get valuable feedback for the complete version of the prototype. The main functions were further divided into subfunctions as shown in Table 6 to realise the stated iteration aim.

Table 6. Detailed functionality of the VR training system.

Function	Subfunctions
Loading scenario and user avatar	Representing the player’s body Showing a complete train and train station environment and travellers Allowing walking around of the player
Allowing manipulation of all required parts	Allowing turning of the key Allowing pressing of door buttons Allowing converting of switches Allowing closing door by hand Allowing removing the cover of the door mechanism
Keeping track of mistakes	Listing tasks to be done during each scenario Listing tasks that are done during the execution of the scenario
Providing feedback	Comparing the two task lists Listing the missing tasks Listing tasks that were mixed up Explaining why missing tasks are important and how they need to be done
Showing main menu, train choice menu, or scenario choice menu	Showing clickable buttons Showing controllers and pointers to click Linking buttons to correct menu

The first version of the VR training system was developed using the Unity software. Generally used as a videogame engine, this open-source software also supports VR platforms such as the Oculus Quest, which was used in this research. Starting from the main menu, a conductor could start-up training mode and pick his/her train type. The conductor could then choose one of the three scenarios that had been built in the program:

• A regular scenario

• A scenario where he/she had to close doors from the outside because it was too crowded

• A scenario with a door malfunction

Appendix F provides an overview of how these scenarios should be handled according to the Railway company procedures.

Once a scenario was chosen, the conductor will wake up in a train which slowly comes to a standstill at a train station, after which the conductor was asked to execute the departing procedure. The conductor could use the controllers in his/her hands for all the interactions and walk around to walk in the virtual environment. The conductor could also see the movement of his/her own hands via the controllers. In this way, most of the tasks were a fair representation of the real world: walking, looking around, and opening and closing doors. For instance, to press the buttons, the conductor needed to extend his/her index finger, and to turn the key the conductor needed to grab the key by holding a button on the side of the controller and then turn the controller simultaneously. Moreover, instructional texts to guide through different tasks, and the possibility of asking for hints were also incorporated in the program to facilitate the conductors in their task completion. Similarly, audio recordings for giving departure signals and opening and closing of the doors were also embedded along with the overall platform layout to enhance the conductors feeling of being on a train station and address the phenomenological realism, something that was missing in the current door simulator. Lastly, the solution also provided feedback to the conductors at the end of each scenario by displaying the task that was missed or not executed at the right time.

The first version of the VR training system was evaluated by eight conductors. Key comments and observations for each scenario are presented in Appendix G. The participants appreciated the enhanced realism both in terms of how the solution looked and in terms of the overall flow of the exercise. In contrast, the participants had concerns regarding the controls of the program. For instance, turning the key and navigation using the two joysticks was difficult for some participants. Interestingly, the participants stated that communication was a vital part of executing a procedure correctly. A few complaints of nausea and dizziness were also noted during practice sessions. Nonetheless, many participants stated that they think it is fun and can be a good way of learning. Additionally, the presented comments and observations in Appendix G led to the following changes for the next version of the prototype. Turning the key, especially above the door while solving the door malfunction was made easier. Moreover, participants were provided with the possibility of using a transceiver to send and receive safety-related communication with the train driver. Finally, a small tutorial was added before their first practice scenario to make practice and test sessions more effective. All these changes were meant to provide a preview for the adaptive learning environment in the third iteration. Additionally, a base level test was developed and incorporated in the VR training system to provide insight into the adaptive learning environment. Figure 8 presents the design logic of the adaptive base level test which was developed with the consultation of an expert conductor.

Figure 8. Design logic of the adaptive base level test.

The base-level test was divided into the general and specific categories as not all the consolidated conductor knowledge is rolling stock specific. Similarly, based on gained scores the conductors could continue or had to redo certain parts of the test or the entire test. Moreover, based on the number of correct responses in the specific test, the conductors could end up in one of three scenarios as shown in Figure 8. To incorporate adaptive learning features, level three was locked for the conductors until the point that they answered all the questions correctly. In addition, considering that the conductors had no prior knowledge of scenario three, the scenario started with a short introduction. The flow chart in Figure 8 outlines the described process that ensured that every conductor practiced at the same level. This is incorporated to address the concerns of being assessed during practice, by the conductors. Considering this, some conductors had more scenarios in levels two and three than others (depending on questions they answered correctly beforehand) which was still seen as more desirable than everyone practicing on different levels. Lastly, the learning content was tailored to the individual conductor’s needs as much as possible. The program provided instructions at the start of a scenario placed in level three so that the conductors understood which type of skills they needed to learn to complete the scenario.

The second key enhancement of the final prototype was made by adding a tutorial at the start of the program. This guides them using the buttons on the controllers in ten simple steps. Similarly, some features were added to make scenarios more realistic and closer to the real-life situation. These included features such as people running to catch their train, a station clock, and the sounds of people chatting on a busy train. Figure 9 shows some screenshots of the designed virtual environment in the final VR training system such as using the transceiver and pulling the emergency brake.

Figure 9. Screenshots of the final version of the prototype.

5. Results of the pilot evaluation

The pilot evaluation of the VR training system was done with ten conductors following a structured approach as shown in Figure 10. After identifying the group characteristics through the pre-exercise questionnaire, a short introduction to the research project and the developed VR training system was provided to the participants of the evaluation sessions. The participants then performed exercises with the prototype (base level test, tutorials, in-depth exercises on different scenarios) where time for each exercise and the number of tips given were recorded. After completion of the exercises, the participants were given a post-exercise questionnaire to determine the effect measurement of the learning outcome and their learning preference for the VR training system, like what they did for the current door simulator. Lastly, qualitative feedback was gathered in an open discussion with the participants of the evaluation before finishing the evaluation session. In total each evaluation session took on average ninety minutes. This section outlines the most important results of the pilot evaluation.

Figure 10. Schedule for evaluation sessions.

First, to measure learning performance, two parameters were recorded while the participants performed the exercises. These were the average time each participant took to finish each exercise and the number of tips they required to solve each exercise. Table 7 presents the averages times and the number of tips per exercise.

Table 7. The average time needed, and the average number of tips asked per exercise by the 10 participants, with one standard deviation.

Exercise	Average Time (mm: ss)	Average Number of Tips
1	03:08 ± 00:50	1.3 ± 0.8
2	06:52 ± 1:35	2.1 ± 0.7
3	13:01 ± 2:50	3.8 ± 1.2

Second, to investigate the learning outcome of the developed VR training system, an evaluation like the one done by the Railway company for the physical door simulator was conducted. Figure 11 provides an overview of the learning outcome, by highlighting the education impact on the key knowledge subjects of conductor education, from the physical door simulator and the VR training system.

Figure 11. Evaluation of the educational impact of both solutions.

As Figure 11 shows, the VR training system resulted in knowledge gain of the conductors in two out of six knowledge subjects. These included a knowledge gain, of 1.1% each, in learning the effect of not acting according to agreements during irregularities in the departure procedure and in knowing how to act during irregularities in the departure procedure, respectively.

Finally, to measure the degree to which the developed VR solution succeeded in presenting itself as a viable alternative to the physical door simulator, the overall learning preference of the conductors among both solutions was measured and compared. This included measuring the conductors’ motivation based on five parameters of individual motivation by Owen (2017) namely added value, anxiety, phenomenological realism, semantic realism, and physical realism. Additionally, the perceived usefulness of both solutions was also measured to understand the degree to which conductors believed that using the VR training system would enhance their job performance in comparison to the physical door simulator (Davis, 1989). Figure 12 presents the results of these parameters for both cases.

Figure 12. Overall learning preference of the conductors with both solutions.

6. Discussion

The conducted research stresses the value of VR-based industrial training by outlining several insightful findings based on a case study. Considering that the main research question was aimed at developing an adaptive and flexible VR-based industrial training for operators, the research offered a functional, substantive, and testable VR training system for conductor training. The co-design approach used during the design and development stage helped in achieving the stated attributes. Additionally, the performed evaluation provided key insights into the strengths and limitations of the developed solution. These strengths and limitations revolve around three main measures, namely the learning preference, learning outcome, and the learning performance of the VR training system.

First, as far as the learning preference is concerned, the higher scores of the phenomenological realism for the VR training system indicate it to be the preferred learning mode in comparison to the physical door simulator. The evaluation showed that the VR training system was able to incite a feeling of realism for the conductors even outperforming the one incited by their current physical setup. During the evaluation discussions and co-design process, the conductors stated that the sound effects, travellers, and the train station helped a lot in this case. More importantly, it is worth pointing out that the developed VR training system outperformed the physical door simulator not just in phenomenological realism but also on the other three of the remaining four parameters of individual motivation (Owen, 2017). These include an increase in semantic realism, added value, and physical realism. It shows that even though the conductors were in a virtual environment and not performing the tasks with actual physical components such as a key, the VR training system managed to provide a genuine experience. An experience where the enabling environment was perceived as phenomenologically, semantically, and physically realistic and value-adding to individual motivation. However, a slight increase in anxiety for the VR training system was identified in comparison to the physical door simulator. This is understandable as the performance of the participants was being investigated, and most of them had little or no prior VR experience (Table 1). Besides this, the VR training system was also perceived as more useful in comparison to the physical door simulator. This can be partly explained by the increased versatility, easy accessibility, and learning experience provided by the VR training system. Consequently, the results suggest that the VR training system managed to get the conductors motivated and generate a belief of perceived usefulness, which is the main prerequisite for embracing shared knowledge and building on one’s tacit skillset.

Second, in terms of the learning outcome, the research could only provide a preliminary insight into the educational impact of the VR training system. Evaluation done by the railway company on potential knowledge gain or loss because of practicing through physical door simulator showed a 12.5% increase in learning how to close off doors on different trains by hand and a 4.5% increase in learning how to act during irregularities in the departure procedure (Figure 11). In comparison, although a knowledge gain in two of the six knowledge subjects with VR training system was identified, the varying number of participants who took part in the evaluations and the different scope of each evaluation meant that the results could not be directly compared. More specifically, where VR training system evaluation for learning outcome focused on practice with three exercises only, the physical door simulator evaluation on learning outcome encompassed the entire training program. Nonetheless, the results of the VR training system for learning outcome did show that the solution triggered a self-reflective experience and simulated authentic tasks that allowed the conductors to become aware of their skills and knowledge. This aspect was also evident from the reported knowledge loss, or more accurately in this case knowledge adjustment, for the four key knowledge subjects for conductor education. This showed the added value of the developed base level test as a way of implementing an adaptive learning system.

Third, in terms of learning performance, the average times per exercise and number of tips required for each exercise showed that further improvements are needed both on design and development and on building technological familiarity. Particularly, to minimize lost time due to lack of technological familiarity, further optimization of controller use is suggested to reduce the time per exercise. Similarly, considering that most participants had little VR-experience, the effect of gradually increased VR-experience must also be investigated as it may potentially reduce the time needed per exercise. These improvements are needed to make the VR training system also professionally competitive from a learning performance point of view. Especially, to achieve the lower time per exercise, which in the current evaluation of the VR training system was slightly higher than the prescribed time by the Railway company.

7. Managerial implications

The presented research offers specific managerial implications for transportation organisations in general and railway organisations in particular. Such organisations need to continuously train their employees to account for the changing requirements in the operational world. To enable learning and efficient operations, managers need to exploit the potential of new technologies, such as VR, when training their employees. From a managerial standpoint, this research is particularly useful for railway organisations as it provides insights into developing affordable, adaptive, and flexible VR-based training programmes for training employees for operational procedures, system changes, and system upgrades.

The results of this study provide evidence that VR-based training systems, when designed following employee needs, can be the preferred learning mode compared to current physical solutions that are traditionally product-specific and more expensive. The results also show that such training systems can make employees more aware of their skills and knowledge on a particular subject through the simulation of authentic tasks and self-reflection. These results have three-fold implications for the transport managers in general and railway managers in particular. First, given the identified learning preference, they need to invest in creating an open culture and provide leadership so that employees feel encouraged to participate in the development of VR-based training systems that can achieve desired organisational performance goals. This entails that the managers need to be convinced of the potential performance gains from investing in the Information Technology (IT) maturity of their organisations. Second, given the preliminary insights into learning outcomes, the managers need to consolidate policies for developing and testing VR-based training systems in different organisational settings to improve their reliability, and identify suitable application areas within their organisations. Finally, to obtain long-term positive outcomes from VR applications the managers need to create a dedicated unit of their organisations for handling digital technologies and keeping important organisational know-how in house. Outsourcing these services can damage the organisational capability to manage different scenarios such as training employees for changing requirements.

8. Conclusion and future work

The presented research provides insights into developing a VR-based industrial training program to educate train conductors. As stated in our research question, a VR-based training system was developed with a co-design approach to investigate whether it can provide a viable and more flexible alternative to the currently used physical door simulator for conductor training. Additionally, an adaptive learning system was incorporated into the VR training system to understand its potential impact on learning. The combination of these elements is the main novelty of our approach. A pilot evaluation of the developed system was performed with ten active train conductors. The results indicated an inclined learning preference towards the VR training system in comparison to the physical door simulator. More specifically, the conductors ranked the VR training system better on five out of the six parameters identified to measure the overall learning preference of the conductors with both solutions. Similarly, although the evaluation of the learning outcomes could not provide a direct comparison between both solutions, it did highlight the added value of the adaptive learning system by showing a knowledge gain in two out of six knowledge subjects for conductor education. Lastly, the carried-out evaluation on the learning performance assisted in identifying the improvements needed in the design and development to lower the time needed per exercise.

The authors acknowledge that the results presented, although insightful in the context of a pilot evaluation, have limited statistical significance. Future work can include a larger sample size and a thorough statistical analysis of the presented parameters to improve the generalizability of results. Moreover, conducting longitudinal and cross-sectional studies can further explain and improve the educational impact of the developed VR-based adaptive learning system on industrial training.

The gained insights on learning preference, learning outcome, and learning performance demonstrate the added value of this research for the educational technological research community. It reiterates the potential of VR-based training in an industrial setting and sheds light on technologically novel and educationally impactful ways of developing such customized industrial solutions. Moreover, considering the significance of individual motivation and perceived usefulness for building tacit skillsets, our research underlines the importance of co-design in the development of industrial training solutions and outlines three explicit managerial implications. To conclude, the presented VR-based industrial training program can be a suitable candidate for operator training, able to motivate the trainees and provide an authentic learning experience.

Acknowledgments

This research is co-financed with a PPP surcharge for Research and Innovation from the Dutch Ministry of Economic Affairs and Climate. Moreover, the paper was made achievable with the precious suggestions by colleagues from the learning department of the Netherlands Railways and the University of Twente. Lastly, the authors acknowledge the peer-review feedback leading to the improvement of this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Ministerie van Economische Zaken: [Grant Number This research is co-financed with a PPP surcharge].

Appendices

Appendix A: Specifications of the hardware concepts

Standalone

Table

Price:	€449 (64GB) €549 (128GB)
Field of View (Horizontal)	90°
Resolution	1440 x 1600 per eye
Weight	570g
Battery time	2-3 hours
Charging time	2 hours
Speakers	Built-in
# controllers	2

Mobile

Table

Price	€120
Field of View (horizontal)	100°
Resolution	Dependent on smartphone
Weight	260 grams + weight of smartphone
Smartphone size	4,7” - 6”
Smartphone requirements	60 Hz refresh rate, Full HD display, Dual core processor
Speakers	Smartphone
# Controllers	1

Kinect

Table

Price	€1500-2500 (Kinect + PC)
Resolution	Preferably 4K
Display size	To be decided
Sensor range	50cm – 450cm
FOV horizontal	70°
FOV vertical	60°
Max number of people	6
Speech recognition	Yes

Appendix B: activity and functional block diagrams of the software concepts

Activity block diagram basic functionality

Functional block diagram basic functionality

Activity block diagram multiplayer functionality

Functional block diagram multiplayer functionality

Activity block diagram adaptive learning functionality

Functional block diagram adaptive learning functionality

Activity block diagram serious gaming functionality

Functional block diagram serious gaming functionality

Appendix C: expected benefits of VR

Table

Expected benefits of VR	Number of responses (respondents could choose multiple benefits) N=10
More realistic scenarios	9
Not having to go to a training point	5
More or flexible learning content	5
Practice without instructor	4
Practice quickly when you want	1

Appendix D: scoring schema for each evaluation criteria point

Appendix E: received feedback on iteration 1: paper prototype

Appendix F: the departing procedures

This appendix contains the steps a conductor should do during the three departing procedures featured in the VR training system of the physical door simulator.

Regular departing procedure

1. Step outside the train

2. Look from the platform if the doors can be closed safely

 a. Has everyone gotten in and out?

 b. Is the departing signal on?

3. Give the attention signal

4. Use the key to close all other doors but your own

5. Step outside the train again to check if everything is still safe

 a. All doors have closed

 b. The departing signal is still on

6. Step inside and close your own door

Close door from outside

1. Step outside the train

2. Inform the train driver that you are closing your own door from the outside

3. Look from the platform if the doors can be closed safely

 a. Has everyone gotten in and out?

 b. Is the departing signal on?

4. Use the key to close all other doors but your own

5. Step outside the train again to check if everything is still safe

 a. All doors have closed

 b. The departing signal is still on

6. Close your own door and step outside again while the door closes

7. Check if the door has closed properly

8. Get in the train via the closest unmanned train cabin

9. Inform the train driver that the train is ready to depart

Own door malfunctions (VIRM)

1. Perform the regular departing procedure as described earlier

2. Upon discovering the door malfunction, pull the emergency brake

3. Contact the train driver and make agreements

4. Reset the emergency brake

5. Open the cover above the door

6. Use your key to release the air pressure

7. Pull the doors inwards as far as you can

8. Use your key to rebuild air pressure

9. Push the “close door button,” the door will now be fully closed

10. Release the air pressure again

11. Ensure that the door is locked by pulling the door and pushing the “open doors button”

12. Close the cover above the door

13. Place defect stickers

14. Notify the MBN about the door malfunction

15. Notify the travellers about the delay

16. Contact the train driver to confirm that the disturbance is solved

17. Restart the departing procedure on a different door

Appendix G: comments made and observations from the first evaluation

Table

Scenario	Comments	Observations
Regular departure	It does feel a little bit like a game The environment is highly realistic Some people prefer sitting down because of fear of instability/nausea	Turning the key can be troublesome They are primarily struggling with the controls/remembering which buttons to use
Lock from outside	They would like to practice communication using a transceiver too They have to walk too far to the train cabin. In reality, they would choose a different door to do this procedure on	Using controllers is going significantly smoother. After having practiced the first scenario, this one poses little trouble Many still ask what steps to do, even though they know the procedure by heart
Door malfunction	Time pressure could be something to make this scenario more realistic The scenario represents the difficulties conductors face in this situation quite well	Goes well except for using the key at heights There seem to be different opinions on how to solve a door malfunction: the goal of this scenario is not clear for some