Preparing for skill transfer: a decision tree tool for curriculum design and assessment of virtual offshore emergency egress training

ABSTRACT

To prepare personnel for offshore emergencies, safety training should focus on transferability. Virtual environment (VE) training is designed to support the transfer of acquired egress skills to novel offshore emergencies. Decision trees (DT) are useful tools to evaluate training transfer. DTs use performance data collected during VE training to model participants’ behavioural patterns. A DT that reflects ideal behaviour is used as a benchmark to compare the compliance of the participants’ data-informed behavioural patterns. Employing the diagnostic and predictive capabilities of DTs can indicate whether a person is capable of responding to a wide variety of emergencies using inference. This paper demonstrates the use of DTs as a curriculum design and assessment tool to determine if the VE training adequately prepared participants to transfer their egress skills to new emergencies in the same virtual setting.

KEYWORDS:

• Offshore emergency egress
• virtual training
• decision trees
• training transfer

1. Introduction

Mandatory offshore safety training has predominantly been participatory-based and the resulting certification often represents nominal competence. To better prepare the offshore workforce for emergencies, operators and regulators need to pursue evidence-based safety training to guarantee workers achieve demonstrable competence. This gap in offshore training can be addressed by combining simulation technology with a well-designed, pedagogically informed and data-driven training program. Training simulators, such as virtual environments (VE), allow workers to rehearse safety protocols in various emergency conditions and provide formative feedback to help personnel learn and improve their performance. Performance data acquired during the training can provide valuable insight into the quality of learning that takes place, and the efficacy of the training program as a whole. Training simulators are widely used for maritime education; however, few studies have considered the pedagogical aspects of simulation-based training (Sellberg et al., 2017). This paper draws on pedagogical frameworks and data mining methodology to provide empirical and modelling evidence to inform the offshore and maritime industries on how to deliver training and how to assess trainee performance using VE technology.

1.1. Pedagogical perspective to virtual offshore training

To develop VE training for offshore egress, operators must determine how comprehensive focused virtual offshore egress training should be to prepare personnel for the multitude of emergencies that could arise. Virtual practice exercises should match the conditions the workers are expected to experience in real situations. However, it is impracticable to rehearse for all possible situations. As a guiding pedagogical principle, VE technology should be designed to support the transferability of training; specifically, the application, generalization, and maintenance of knowledge and skills learned in a training context to new situations (Blume et al., 2010). Therefore, VE training and supporting tools should assess when the trainee has achieved competence and is sufficiently equipped to apply their skills to new situations within the context of the training.

Many factors influence the transfer of training, Grossman and Salas (2011) outlined three factors that are most relevant to training organisations:

1. Trainee characteristics, such as the learner’s cognitive ability, self-efficacy, motivation, and their perceived utility of training;

2. Training design, such as behavioural modelling, error management, and realism of training environment; and

3. The work environment, such as transfer climate, as well as the opportunity and support from management to allow workers to apply their training.

1.2. Data mining to inform training design

Focusing on the factors that affect the training design, behavioural modelling can be used to inform the design and assessment of VE training. Decision trees (DT) are useful behavioural modelling tools to evaluate the efficacy of training transfer from both the individual and systemic perspectives. At an individual level, DTs can be used to inform how much training is needed to achieve competence for each learner. Musharraf et al. (2020) demonstrated the utility of DTs for identifying trainees’ strategies and recommended ways to use DTs to assess individual learning, as well as pedagogical approaches. The aim of Musharraf’s work was to understand how people made decisions in emergency egress in order to improve training and to create artificially intelligent agents with similar behaviours (Musharraf et al., 2020). DTs provide a visual representation of participants’ decision-making strategies in the context of choosing the safest egress routes in emergencies.

Decision tree modelling was selected for its visual simplicity and diagnostic capabilities when dealing with sparse data and categorical variables (Liu, 2009; Musharraf et al., 2020). DTs employ supervised learning, which requires a repository of attributes of solved problems to draw inferences. From the individual learner perspective, the supervised learning mechanism of DTs help inform learning analytics by developing generalized decision rules from each participant’s behavioural data collected during the scenarios in the VE training. At a systemic training level, the collection of DTs developed from the participants’ data throughout the VE training help to identify emerging patterns of successful or unsuccessful behaviours. Musharraf et al. (2020) suggested that an effective training curriculum would result in the convergence of participant’s DTs to strategies that lead to success and that systemic exceptions indicate gaps in the training approach itself.

DTs were used for their diagnostic and predictive capabilities to determine if a virtual offshore training program has adequately prepared learners to transfer their egress skills to new emergencies in the same virtual setting. In particular, the research aims to answer the following questions:

1. Can DTs help determine when participants are sufficiently prepared for training transfer?

2. Can DTs show the development of credible heuristics throughout the VE training?

3. Can DTs predict participants’ performance in new situations?

2. Background

To demonstrate the use of DTs as data-informed curriculum design and assessment tools, a longitudinal experiment was conducted using a VE as a human behaviour laboratory. The three-phased experiment collected performance data during skill acquisition (phase 1), retention and retraining (phase 2), and transfer of training to new emergencies (phase 3). The context of the study was to teach naïve participants the necessary egress skills to evacuate a virtual oil platform during an emergency. The experiment first taught basic egress skills, then assessed skill retention after a 6- to 9-month period, and subsequently tested the transfer of egress skills to new emergencies.

The VE collected participants’ behavioural data in training and testing scenarios throughout each phase of the experiment. Portions of the participants’ performance data were used to iteratively create DTs for each phase of the experiment. Multiple DTs were developed for each participant to determine if the participant’s strategies changed across the phases. This process was used to see how people develop and modify their route strategies or heuristics throughout training.

An ideal decision strategy for responding to offshore egress emergencies was defined based on the results from Smith et al. (2017). The ideal DT depicted in Figure 1 was created to guide the development of the curriculum and used as a benchmark to assess the correctness of the participants’ DT behavioural patterns. As shown in Figure 1, the expected strategy was for the participant was to listen to the public address (PA) announcement for the safest route and choose a route accordingly. If the PA indicated that the primary route was the safest option, then the participant would follow the primary route. Alternatively, if the PA indicated that the secondary route was the safest option, then the participant would follow the secondary route. When no route direction was provided in the PA, the participant was expected to follow their preferred route. The training curriculum was designed to encourage participants to develop the intended DT strategy.

Figure 1. Ideal decision tree for emergency egress

The ideal decision strategy was used as a benchmark to compare the participants’ behavioural patterns identified by their DTs to check their competence. Figure 2 summarizes the DT comparison process and the expected outcomes from the comparison. This comparison determined whether the training the participants received was sufficient to prepare them for emergencies. The compilation of DTs developed throughout the VE training program was used to identify emerging group patterns and to evaluate the efficacy of the training curriculum. The final DTs at the end of the VE training were used to predict the transfer of skills to new emergency scenarios (e.g. how people will perform in new situations).

Figure 2. Process used to compare participants’ route selection strategies with ideal DT

Section 3 describes the design of the longitudinal study; Section 4 describes the DT development and the DT curriculum assessment methodology; Section 5 presents and discusses the results, while Section 6 summarises and concludes the findings.

3. Design of the longitudinal experiment

The longitudinal experiment investigated the skill acquisition, retention and transfer of egress skills taught using a virtual offshore platform. Figure 3 depicts the three phases of the experiment.

Figure 3. Longitudinal experiment timeline

The first phase of the experiment used a simulation-based mastery learning (SBML) pedagogical approach to teach naïve participants the necessary skills to evacuate safely (McGaghie et al., 2014; Smith & Veitch, 2018). The practice and test scenarios are denoted in Figure 3 as P1-8 and T1-4, respectively. The scenarios involved participants practicing their egress routes in varying conditions from muster drills (T1 and T2) to emergency evacuation (T3 and T4). Full details of the skill acquisition phase, including the training curriculum and performance metrics are described in Smith and Veitch (2018).

In the second phase, participants returned after a 6 to 9-month period and were tested on their ability to respond to the same egress scenarios (T1-4). Participants who had trouble remembering the egress procedures were provided retraining on deficient skills (e.g. practice scenarios P1-8). Participants were required to complete retraining for any deficiencies before moving on to the third phase. Full details of the retention phase are described in Smith et al. (2019).

In the third phase, participants were tested on emergencies that differed from the training exercises to which they were exposed in the acquisition and retention phases. To inform the transferability of the participants’ training to new emergencies, the transfer scenarios were designed to vary attributes to create an assortment of emergencies. Three attributes were varied: (1) the amount of information (or lack thereof) provided over the PA announcements, (2) the participants’ proximity to hazards during the emergency, and (3) the participants’ familiarity to the starting locations on the platform (Blundon, 2019). Eight scenarios were developed to incorporate a combination of high- and low-level attributes (shown in Table 1). As depicted in Figure 3, participants performed the transfer scenarios in a random order.

Table 1. Design of transfer scenarios (after Musharraf et al., 2019)

#	Set	Quality of Communication		Proximity to Hazards		Situation Familiarity
U1	1	+	Clear PA with all relevant info	+	Hazard not accessible	+	Starts in cabin
U2		-	Poor PA that lacked info	+	Hazard not accessible	+	Starts in cabin
U3		+	Clear PA with all relevant info	-	Hazard blocking route	+	Starts in cabin
U4		-	Poor PA that lacked info	-	Hazard blocking route	+	Starts in cabin
U5	2	+	Clear PA with all relevant info	+	Hazard not accessible	-	Starts in Bridge
U6		-	Poor PA that lacked info	+	Hazard not accessible	-	Starts in Bridge
U7		+	Clear PA with all relevant info	-	Hazard blocking route	-	Starts in Bridge
U8		-	Poor PA that lacked info	-	Hazard blocking route	-	Starts in Bridge

Varying the high and low values of the attributes affected the scenario difficulty. The scenario difficulty is related to the transfer proximity as defined by Barnett and Ceci (2002). Training transfer is considered near transfer when the test setting is very similar to the knowledge covered in the training, and far transfer occurs when the test setting is very different from the training (Ford et al., 2018). For example, in Table 1, the scenarios are listed in increasing order of difficulty by design, (scenario U1 being the least difficult or a near transfer setting and scenario U8 being the most difficult or a far transfer setting from the training). The first set of scenarios (U1-4) had the same starting location that participants had trained for in the VE training. The second set of scenarios (U5-8) had a new starting location, different from where participants had practised, but a location they had seen before.

Scenarios U1 and U2 closely resembled the conditions of the skill acquisition and retention test scenarios. The scenarios gradually increased in difficulty by removing information from the PA announcement and adding hazards to block viable egress routes. Scenarios U7 and U8 were the most different from the conditions of the training and retention scenarios. For example, scenario U8 assessed participants’ ability to respond to a situation in which they started in an unfamiliar location, were provided little information from the PA and had to manage hazards blocking their egress path.

3. Methods

Data from the first two training phases were used to model participants’ decision strategies in the form of DTs. The DTs were used to predict training transfer in phase 3. This section describes the experimental design, including the participants, AVERT simulator, decision tree development, and application of the DTs.

3.1. Participants

Participants from the skill acquisition phase (n = 55) were invited to return after a period of six months to participate in the retention and transfer assessment. Thirty-eight participants completed all three phases of the experiment. Twenty-eight participants were male and ten participants were female. Participants ranged in age from 19 to 54 years (M = 28 years, SD ±8.7 years).

3.2. AVERT simulator

All phases of the experiment used the same emergency preparedness training simulator called AVERT. AVERT is a desktop VE that allows participants to interact with the virtual offshore platform using a gamepad controller (Xbox). The VE depicts a realistic representation of an offshore Floating Production Storage and Offloading (FPSO) vessel (Smith et al., 2019). Participants can move onboard the FPSO by controlling a first-person perspective avatar of an offshore worker. AVERT was used to train participants in basic offshore emergency duties, such as how to navigate the platform and muster at their designated stations during an emergency.

3.3. Decision tree development

DT models depicted participants’ decision-making in selecting the safest egress route during an emergency. The participants’ performance data recorded during the practice scenarios in the VE training was separated into training and testing datasets that were subsequently used to develop and verify generalized decision rules. The training dataset, which consisted of a knowledge base (KB), was used to ‘train’ the decision tree algorithm. The testing dataset was put aside to check the accuracy of the DTs to predict the participants’ route choice in future scenarios.

An individual KB was developed for each participant by tabulating the participant’s successful performance in the practice scenarios and storing it in a two-dimensional matrix. Each row in the KB contained the different programmed attribute values and the corresponding route choice taken by the participant in the scenarios. Table 2 provides a list of attributes and the possible values. Table 3 provides an example of the KB for a sample participant after finishing all of the training in phase 1. The KB was a single data file for each participant.

Table 2. Description of scenario attributes (Smith et al., 2017)

Attributes	Possible Values
End Location	Muster, Lifeboat
Alarm type	None, General Platform Alarm (GPA), Prepare to Abandon Platform Alarm (PAPA)
Route directed by PA	None, 1st, 2nd
Hazard presence	No, Yes
Obstructed route	None, 1st, 2nd
Previous route selected	1st, 2nd

Table 3. Sample participant knowledge base developed from data in phase 1

Scenario	Attributes						Route choice
	End Location	Alarm	Route by PA	Hazard	Obstructed Route	Previous Route
P1	Muster	None	1st	No	None	N/A	Primary
P3 (F1)	Lifeboat	None	1st	No	None	1st	Primary
P3 (F2)	Muster	None	2nd	No	None	1st	Secondary
T1	Lifeboat	None	None	No	None	2nd	Secondary
P4	Muster	GPA	1st	No	None	2nd	Primary
P5	Muster	GPA	None	No	None	1st	Primary
T2	Muster	GPA	1st	No	None	1st	Primary
P6	Lifeboat	PAPA	1st	No	2nd	1st	Primary
T3 (F1)	Muster	GPA	None	No	None	1st	Primary
T3 (F2)	Muster	GPA	2nd	No	1st	1st	Secondary
P7	Lifeboat	PAPA	2nd	Yes	1st	2nd	Secondary
P8 (F1)	Muster	GPA	1st	Yes	2nd	2nd	Primary
P8 (F2)	Lifeboat	PAPA	1st	Yes	2nd	2nd	Primary
T4 (F1)	Muster	GPA	None	Yes	None	1st	Primary
T4 (F2)	Lifeboat	GPA	2nd	Yes	1st	1st	Secondary
T4 (F3)	Lifeboat	PAPA	2nd	Yes	1st	1st	Secondary

The decision tree algorithm of Musharraf et al. (2020) was applied to the KBs to identify participants’ egress route selection strategies. During the DT induction, the algorithm (employing the ID3 engine) iteratively classified data using the attribute that has the highest information gain. This process was repeated until no attributes were left for classification, or the dataset was empty, or data in each group belonged to the same class and no further classification was needed. The DT development methodology is described in detail in Musharraf et al. (2020) and Smith et al. (2017).

The DTs depict the participants’ understanding of emergency egress. Figure 4 shows the strategy of the sample participant. This participant’s strategy was to listen to the PA announcement for information regarding the safest route direction and choose a route accordingly. When no route direction was provided in the PA, the participant followed the primary or secondary route based on their intended end location. If they were required to go to the muster station, then they would take the primary route, and if they were required to go to the lifeboat station, then they would take their secondary route.

Figure 4. Sample participant decision tree developed from the KB shown in.Table 3

As participants completed the retraining in phase 2, data were updated in their KB. This updating resulted in the formation of different DTs throughout the retraining.

Once the DTs were generated based on the VE training dataset from phases 1 and 2, the testing dataset was used to check the prediction accuracy of the decision trees. The DTs were compared against the participants’ route choices in the test scenarios. The prediction accuracy was calculated as a percentage of correctly matched trees for each test scenario.

3.4. Application of decision trees

Three applications of DTs were used to assess the training curriculum: (1) comparing the participants’ DTs to the ideal DT as an indicator of whether the curriculum adequately prepared participants for transfer, (2) tabulating the different DTs formed throughout the training as a measure of the heuristics developed in the VE training, and (3) using the DTs to predict performance in the new scenarios.

3.4.1. Comparing participants’ DTs with ideal DT

The ideal strategy (depicted in Figure 1) was used as a benchmark to compare with each participant’s behavioural patterns. The participants’ DTs at the end of the acquisition phase and at the end of the retraining phase were compared to the ideal DT. This comparison was performed to check that the participants’ DTs met the curriculum criteria. There were three possible outcomes of the comparison: match, incomplete or incorrect.

If a participant’s DT matched the ideal DT, this was a predictor that the participant had reached the desired competence and was ready for training transfer scenarios. If a participant’s DT did not match the ideal DT, this was a predictor that the participant was not prepared for the new transfer scenario. The incomplete or incorrect trees were indicators that the participant required additional adaptive training before they would be successful in the transfer scenarios. If no behavioural pattern could be determined from the participant’s performance data, it was likely that the training was not suitable for this person.

3.4.2. Tabulating DTs to evaluate training curriculum

The DTs were tabulated to identify emerging group patterns to inform the efficacy of the training. Several heuristics emerged from the participant performance data, as illustrated in Figure 5 (Smith et al., 2017). These heuristics represent decision rules that were directed by the end location, the alarm type, and whether or not the travelled route is obstructed. These strategies were not explicitly taught in the training but are acceptable decision rules in many situations.

Figure 5. Accepted heuristics for emergency egress

The percentage of participants whose DT converged to the ideal DT or an acceptable heuristic (shown in Figure 5) were tabulated. This grouping of DTs was used to determine if the training program adequately prepared participants to apply their egress skills to new emergencies. If the participants’ DTs converged to the ideal DT or an acceptable heuristic, this was an indicator that the training curriculum was successful in preparing participants for the new transfer scenario. If the participants’ DTs diverged from the ideal DT, or developed unacceptable decision rules, this was an indicator that there were gaps in the training curriculum or that the trainee was not suitable for the training.

3.4.3. Use of DTs to predict skill transfer to new scenarios

DTs show the decision rules a participant is expected to follow based on inferring from previous performance in similar scenarios. Thus, DTs can be used to predict the participants’ route choice for future scenarios. To predict how the participants would perform in the eight transfer scenarios from phase 3, the modelled DTs from the retraining dataset in phase 2 were used. The DTs predicted performance was compared against the participants’ actual route choices in the transfer scenarios. This comparison formed the basis of a prediction accuracy calculation, which was calculated for the eight transfer scenarios.

4. Results and discussion

A combination of DT modelling and empirical results was used to investigate the transferability of the VE training. To inform training transfer, three metrics were used: (1) the DTs’ diagnostic assessment of participants’ performance and the training curriculum efficacy; (2) the overall performance scores in the transfer scenarios; and (3) the DTs’ prediction of participants’ performance in the transfer scenarios.

4.1. Learning analytics and curriculum assessment with DTs

At a systemic level, grouping emergent DT patterns is a useful predictor of how well the training curriculum equipped participants to apply their egress skills to new situations. This section provides a comparison of the participants’ DTs with the curriculum designed ideal DT and discusses the implications of emergent DT patterns on the efficacy of the training curriculum.

DT patterns that emerged from the training were tabulated. Eight different types of DTs were modelled using the DT updating process throughout the VE training. Table 4 shows the percentage of participants who formed each type of DT developed from data collected at the end of the acquisition (phase 1) and retraining (phase 2).

Table 4. Decision trees after acquisition (ACQ) and retraining (RET) phases

Type	Decision Rules	% Participants		Compared to Ideal DT
		End Phase 1	End Phase 2
1		60%	68%	Match
2		16%	10%	Match
3	*1 participant had a similar DT but reversed rules for PAPA and None	10%	13%	Match
4		3%	0%	Match
5		3%	0%	Incomplete
6		3%	0%	Incorrect
7		0%	3%	Incorrect
8		0%	3%	Incorrect
	*No behavioural pattern or strategy identified	5%	3%	Trainees not suitable

The ideal DT was used as a benchmark for comparison to determine if the participants’ behavioural patterns, identified by their DTs, converged to the intended strategies. This comparison was performed to determine if the participants achieved the target competence. The ideal DT was compared to participants’ modelled DTs at two stages: after the initial acquisition (phase 1) and after the refresher training (phase 2). Four types of DTs were identified as successful matches: types 1, 2, 3 and 4 (depicted in Table 4, column ‘Match’). Four other DTs did not meet the requirements for safe evacuation: types 5, 6, 7 and 8 (depicted in Table 4, column ‘Incomplete’ or ‘Incorrect’).

As shown in Table 4, grouping emergent DT patterns provided an indication of whether the training curriculum adequately prepared participants to apply their egress skills to new emergencies in the virtual offshore platform. In phase 1, 89% of participants’ behaviour represented the ideal DT or an accepted variation of the tree, and the remaining 11% of participants (four participants) did not form a tree to match the intended decision strategy. In phase 2, 91% of participants’ behaviour represented the ideal DT or an accepted variation of the DT, and the remaining 9% of participants (three participants) formed incorrect trees. The following subsections discuss the implications of the emergent DT patterns on measuring participants’ competence and assessing the efficacy of the training curriculum.

4.1.1 Description of successful decision trees

The training curriculum taught participants three main tasks for responding to emergencies: (1) to listen to the alarm type as it indicated the severity of the situation, and indirectly, the muster location; (2) to listen for the PA announcement as it sometimes provided information on the safest egress route; and (3) to re-route if their egress route was obstructed by hazards. Successful decision rules included the DTs that started with information for the PA announcement (e.g. types 1, 2, 3, and 4). For these types of DTs, the participants’ route selections were decided primarily based on their understanding of the PA announcement. Two-thirds (68%) of participants formed type 1 DTs, which matched the decision rules of the ideal decision strategy in Figure 1 (e.g. relied solely on information from the PA in choosing the egress route).

Three other DTs developed contingency branches for situations where the PA provided little route information. In such cases, the DT included using attributes such as the end location (type 2), alarm type (type 3) and route obstructions (type 4) in making the route choice. These three types of DTs represent heuristics that participants formed throughout the training. For example, the type 4 DT specified that in the absence of a PA announcement, the participant’s choice was based on whether the route was obstructed by a hazard. If the primary egress route was obstructed, the participant would re-route to the secondary route, and if the secondary egress route was obstructed, they would re-route to the primary route.

The decision rules for type 2, 3 and 4 are good examples of decision strategies that participants developed on their own in the training program. These emerging behavioural patterns are positive indicators of how the training resulted in learning strategies that surpassed the intended learning objectives.

4.1.2 Description of incorrect trees

Four other DTs developed from participants’ data represented decision rules that were either incomplete, too specific, or incorrect (types 5, 6, 7 and 8). These are examples in which the participants may not have received sufficient training to develop the ideal DT, or they developed their own heuristics that were not safe strategies for emergency egress. For these types of DTs, there was too much emphasis placed on incorrect attributes. For example, the type 5 decision rules were incomplete and focused solely on whether or not the egress routes were obstructed. Type 6 decision rules had too many specific branches due to overfitting the data (i.e. representing a specific rule for each scenario attribute). For type 7 decision rules, the participant’s choice was based on the presence of hazards. If the scenario had a hazard, the participant would take the primary route, and if there was no hazard, they would take their secondary route. This rule suggests that the primary route is the only safe route if there are hazards in the scenario and incorrectly presumes that there will never be hazards in the accommodation block. For type 8 decision rules, the participant’s choice was based on the last route taken in the previous scenario. This participant would indiscriminately alternate between egress routes regardless of the scenario attributes. If the participant had taken the primary route in the previous scenario, then the participant would alternate and take the secondary route in the next scenario and vice versa. Rules based on previous route choices provided no advantage in responding to emergencies.

The unsatisfactory trees identify weaknesses in the individual trainee as well as training curriculum. To improve the curriculum design, these weaknesses can be addressed by providing individuals with more practice exercises to target the errors they are making and reinforce the decision rules of what to do in these circumstances (e.g. offering experience to build the correct contingency branches). In the case of building decision rules based on the presence of hazards (type 7), this is an indicator that the participant has not received enough practice scenarios to know what to do if a hazard occurs on their secondary route. Additional training scenarios should be provided to modify the branches on this participant’s DT. Similarly, in the case of type 8 decision rules, this is evidence that the participant might not be challenged by the exercise and is simply gaming the experiment by taking the opposite route for each scenario. This participant should receive training scenarios to challenge them and reinforce the importance of the training.

In a few situations, the VE training was not compatible with the learner. Some participants’ performance data did not have a recognizable pattern and the DTs were not drawn. If no behavioural pattern could be determined from the participant’s performance data, it was likely that the trainee was not suitable for this type of VE training.

4.2. Performance in transfer scenarios

The performance scores provided another indicator of how well participants were able to apply their egress skills to new situations in phase 3 of the experiment. Figure 6 provides a visual representation using boxplots of the participants’ performance in the eight transfer scenarios.

Figure 6. Boxplot of performance scores for the transfer scenarios

The overall performance showed that participants successfully applied their egress skills in the transfer scenarios that were similar to the training conditions (e.g. near transfer for test scenarios U1, U2 and U3). The conditions of these scenarios involved egress from the cabin. If there was a hazard in the scenario, its location was often identified in the PA announcement. The participants’ performance dropped in scenario U4. This scenario also involved egress from the cabin and a hazard blocking the primary egress route, but in this case, there was no information from the PA announcement about the hazard. Almost all participants (97%) encountered the fire in the main stairwell, blocking their primary egress route, and were forced to re-route to the secondary egress route to their muster station.

As the conditions in the transfer scenarios moved beyond the scope of the training, the overall performance of the participants decreased (e.g. for far transfer test scenarios U5, U6, U7 and U8). The conditions in these scenarios involved egress from a less familiar starting location (the bridge). The high and low values of the scenario attributes for hazards and the PA were also varied (e.g. scenario U6 provided little information about the emergency over the PA and scenario U7 had a major egress route obstructed by hazardous conditions). In particular, participants’ performance dropped in scenario U8. This scenario deviated the most from the training conditions and assessed the participants’ ability to respond to a situation starting in the bridge, with smoke and fire hazards blocking their egress path, and little information from the PA. Seventy-nine percent of participants encountered the fire and deluge system in the external stairway, blocking their secondary egress route, and were forced to re-route to the primary egress route to reach their muster station.

4.3. DTs to predict training transfer

The DTs from the retraining (depicted in Table 4, column ‘Phase 2’) were used to predict participants’ route choices in the transfer scenarios. The prediction accuracies were calculated for the transfer scenarios by comparing the modelled DTs to the participants’ corresponding route choices in the 8 scenarios. Table 5 shows the percentage of DTs that predicted participants’ successful performance in phase 3.

Table 5. Percentage of DTs that predicted participants’ successful performance

Tree Type	Percent DTs predicted successful performance in transfer scenarios
	Near transfer				Far transfer
	U1	U2	U3	U4	U5	U6	U7	U8
All Types	66	85	82	87	13	16	53	16
Type 1	47	66	58	63	5	13	40	8
Type 2	8	8	11	11	0	0	5	3
Type 3	11	11	13	13	8	3	8	5

The DTs at the end of the retraining had mixed success in predicting the route selection choices of participants in the transfer scenarios. The prediction accuracy of the DTs was sensitive to the degree of transfer (i.e. near and far transfer). Near transfer scenarios (U1-4) tested participants’ application of egress skills in a transfer setting that was very similar to the training (e.g. scenarios requiring egress from the cabin). Far transfer scenarios (U5-8) tested participants’ generalization of egress skills in a transfer setting that was very different from the training (e.g. scenarios requiring egress from the bridge). The DT prediction accuracy was higher for the scenarios that started in the cabin (66–87%) than the scenarios that started in the bridge (13–53%). As a result, the DTs could not predict successful performance in all the transfer scenarios.

The poor prediction accuracy of the second set of transfer scenarios is not a reflection of the assessment quality of the DT modelling approach, but an indicator of the gaps in the training curriculum and weaknesses in individual trainees. This indicator shows that the VE training did not adequately prepare all participants to respond to the emergency scenarios in phase 3 because some of the training transfer scenarios (U5-8) were beyond the scope of the initial training. DTs use supervised learning, which involves a database of solved examples to infer choices or actions in future cases. Supervised learning is applicable when the future cases are within the same parameters of the trained model. Therefore, if the scope of the transfer scenarios reaches beyond the training context, the DTs will not be able to predict actions for unforeseen or ill-defined future states. This can be considered a similar mechanism to how trainees learn, apply and generalize knowledge. Trainees learn information in training context and they are expected to apply their skills to real-world situations. It is unreasonable to expect a trainee to transfer skills to situations beyond what they have been trained for. Thus, if the DT depiction of the participants’ training experience cannot adequately predict their performance in the transfer scenarios, it is possible that the reason for the poor prediction is that the training did not adequately prepare the participants for the new emergency scenarios. The DT prediction analysis provides a valuable diagnostic lens to systemically assess the efficacy of a training program.

4.4. Recommendations to improve transferability

The results from the DT analysis identified gaps in the design of the VE training curriculum. Specifically, this process found limitations in the benchmark DT and the size of the participants’ KBs (i.e. the number of training examples).

The ideal DT must be improved to address the participants’ deteriorating performance and to increase the reduced DT prediction accuracy for complex transfer scenarios. A way forward is to modify the ideal DT to improve the learners’ preparedness for both near and far transfer scenarios. Figure 7 provides an example of an improved DT with additional branches to address the situations participants faced in the far transfer scenarios (e.g. U5-8, when the PA does not provide information about the situation and the participant is required to make a decision based on obstructed egress routes).

Figure 7. Revised decision tree for emergency egress

To achieve the revised DT, participants should be provided with more training scenarios to target the intended decision strategies and thereby increase the number of entries in their corresponding KB. Adding more practice scenarios (e.g. situations without a PA announcement and obstructed routes) will improve the participants’ repository of solved examples, which is necessary to develop the contingency branches to match the revised DT (in Figure 7). However, practice scenarios should target the needs of each individual learner. Combining DT modelling with adaptive training would provide a customized tool to ensure each learner achieves demonstrable competence by developing a DT to match the intended decision strategies (e.g. developing the contingency branches needed to help learners respond to many plausible emergencies).

Another gap identified by this process was that the target DT represented the existing protocols as defined by regulations. However, this DT was found to be not comprehensive enough to predict performance in the complex transfer scenarios. The transfer scenarios in this experiment were plausible scenarios that challenged the scope of the VE training by testing situations that go beyond the competencies dictated by industry regulations in order to measure skill transfer to emergency conditions. This is important to highlight because emergencies are characterized by complexity, time pressure, and uncertainty. Offshore safety training must be robust enough to apply to a wide range of circumstances and therefore VE training should be designed to encourage training transferability.

5. Conclusion

This research used DTs as data-informed curriculum design and assessment tools to evaluate the transferability of virtual offshore training. Visualizing participants’ decision strategies can help instructional designers determine if participants are adequately prepared for new training transfer situations. The results of this work showed that DTs can model participants’ decision-making strategies throughout the acquisition, retention and retraining, and transfer phases of the experiment. The majority of participants’ DTs converged on intended strategies. This convergence is an indicator that the participants achieved the intended demonstrable competence. Similarly, the majority of participants demonstrated training transfer in test scenarios that were similar to the training conditions (near transfer test scenarios). The DT analysis provided evidence that participants accrued sufficient training to transfer their skills (e.g. promote the application and generalization of skills).

However, not all participants developed the ideal DT, which shows the limitations of the trainee and the training. For some participants, their performance data developed credible heuristics (e.g. specific decision rules that were not explicitly taught). This finding highlights the diagnostic utility of DTs as they show how participants develop strategies that do not always match the intended training material. From a systemic perspective, DTs are useful for identifying the strengths and weaknesses of a training curriculum. To improve the training, some participants required more opportunities to practise scenarios with varied attributes to ensure they develop the targeted route selection strategies. This retroactive analysis could be performed in real time as a basis of adaptive training. VE training can be further optimized by combining DT methodology with adaptive training mechanisms to provide participants with customized scenarios to meet their specific learning need and to prepare them for training transfer.

Decision trees also have the potential to predict when a person is prepared or not prepared for more advanced situations. Overall, the participants’ data-informed DTs were successful in predicting the participants’ performance in new scenarios that represented near training transfer. As a group, the performance decreased as scenario conditions moved beyond the scope of the initial training (far transfer). Similarly, the DTs developed throughout the training were not good at predicting participants’ route selection strategies in situations that went beyond the training context. The poor DT prediction of participant performance in the transfer scenarios is an indicator of gaps in the VE training. The far transfer scenarios were plausible scenarios that challenged the scope of the VE training by testing situations that extended past the competencies prescribed by regulations. This finding suggests that DT can push the boundaries of existing training by highlighting its shortcomings and challenging instructional designers to develop training that prepares people for emergencies, rather than for the somewhat nominal requirements of regulations.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and Husky Energy Industrial Research Chair in Safety at Sea, grant number IRCPJ500286-14.

Notes on contributors

Jennifer Smith

Dr. Jennifer Smith is the Human Factors Research Coordinator for the Safety in Sea project at Memorial University. This paper is a part of her PhD research. Her research focuses on pedagogical and human factors issues in marine and offshore simulation training.

Mashrura Musharraf

Dr. Mashrura Musharraf is an Assistant Professor of Marine Technology at Aalto University and a former Post Doctoral researcher with the Safety at Sea project at Memorial University. Her research focuses on using data mining, machine learning, and AI to build and deploy human-centered systems and solutions and create a safer marine industry.

Allison Blundon

Allison Blundon is a former graduate student of the Safety at Sea project at Memorial University and assisted with the experimental design and data collection for this paper.

Brian Veitch

Dr. Brian Veitch is a professor in Ocean and Naval Architectural Engineering with the Faculty of Engineering & Applied Science at Memorial University. He is the Natural Sciences and Engineering Research Council of Canada (NSERC) / Husky Energy Industrial Research Chair in Safety at Sea.