Designing simulator-based training: An approach integrating cognitive task analysis and four-component instructional design

Abstract

Most studies of simulator-based surgical skills training have focused on the acquisition of psychomotor skills, but surgical procedures are complex tasks requiring both psychomotor and cognitive skills. As skills training is modelled on expert performance consisting partly of unconscious automatic processes that experts are not always able to explicate, simulator developers should collaborate with educational experts and physicians in developing efficient and effective training programmes. This article presents an approach to designing simulator-based skill training comprising cognitive task analysis integrated with instructional design according to the four-component/instructional design model. This theory-driven approach is illustrated by a description of how it was used in the development of simulator-based training for the nephrostomy procedure.

Introduction

It is generally acknowledged that a ‘training needs analysis’ should be one of the first steps in the development of educational interventions, including simulator-based training (Salas & Cannon-Bowers 2001; Schout et al. 2010b), but other aspects that need to be considered in designing and developing simulator-based training of complex surgical skills have received scant attention in the literature. Studies evaluating surgical simulators generally emphasise either psychomotor skills or theoretical procedural knowledge (Hacker et al. 2007; de la Rosette et al. 2008; Mishra et al. 2010a, b; Schout et al. 2010a, b), but surgical procedures rely not solely on ‘how to’ perform skills appropriately, they also require appropriate decision making at certain points during the procedure (Hall et al. 2003; Flin et al. 2007). Although it seems clearly preferable for surgical simulators to be aimed at integrated training of psychomotor and cognitive skills, systematic methods for designing such simulators remain to be developed.

The need for integrated simulator-based training has become more urgent today, now that the recent introduction of competency-based programmes in postgraduate specialty training, the restrictions of residents’ working hours and financial constraints are all contributing to a decrease in the opportunities for ‘learning by doing’ in the clinical workplace (Hall et al. 2003; Brandenberger et al. 2010; Parsons et al. 2011), the cornerstone of the traditional master-apprentice model characterised by skills acquisition starting with observation, followed by imitation under gradually fading supervision and culminating in independent performance. During this process, trainees learn to integrate manual and cognitive skills, the latter including the ability to take appropriate decisions at appropriate points during a procedure. Nonetheless, when experts are asked to explain their performance in detail, it seems quite difficult for them to identify decision moments with any precision (Clark et al. 2008). Simulators could complement master-apprentice learning by explicitly pinpointing such moments, thus enabling integration of psychomotor and cognitive training for learning surgical tasks (Schout et al. 2010b; Tun & Kneebone 2011).

Surgical tasks range widely from simple procedures, for example removing skin lesions and closing wounds, to highly complex surgical tasks such as laparoscopic removal of the prostate. The level of task complexity has been defined by the number of actions (either mental actions or movements) and the interactions between them; the more interactions, the more complex the task (van Merriënboer et al. 2006; Brydges et al. 2007). Both mental actions and movements related to complex surgical tasks should be carefully identified before implementing in training programmes (Kneebone 2003).

Development of simulator-based training should preferably start with a structured analysis of the complex procedure to be simulated using ‘cognitive task analysis’ (CTA), a generic term referring to a variety of interview and observation methods aimed at unravelling experts’ performance of complex tasks in order to capture the knowledge, thought processes and goal structures experts rely on during task performance (Clark et al. 2008; Yates & Feldon 2010). Before this information can be used in developing simulator-based training, it needs to be translated into instructional design focusing primarily on the identification, ordering and presentation of information about a skill in a blueprint for training that can help learners acquire skills in a coordinated and integrated manner (Clark et al. 2008).

In learning complex surgical tasks, trainees can either practice according to a ‘whole-task approach’ or a ‘part-task approach’. In a whole-task approach, learners will typically practice simple to complex versions of the whole task; in a part-task approach, the whole task is divided into several parts, and learners practice the parts apart from each other. Which method to use depends on the complexity of the task: for simple tasks with few interactions between actions, a part-task approach is more effective, whereas for complex tasks with many interactions between actions, a whole-task approach is more effective. For example, Brydges et al. (2007) found that trainees learning complex tasks according to the whole-task approach reached significantly better outcomes than trainees who practiced according to a part-task approach.

One of the most extensively researched systematic models for instructional design is the four-component/instructional design (4 C/ID) model developed by van Merriënboer (1997), which differs from traditional instructional design – where tasks are deconstructed into parts to be learned separately before being reassembled for whole task practice – by proposing that learners should perform skills of increasing complexity as whole tasks comprising four interrelated components: scenarios, supportive information, just-in-time (JIT) information and part-task practice (Janssen-Noordman et al. 2006; van Merriënboer & Kirschner 2007; Clark et al. 2008).

In this article, we present an approach combining CTA and the 4 C/ID model to create a blueprint for simulator-based surgical training, illustrating this approach by describing how we applied it in designing simulator-based training for a complex procedural skill, the nephrostomy procedure.

No ethical approval was required for this study as there was no involvement of patients and no substantial effect on direct patient care.

Table

Practice points

Simulator developers, educationalists and medical experts should collaborate in the development process of simulator-based training. Cognitive task analysis (CTA) is a suitable method for obtaining cognitive and procedural information from experts. CTA methods integrated with a training design model such as the four-component instructional design, results in a blueprint useful for simulator-based training developers.

What is a nephrostomy procedure?

Urine production is one of the major functions of the kidney, and the renal system is composed of calices where urine is drained into the collecting system before moving down into the bladder. Some of the multiple scenarios causing obstruction of the renal system require an intervention like the nephrostomy procedure (Horton et al. 2008), a common procedure in urology for gaining direct renal access, from the skin into the kidney, for urinary drainage using a nephrostomy catheter. Before the catheter is introduced, access from the skin into the kidney should be obtained with a hollow needle and guidewires. The procedure is ultrasound guided and conducted by a urologist or a radiologist, generally under local anaesthetic. The severe discomfort experienced by patients when the procedure takes too much time to complete or is complicated by events like the need for multiple punctures or haemorrhage emphasises the need for training to acquire the appropriate skills before performance in real patients so as to prevent complications and reduce performance time. Skilful performance of nephrostomy prevents damage to renal vessels, surrounding organs and the urinary collecting system as well as major complications, such as damage to the renal vessels causing massive haemorrhage, bowel perforation, pleural perforation and damage to other surrounding organs. Complication rates of performance by experienced specialists vary from 1% to 4%, and correct choice of calyx and correct performance of the procedure can lower the risk of complications (Horton et al. 2008; Karim et al. 2010).

How to conduct a CTA?

On the basis of the principle that performance of any complex task requires a combination of psychomotor and cognitive skills, CTA is aimed at unravelling how experts make decisions while performing a complex task (Salas & Cannon-Bowers 2001). Because novices are by definition still learning, their task performance is likely to be slower, clumsier and more prone to error than that of experts. Not only do novices’ manual skills fall short, due to their lack of familiarity with all the steps of a procedure, each step requires conscious and deliberate decision making. Experts, by contrast, perform procedures more fluently and with greater ease having automated many of the steps to the extent that performance requires little if any conscious effort. It is precisely because experts are no longer conscious of every step of their performance that they, when asked to elaborate on the steps of a certain task, have difficulty identifying the points where decisions are made (Clark et al. 2008). In CTA, it is therefore important to identify which parts of a task can be classified as ‘automatic’, i.e., being generally performed by experts without conscious awareness, and which parts as ‘non-automatic’, requiring conscious attention also from experts. We will describe the CTA method we used to analyse the nephrostomy procedure.

CTA of the nephrostomy procedure: A practical approach

In order to unravel the cognitive steps of a nephrostomy procedure, we used a two-stage CTA approach. The first stage comprises questionnaires to identify the main steps of the procedure, and the second stage was a semi-structured interview to unravel automatic and non-automatic sub steps.

• Participants: Four urologists, all from the same hospital, were asked to complete a questionnaire, and eight expert urologists from seven different hospitals were invited to participate in a semi-structured group interview. All participants performed nephrostomy procedures on a regular basis. Two urologists, who completed the questionnaires, were also included in the semi-structured interview; however, they were unaware of the results before attending the semi-structured interview. The eight urologists were all male, had been practising urologists for a mean period of 12 years (range 5–25 years) and had performed a median number of 175 nephrostomy procedures (range 5–2000).

• Method: The four urologists were asked to complete a questionnaire about the sequential steps of nephrectomy, asking them to identify the steps and list for each step which instruments and materials were needed and which complications could occur. After the questionnaires were analysed, a second round of the CTA was held aimed at reaching agreement on the various elements of the procedure based on the results of the analysis and the literature. A third round was not required as consensus was reached. The eight urologists took part in a four-hour expert meeting starting with an introduction of the educational principles of skills training and an explanation of the purpose of the meeting, followed by a semi-structured group interview conducted by an educational expert, a researcher and an expert urologist. A scenario of a non-acute patient requiring a standard nephrostomy guided the questionnaire, the initial results of the questionnaire, supported by five main questions (Table 1) and any new questions prompted by issues brought up during discussion with the experts, guided the interview. Each step was analysed in the order shown in Figure 1 to identify automatic and non-automatic sub-steps according to their own experience. For each step and sub-step all questions had to be answered, and the step had to be classified as automatic or non-automatic. Not until all sub-steps had been dealt with and consensus was reached, did the interview move on to the next step of the procedure. If consensus was not achieved, the differing opinions were recorded in writing.

• Data analysis and interpretation: The CTA was documented in a flowchart, and the interview was recorded using a digital voice recorder. The data were analysed by the researcher who additionally described the decision-making process leading to the categorisation of (sub-)tasks as automatic or non-automatic.

• Results: The first column in Figure 2 shows the sequential steps of the nephrostomy procedure arrived at after the two stages of the CTA. It proved impossible to reach consensus on the place in the sequence of ‘incision in the skin’ and ‘local anaesthetics’, but extensive discussion resulted in agreement on the sequence of steps shown in Figure 2. During the interview, several sub-tasks with high levels of automaticity were identified. The following sub-tasks were classified as requiring conscious attention and thus classified as non-automatic: ‘Following the tip of the needle continuously during insertion from the skin into the renal system’ and ‘Introducing the nephrostomy catheter over the guidewire into the renal system’. After a thorough discussion, consensus was reached among all eight urologists, considering the identification and classification of automatic and non-automatic steps.

Figure 1. Flowchart of CTA.

Table 1.  Five questions that guided the semi-structured interview

1.	Please identify all sequential steps of the nephrostomy procedure?
2.	Which steps can be considered as automatic and which can be considered as non-automatic? (follow flowchart CTA, Figure 1)
3.	Which complications are likely to occur?
4.	How do you prevent the complications mentioned in question 4?
5.	Please report five cases with different difficulty levels.

Figure 2. Flowchart of tasks and sub-tasks of the nephrostomy procedure. Notes: All tasks are classified as automatic (A) or non-automatic (NA). All steps of the left vertical sequence are considered main steps, of which some are differentiated into sub-steps, because they were first considered as non-automatic. The tasks in the grey boxes require high automaticity according to the experts.

Figure 3. Blueprints for high fidelity simulator-based training for the nephrostomy procedure, developed using CTA and 4 C/ID model in an integrated manner.

Figure 2 shows a flowchart presenting details of the sequential steps and sub-steps of the nephrostomy procedure resulting from the interview. All steps were classified as either automatic or non-automatic.

How to design simulator-based training according to the 4 C/ID-model

Developed by van Merriënboer (1997), the ‘4 C/ID model’ is one of the most extensively developed instructional design models (van Merriënboer & Kirschner 2007; Clark et al. 2008). It is aimed at creating whole-task training integrating different competences, based on the assumption that effective training consists of four interrelated components: scenarios, supportive information, JIT information and part-task practice (van Merriënboer & Kirschner 2007; Clark et al. 2008).

• Scenarios are whole tasks of increasing levels of difficulty.

• Supportive information is the knowledge prerequisite for performing a task, particularly the skills classified as non-automatic, and it is divided into descriptive information, prescriptive information and cognitive feedback. Descriptive information relates to theoretical knowledge, such as knowledge about the pathophysiology of post-renal problems and human anatomy. Prescriptive information consists of ways to tackle a problem, such as systematic performance of a procedure. Cognitive feedback helps learners reflect on the quality of their acquired supportive information. According to the 4 C/ID model, trainees need more, or more detailed, supportive information as scenarios increase in difficulty.

• JIT information provides ‘how-to’ instructions and direct feedback on the performance of skills classified as automatic and with potentially high-impact complications in the case of failure, such as haemorrhage or mortality.

• Part-task practice can be developed for skills that have to be trained to a high level of automaticity and can potentially benefit from isolated complementary training.

Designing nephrostomy training: A practical approach

We illustrate the development of a design for simulator-based surgical skills training according to the 4 C/ID model by describing how we arrived at a blueprint for simulator-based nephrostomy training in a process in which CTA and the 4 C/ID model were integrated.

• Participants: The same eight urologists who participated in the semi-structured interview.

• Methods: In order to elicit information for use in the instructional design, two additional questions relating to error analysis and scenario design were included in the semi-structured interview, requesting the urologists to create a variety of scenarios representing at least two levels of difficulty. Information gathering for the other three components of the 4 C/ID model was integrated in the semi structured interview as is shown in Figure 1. Component 1 requires scenarios of different levels of difficulty; component 2 requires supportive information for non-automatic skills; component 3 requires ‘how-to’ instructions, particularly direct feedback on performance of automatic skills (JIT information) and component 4 requires ‘part-task practice’ for sub-tasks to be trained to a high level of automaticity.

• Data analysis and interpretation: The data from the interview were used as input for the design of a blueprint comprising the four components.

• Results of the training design according to the 4 C/ID model: We analysed the results of the interview and allocated the answers from the interview to one of the four components. The final result was a blueprint consisting of different simulator training scenarios (Figure 3).

Scenarios

The semi-structured interview resulted in six scenarios offering different levels of difficulty depending on different combinations of standard anatomical features (Table 2) and on the presence or absence of time pressure. Puncture, for example, is more difficult in a morbidly obese patient compared to a non-obese patient, because of the distance between the skin and the kidney. If ‘non-echoic’ fatty tissue is present, the scenario becomes even more difficult. With a septic patient, it is urgent that a procedure should be performed accurately and within the shortest possible time. Given these variations between patient cases, it is advisable to design training options for the procedure with and without time pressure. The six scenarios involved patients with: dilatation of the renal system, kidney stones in the renal system, scoliosis, cystic kidneys, medullary sponge kidney and horseshoe kidney.

Table 2.  Case variations: Human features

	Easy	Difficult
Kidney	• Fixed	• Mobile
Fatty tissue	• Echoic	• Non-echoic
Distance between skin and kidney	• Small	• Large
Surrounding organs	• No other organs in trajectory	• Other organs in trajectory
Renal calyx of preference	• Lower pole	• Upper pole
Scar tissue	• No scar tissue	• Scar tissue

Table 3.  List of possible errors during nephrostomy procedure mentioned by the experts

Major

• Penetration of other organ: liver, spleen, colon and renal vessels • Penetration of renal system causing leakage

Minor

• Taking wrong trajectory and entering another calyx • Taking wrong trajectory and puncture of lower rib • Puncture outside the kidney, without causing trauma to surrounding tissue/organs • Insecure fixation of the outer needle or inaccurate removal of the inner needle possibly causing dislocation of all instruments and fall outside the renal system • Insecure fixation of the guidewire while dilating or introducing the nephrostomy catheter potentially causing dislocation of all instruments and fall outside the renal system • Excessive duration of procedure increasing risk of sepsis • Frequent and excessive duration of radiation

Supportive information

The urologists unanimously agreed that trainees should have knowledge about normal human anatomy and its variations as well as about how to obtain optimal ultrasound images. The simulator should indicate whether skills and sub-skills are performed correctly during training. When the trainee starts a scenario of a higher level of difficulty, the simulator should provide more or more detailed supportive information. For the scenario of a patient with serious scoliosis, for example, the trainee must have knowledge about the differences in anatomy and organ position compared to a patient without scoliosis.

JIT information (error analysis)

All urologists agreed that the potential major complications of nephrostomy were penetration of the needle into another organ (colon, spleen, liver or lung) or a blood vessel, whereas minor errors were situations that might result in failure of the procedure, but with a re-procedure still remaining an option (Table 3). All urologists agreed that trainees must learn to recognise complications and how to handle them. In the cases of major haemorrhage, for example, trainees should immediately intervene by proceeding to open surgery in the operating theatre. All information resulting from the error analysis was suitable for use as JIT information in the blueprint.

Part-task practice

The flowchart showed that training of all bimanual skills, with one hand being fixated while the other hand is used to perform the skill, was considered important and suitable for additional, separate ‘part-task practice’. The semi-structured interview yielded the following skills for which ‘part-task practice’ was considered useful: puncture a mobile kidney, puncture a kidney with scar tissue and handle different human features (Figure 3).

Reflection on the approach to designing surgical training integrating CTA and 4 C/ID

In a review, Hall et al. (2003) underlined the link between surgery and cognition by pointing out the importance of using educational tools in surgical training to reduce learning curves of new procedures.

To identify different steps in a procedure, Jonassen et al. (1999) describe ‘behavioural task analysis’, which focuses mainly on observations of task performance. A clear disadvantage of this approach for analysing experts’ performance and goal structures is that results are based on the observers’ interpretation of the technical and observable aspects of performance only. In this study, we aimed to identify automatic and non-automatic steps that may also include non-observable decision-making processes, thereby making the approach described by Jonassen et al. less suitable. CTA covers both cognitive and technical skills, offering a broad range of methods to gain knowledge and thus makes it appropriate for different research purposes, aimed to identify conscious and unconscious skills.

An alternative theory for the process of instructional design is presented by Gagné and Merrill. For situations where integrated learning objectives must be reached, ‘enterprises’ (cf. complex tasks) should help learners to develop the necessary skills in a whole-task approach (Gagné & Merrill 1992). Similar to the 4 C/ID model, instructional design according to Gagné aims to develop multi-objective lessons or whole-task practices; and in both theories, the learning tasks comprise both cognitive skills and technical skills to support real-life task performance. An advantage of the 4 C/ID method above Gagné's theory is, however, that it offers a format to elicit knowledge, which can be subsequently translated into a training blueprint for non-experts (van Merriënboer & Kirschner 2007).

Different from studies using a single-theory approach, in our study to design simulator-based surgical skill training, we researched a systematic approach in which CTA and 4 C/ID were used in an integrated manner. This approach offered the additional benefit of enabling us not only to identify the required psychomotor skills but also to unravel and make explicit experts’ intuitive decision making, something that is generally very difficult for experts to achieve without support, and which resulted in the identification of concrete points of decision-making during the procedure, thereby making these cognitive skills amenable to training. Identification of highly automatic components is of particularly great importance, as it enables inclusion in the training blueprint of separate practice opportunities for these components. The identification of automatic components in the procedure was based on the results of the semi-structured interview.

When CTA is fully integrated in the instructional design process, optimal use can be made of its results. A major benefit of this integrated approach is that it yields a blueprint for scenarios that is expected to be easily understood by engineers without any medical or educational background who can use it in building a simulator. However, whether engineers will actually be able to accurately interpret the blueprint and use it to inform simulator design will have to be investigated in further studies. As the integrated design method we used is generic, it can be used in designing any type of surgical skill training.

A drawback of CTA is the considerable time investment involved in an expert meeting lasting several hours preferably attended by experts from different hospitals. It seems likely that more complex procedures, radical prostatectomy for example, will require several meetings as well as observational studies to analyse all the steps. Semi-structured interview was supposed to be most appropriate for the purpose of this study, identifying not only experts’ observable actions but also intuitive decision-making processes during the nephrostomy procedure by recalling their own experience. Using tools such as video or stimulated recall as an alternative would narrow the scope of this study because the urologists are then asked to analyse a single, idiosyncratic procedure. In addition, offering a video could interfere with the identification of steps that are performed by intuitive decisions.

As we stated previously, CTA is a generic term covering a variety of interview and observation methods, and the semi-structured group interview we used should definitely not be regarded as the only method possible for gaining insight. Researchers should consider other interview and observation methods and carefully define their research population as well, to find the most time-efficient method for each distinct research purpose.

In this study, different levels of difficulty were identified by consensus in expert opinions, resulting from a single item in the CTA process. Focusing on each difficulty level, it would be interesting to investigate if each level contains similar automatic and non-automatic steps. Imagine an emergency situation, hypothetically this situation requires other automated steps or maybe less automated steps than a non-emergency situation. We recommend future studies to focus on CTA within different levels of difficulty of a single surgical procedure.

Although at first sight, the 4 C/ID model may seem to be primarily theoretical and not directly useful for translating expert performance into a training design, the integrated use of CTA and the 4 C/ID model facilitated incorporation of the information from CTA into the blueprint. Interestingly, the information that emerged from the error analysis was eminently suitable for the ‘just-in-time information’ component of the 4 C/ID model.

Finally, scenarios for medical skills training are somewhat different to scenarios used in other disciplines. In aviation training centres, for example, pilot trainees also learn how to adapt to different (flight) situations, but aviation scenarios enable trainees to train in a simulator that is identical to the airplane they will fly after training (Schwaitzberg et al. 2009; Rogers & Gaba 2011), whereas physicians perform skills in patients that are not identical but each present with their own unique anatomy and pathology. Consequently, scenario differentiation in medical skills training should focus not only on differences in pathology but also on differences between individual patient characteristics. In our analysis, several physical characteristics were identified, and the blueprint provides varying levels of difficulty for different features.

Discussion and conclusion

We have described how CTA integrated with the 4 C/ID model can be used to design simulator-based training for complex medical procedures, illustrating this with our experiences in designing a blueprint for simulator-based training using the integrated approach in a semi-structured group interview with expert urologists. The final blueprint that resulted can be used by simulator developers, software engineers and even simulator users, when the provided simulator comes with a training development kit.

Obviously, a blueprint for training is not an end in itself, but should lead to the realisation of a simulator, for which this article does not provide guidelines as it lies in the hands of simulator developers. Once developed, however, the simulator will have to be evaluated to determine whether it meets the requirements of the blueprint, and more importantly still, the effectiveness of the simulator has to evaluated. Several studies have proven that CTA-based training is substantially more effective than non-CTA based training (Velmahos et al. 2004; Feldon et al. 2010; Pugh et al. 2011), but further studies are needed to evaluate whether simulator-based training according to a design resulting from the integrated design approach of CTA and the 4 C/ID model is an effective tool to foster novices’ and residents’ competency development.

Finally, we wish to underline the importance of collaboration between medical experts, educational experts and simulator developers in the process of simulator-based training development. The type of blueprint we have described, resulting from CTA integrated with the 4 C/ID model, may hold the key to successful simulator development by enabling simulator designers to take on board educational principles as well as medical competences.

Acknowledgements

The authors thank the eight expert urologists from; the Catharina Hospital, Radboud University Nijmegen MC, University Medical Centre Utrecht, Maastricht University Medical Centre, Atrium Hospital, Isala Clinic, who participated in pilot and semi-structured interview. The authors also thank Mereke Gorsira for her editorial assistance.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.