The art and science of debriefing in simulation: Ideal and practice

Abstract

Objectives: Describing what simulation centre leaders see as the ideal debriefing for different simulator courses (medical vs. crisis resource management (CRM)-oriented). Describing the practice of debriefing based on interactions between instructors and training participants.

Methods: Study 1 – Electronic questionnaire on the relevance of different roles of the medical teacher for debriefing (facilitator, role model, information provider, assessor, planner, resource developer) sent to simulation centre leaders. Study 2 – Observation study using a paper-and-pencil tool to code interactions during debriefings in simulation courses for CRM for content (medical vs. CRM-oriented) and type (question vs. utterance).

Results: Study 1 – The different roles were seen as equally important for both course types with the exception of ‘information provider’ which was seen as more relevant for medical courses. Study 2 – There were different interaction patterns during debriefings: line – involving mostly the instructor and one course participant, triangle – instructor and two participants, fan – instructor and all participants in a dyadic form and net – all participants and the instructor with cross references.

Conclusion: What simulation centre heads think is important for the role mix of simulation instructors is (at least partly) not reflected in debriefing practice.

Introduction

Simulation plays an increasing role in medicine, especially in acute care and emergency settings (Bond et al. 2007; Fitch 2007). It allows for optimizing the interplay of people, tasks and organizational conditions (Small et al. 1999; Rall & Dieckmann 2005b; Dieckmann 2009). Simulation might be used to train, for example, the management of specific situations, like caring for trauma cases (Lee et al. 2003; Barsuk et al. 2005) or the handling of difficult airway situations (Good 2003; Rosenstock et al. 2004). Simulation might also be used for training principles of crisis resource management (CRM) (Reznek et al. 2003; Rall & Gaba 2005a) or generic cognitive abilities (Bond et al. 2004; St Pierre et al. 2008). Especially with mock-code training (Hunt et al. 2008) and mobile ‘in-situ’ training concepts (Rall et al. 2008) the use of simulators opens possibilities beyond the education and training of individuals towards organizational development and process optimization (Rall & Dieckmann 2005b; Rudolph et al. 2006; Small 2007). Regardless of the details of simulator usage, the post scenario debriefing is important to maximize learning and facilitating change on an individual and systematic level. Debriefing – as simulation as such – can be conceptualized as a ‘social practice’ during which people purposely interact with each other and the environment, reflecting on the common experience they made during the scenario (Johnson 2004; Rystedt & Lindwall 2004; Issenberg et al. 2005; Bligh & Bleakley 2006; Hodges 2006; Dieckmann et al. 2007a, 2007b; Molin Friis et al. under review).

Amongst the different course formats, settings, target groups and other relevant dimensions for describing simulation practice (Gaba 2004) one might distinguish at least between courses focused on the medial issues and those focused on issues of CRM (Rall & Dieckmann 2005a; Rall & Gaba 2005a, 2005b). Medical courses can be defined as courses focusing on medical expertise like intubation skills, difficult airway management, clinical reasoning etc. The CRM-oriented courses can be described as courses focusing on team training, human factors, performance shaping factors, etc.

Simulation instructors have many roles

The model of the 12 roles of a medical teacher by Harden and Crosby (2000) provides guidance for differentiating the methodological and attitude-related approaches where simulation instructors need to optimize simulation-based training. Harden and Cosby superimpose six role-concepts in their paper; we will apply these six roles to the person who ‘teaches’ specifically in debriefing during simulation-based courses:

1. Information provider: presents information in the lecture context both pre-planned and structured as well as spontaneously (e.g. during a debriefing).

2. Role model: exhibits a certain type of conduct both as health care professional and as teacher/educator.

3. Facilitator: helps participants learn, for example, by asking questions aimed for stimulating deeper reflection.

4. Assessor: judges the learning of the participants by comparing learning goals to their current status, but also assesses the curriculum and educational methods in their effectiveness.

5. Planner: oversees the learning goals for the participants, selects the methods to be used and sequences the material to be presented both in a curriculum and during a course.

6. Resource developer: creates learning material that can serve as a resource for the learner and also provides information on how to best use this material in study guides

Table

Practice points

Simulation instructors encompass different roles during simulation-based courses. The roles should be applied flexibly to the simulation situation. The roles should vary with target groups, learning objectives and partly with content. During debriefing the degree of involvement of individuals is important and should be controlled consciously.

Purpose and research questions

The purpose of this article is to point out areas to optimize debriefings in order to help improving patient safety in acute care settings. Two research questions guided our work:

• How relevant do simulation centre leaders around the world see the different roles by Harden and Crosby for an ideal debriefing in (a) medical courses and (b) courses that focus on concepts of CRM?

• How do simulation instructors interact with their participants during debriefings in terms of talking balance, topics addressed (medical vs. CRM-oriented) and what type of interaction do they apply (utterance vs. question)?

By relating the two views, it will be possible to derive strategies for optimizing debriefing practice based on empirical investigations which are relevant for the different simulation and debriefing settings.

Methods

The methods and results section of the current article are related to the study questions and are split into two parts. For neither study an ethics vote was sought as no identifying person-related data was collected. For the type of research presented here, the ethic committee does not need to be involved according to the rules in Denmark and Germany.

Study 1: Questionnaire on debriefing practice

Study design

Using an electronic questionnaire, simulation centre leaders were asked to describe the debriefing practice in their centre. Participation was voluntary, based on informed consent and data processing was anonymous. The study design included a two-step non-responder tracking. In the first step a reminder mail was sent to ‘non-responders’. In the second step the remaining ‘non-responders’ were asked for their reasons for not responding.

Study protocol and measurements

The questionnaire was programmed using Adobe Acrobat Professional® (Version 8.0.0) and could be filled in electronically or in a paper-and-pencil mode after printing. Filling in the questionnaire took approximately 30 min. In the current article we report the results related to the role model by Harden and Crosby. For this part an ordinal scale was used ranging from 0 to 10, with 10 indicating the highest relevance. While filling in the questionnaire, respondents were asked to summarize their answers for their favourite course in their centre for medical issues and for CRM-related topics, respectively.

Study setting and population

The questionnaire was sent to 89 simulation centre leaders in Europe, USA, Australia and Asia identified from the participants of the annual meeting of the Society in Europe for Simulation Applied to Medicine (SESAM) in 2007.

Data analysis

The ordinal data was analysed with the Statistical Package for the Social Sciences SPSS® (Version 15) and Microsoft Excel® (Version 2003). Wilcoxon's Test for dependent samples was used to investigate significant differences between central tendencies and McNemar's Test was used to investigate significant differences in variability of the data.

Study 2: Debriefing interaction protocols

Study design

This explorative study used a paper-based observation protocol to code the interaction between the simulation instructor and the participants during eight individual debriefings in one simulation course. Informed consent was provided by the participants for the observation.

Study protocol

The first author devised and tested the recording form for feasibility with a trial debriefing. In each of the eight debriefings the first author coded the interaction between the simulation instructor and the four participants using the recording form (Figure 1). Two different coding approaches were used, each for four of the eight debriefings observed. In the first approach the content and type of the interaction was coded as either medical or as CRM-related while at the same time the interaction was coded as question or utterance using four different codes. In the second approach the type of interaction was coded as either question or utterance using two different codes. In both cases, changes of speakers or clear changes of topics were coded by a mark in the graphical recording form (Figure 1). During the analysis the distinction between content and type in the first approach was dismissed as the result graphs otherwise would have been too cluttered. For the same reason the initiative for an interaction was not represented in the results graphs.

Figure 1. Coding scheme and interaction codes. The circles represent the persons (and roles) involved in the debriefing and the arrows represent directed interactions. On the right the two different variants of the coding system are presented. During coding, the directed connections were labelled with the corresponding code.

Study setting and population

The data was collected during a course for Crisis Resource Management for anesthesiologists in a German university-based simulation centre. Three instructors took turns in leading the debriefings in both subgroups (Figure 4). The course groups were heterogeneous in terms of participants’ professional experience, ranging from residency to consultant level. The simulation instructors were all experienced in running the course.

Individual participants enacted different roles in the scenarios in which they took part: anesthesiologist 1 (A1), anesthesiologist 2 (A2), surgical assistant (Surg) and observer (Obs). Observers watched the scenario from a control room. The instructor was coded as (Inst). The single persons have a unique code: for participants PT 1–8 and for the instructors Inst 1–3. PT 1–4 worked in one subgroup, PT 5–8 in a second subgroup. During the debriefing, participants stepped out of the roles they enacted during the preceding scenario and reflected upon them.

Data analysis

Data analysis consisted in counting the marks at the connecting lines between two persons. The qualitative distinctions according to topic and type were coded by different colours (Figure 4). The quantitative distinction was directly translated into the thickness of the connecting lines.

Results

Study 1: Debriefing questionnaire

Sample and response rate

Of the 89 questionnaires that were sent out, 22 were returned after the reminder mail resulting in an initial response rate of 25%. The non-responder tracking showed that 14 persons receiving the questionnaire were either not a centre head or that the centre was not yet in operation. Two persons reported not having received the questionnaire and three indicated that they did not return it due to time pressure. Consequently we reduced the reference number to 70 with a resulting valid response rate of 31%. Questionnaires were returned from eight European countries (17 of 50 questionnaires sent within Europe were returned), USA (3 of 10 sent to the USA were returned) and Australia (2 of 5 sent to Australia were returned). The remaining questionnaires were sent to Asia, but none were returned.

Instructors roles during the ideal debriefing

Respondents indicated how relevant they assumed each of the six roles defined by Harden and Crosby (2000) to be for an ideal debriefing in their view. The only role for which the reported relevance yielded a significant difference was that of the Information Provider (p ≤ 0.005) with higher indicated relevance for medical courses. The results are presented in Figures 2 and 3.

Figure 2. Netgraphs of the relevance of six roles of medical teachers (Harden & Crosby 2000) in the ideal debriefing, divided by course type. The midpoint of the net indicates 0, and the outer limit of each branch 10. The lines with triangles represent responses for medical courses and the lines with squares represent answers for CRM-related courses. Significant differences (p < 0.005) between medians are marked by an asterisk (*).

We analysed the variability of estimations. Figure 3 shows the distribution of perceived relevance across the roles for CRM-oriented courses in the upper part and those for medical courses in the lower part. The only statistically significant difference between the estimations was for information provider (p < 0.005).

Figure 3. Boxplots for the variability of estimations about role relevance by course type (CRM-oriented in the upper part and medical in the lower part). Outliers are indicated for values that deviate more than 1.5 times of the interquartile range from either side of the box.

Figure 4. Overview of the different debriefing interaction patterns. Line (debriefing 2, 3 and 5), triangle (debriefing 1 and 6), fan (debriefing 8) and star (debriefing 4 and 7). Note the different legend for debriefing 1–5 and for 6–8 due to the different coding of the interactions. Note also the difference between the persons and the roles that they enacted in different scenarios.

Study 2: Results from the debriefing interaction protocols

Figure 4 shows the results from all eight-coded debriefings. Each circle represents a person, with labels for the person and the role the person enacted during the preceding scenario. The lines of different thickness represent the interactions between the two connected persons—the thicker the line, the more interactions between the connected persons were counted.

A visual inspection of the debriefing graph indicates several interaction patterns:

• Line (debriefing 2, 3 and 5): Most interactions take place between the instructor and one participant, typically one of the anesthesiologists. Those interactions accounted for 73, 69 and 87% of all interactions in debriefing 2, 3 and 5 respectively.

• Triangle (debriefing 1 and 6): Most of the interactions occur between three persons, involving the instructor and the two most active participants with 88 and 96% in debriefing 1 and 6 respectively.

• Fan(debriefing 8): The instructor interacts with all participants in a dyadic way with little direct interactions between participants. The participants are involved with similar intensity (Obs 10%, A1 19%, A2 17% and Surg 15%).

• Star (debriefing 4 and 7): The participants interact both with the instructor and each other. The distribution of activity amongst all persons is more even than with any other pattern.

Instructors 1 and 3 each led a debriefing resulting in a triangle and a line. Instructor 2 led one line debriefing, two stars and one fan.

Figure 5 shows the percentages with which the instructors asked questions (interpretable as facilitation-oriented interactions) as opposed to making utterances (interpretable as instruction-oriented interactions). The basis for the percentages was all interactions initiated by the instructor in the specific debriefing. During the debriefings the percentages of questions varied between 23 and 57%.

Figure 5. Distribution of question and utterances during debriefings (note that debriefings are sorted by instructor).

Interaction contents were coded for five debriefings as either medical or CRM-related. In this course for CRM, the amount of CRM-related interactions ranged between debriefings (17, 28, 35, 48 and 55% of all interactions).

In a next step the activities during the debriefing of the persons in their different roles were analysed across the entire course. Figures 6 and 7 show that during debriefings, the activity of seven participants varied strongly with the role that they enacted during the preceding scenario. In general, participants were more active in the debriefing, if they enacted an anesthesiologists’ role – which, in general, is an active role in the scenario as well. Participant 5 (Figure 7) is an exception to that rule and might be classified as a ‘silent participant’. In all debriefings in which he was involved, he showed a similar low activity (being involved in 2–10% of the interactions during the debriefing), regardless of the different roles he enacted.

Figure 6. Involvement of participants (PT 1 to PT 4) during debriefings (1, 2, 7 and 8) by the role they enacted in the preceding scenario. The lines represent a person and the labels describe which role the participant enacted during the preceding scenario.

Figures 6 and 7 also show that in five of the eight debriefings the individual instructors were involved in more interactions than any other person. Instructors 1 and 3 showed that pattern in both their debriefings, instructor 2 in one of the four debriefings he led. In debriefing 3, 4 and 7 he was involved in fewer interactions than the primary anesthesiologist.

Figure 7. Involvement of participants (PT 5 to PT 8) during debriefings (3–6) by the role they enacted in the preceding scenario. The lines represent a person and the labels describe which role the participant enacted during the preceding scenario.

Discussion

In the following, we discuss the results from both studies – first independently and then related to each other, deriving implications for simulation practice, instructor development and future research. Finally, we discuss our methodological approaches.

Study 1: Debriefing questionnaire

The participating centre heads see the required roles for the ideal debriefing largely as comparable across different types of courses, except for ‘information provider’. Although not statistically significant, our results also point towards a larger variation in the ‘ideal’ role mix needed for CRM-related courses. The greater variability across the roles for the CRM-oriented courses might indicate a less clear picture about the ‘working methods’ for these topics. While there is a long tradition for teaching medical topics, it seems as if simulation instructors still search for a working mode in CRM-oriented courses. Neither in basic medical education nor in continued specialist training structured possibilities are offered, to build the necessary CRM-related knowledge, facilitation skills and human factor-oriented attitude.

One goal of instructor development could be an increased flexibility in adapting and using the different roles in a context-specific way. For example, faculty development could (a) address competencies to provide more information in CRM-oriented courses and (b) help faculty to rely on facilitation also when teaching medical issues. The goal is flexibility in methods, contents and roles – matched with the requirements of the course and the participants. The methodology introduced in this article might be used to give feedback to instructors, also within a simulator team. They can help instructors reflecting in and on action (Schön 1987; Dieckmann et al. 2007a).

Study 2: Discussion of the interaction protocols

Some simulation instructors emphasize that they prefer to let participants do most of the talking and that their primary role during debriefing is to facilitate this process by asking questions (Molin Friis et al. under review; Dieckmann 2005). This ideal is formulated in the medical literature, as well as other domains (Rall et al. 2000; Dismukes & Smith 2000; Mort & Donahue 2004). The debriefing interaction coding paints a different picture of simulation practice. Many interactions are based on utterances (often explanations of medical issues) by instructors, who are involved in most interactions.

Different debriefing styles?

The interaction dynamics during the different debriefings varied with the instructors leading the debriefing. For example, the participants in group 1 (PT 1–4) interacted strongly with each other during debriefing 7 and 8 which was led by instructor 2. The same participants interacted in a more dyadic way (line and triangle) with instructor 1, during debriefing 1 and 2. Figures 6 and 7 show that the instructor was the single most active person in 5 of 8 debriefings. Certainly this could be due to many factors, not least the features of the scenario, the knowledge of the participants or group dynamical factors. However, our results support an assumption that the role and behaviour (verbal as well as non-verbal) of the instructor influences the debriefing dynamics.

Activity by participants

Except for one person, the activity during debriefing was closely related to the role which participants enacted during the scenario. The most active persons during the scenario were also the most active persons during the debriefing. It might, at times, be beneficial to specifically involve participants who were less active during the scenario and to explore their motives and mental models. A group might miss learning opportunities when discussing, for example, leadership and followership with a strong focus on the (active) leader, letting the (silent) followers get away with burdening the leader even more, showing maybe (too) little responsibility or showing little assertiveness. Debriefing might offer a chance to break the habit of diverting the attention according to the participants’ degree of activity – especially as the less active participants could hold the greatest potential for improving patient safety. Instructors need high diagnostic skills to differentiate between participants who do not want to be more active and participants who do not get a chance to be more active. Instructors also need good facilitation skills to involve such ‘silent participants’, without pushing them too much.

Implications for simulation practice

Our studies have implications for simulation practice as well as for instructor development. The questionnaire study shows variations in the perceived ideal role mixture that is strived for. The observation study suggests getting an improved understanding of how the single roles are put into practice (Salas et al. 1997). A closer look at what instructors do in practice and feedback about this practice might help in optimizing debriefing. The method suggested above might be used in this sense, as well as video-based debriefings of debriefings.

Methodological considerations

The two studies have some methodological limitations. The low response rate of 31% within the questionnaire study limits the generalizability of the results. In addition the sample was biased towards Europe. Our experience with instructor courses (Dieckmann & Rall 2008; Dieckmann et al. 2008), however, supports the face value of the results. Many instructors struggle with a clear role definition.

The sample size in the observation study was small and only a single observer coded the interactions. The psychometric qualities of the instrument were not analysed and the interaction coding does not reflect the time frame involved as the duration of interactions were not coded. The study had a descriptive character and the data is sufficient to hypothesize that there are different debriefing styles and interaction patterns. Increasing the sample size might show more patterns or give some ideas about the distribution of the different styles.

Future research

The role model by Harden and Crosby was helpful to structure investigation. In the future the roles and their application could be defined more specifically to simulation-based teaching. In terms of the behaviours related to the roles, future research should clarify whether more interaction patterns during debriefing can be described and how and with what effect they are used. In addition, it should be analysed which interaction style is best suited for different target groups and topics and how instructors might best be prepared to apply the roles and interaction styles flexibly. Further, concepts need to be developed to include observers in a goal-oriented fashion. Their overall involvement was rather low during the debriefings indicating the need to develop ideas of how to include them in the reflections.

Our results emphasize the need to understand debriefing practice in more detail. We do not yet know enough about which processes facilitate or which hinder a goal-oriented conduction of debriefings and which role diversity is needed to use simulations and debriefings in a goal-oriented way.

Conclusions

In conclusion, we found little differences in the formulated ideal role mix for simulation instructors in medical and CRM-related simulation-based courses. Only the role of the information provider was significantly seen as more relevant for medical courses. The variability for CRM-related courses seemed to be larger, despite not reaching statistical significance. For the debriefing coding we found different interaction patterns between instructors and course participants. Instructors were involved strongly in the debriefing, being involved in most of the interactions, making more utterances than asking questions and the topics in the CRM-related courses were more focused on medical issues. Debriefing practice might thus be, at times, different from the ideal that is strived for.

Acknowledgements

We thank the participants in our studies for working with us. The data collection for the debriefing interaction was conducted in the Heidelberberger Anaesthesie and Notfall-Simulator (HANS) at the University of Heidelberg, Germany. We thank the HANS team for the support. We also thank the reviewers for helping us to improve the original version of this manuscript.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Peter Dieckmann

PETER DIECKMANN is a psychologist and holds a PhD on using simulations in education, training and research. His Primary research interests are human factors and the use of simulation.

Susanne Molin Friis

SUSANNE MOLIN FRIIS is a registered Nurse Anaesthetist, has conducted research on optimizing the use of simulation in education. She is also working in the care of children with chronic pain.

Anne Lippert

ANNE LIPPERT and DORIS ØSTERGAARD, both MD, and Associate Professors with the University of Copenhagen, Denmark, are leading the Danish Institute for Medical Simulation (DIMS).

Doris Østergaard

ANNE LIPPERT and DORIS ØSTERGAARD, both MD, and Associate Professors with the University of Copenhagen, Denmark, are leading the Danish Institute for Medical Simulation (DIMS).