What is the impact of structured resuscitation training on healthcare practitioners, their clients and the wider service? A BEME systematic review: BEME Guide No. 20

Abstract

A large number of resuscitation training courses (structured resuscitation training programmes (SRT)) take place in many countries in the world on a regular basis. This review aimed to determine whether after attending SRT programmes, the participants have a sustained retention of resuscitation knowledge and skills after their initial acquisition and whether there is an improvement in outcome for patients and/or their healthcare organisation after the institution of an SRT programme. All research designs were included, and the reported resuscitation training had to have been delivered in a predefined structured manner over a finite period of time. Data was extracted from the 105 eligible articles and research outcomes were assimilated in tabular form with qualitative synthesis of the findings to produce a narrative summary. Findings of the review were: SRTs result in an improvement in knowledge and skills in those who attend them, deterioration in skills and, to a lesser extent, knowledge is highly likely as early as three months following SRTs, booster or refresher sessions may improve an individual's ability to retain resuscitation skills after initial training and the instigation of resuscitation training in a healthcare institution significantly improves clinical management of resuscitations and patient outcome (including survival) after resuscitation attempts.

Background and context

SRT programmes

SRT programmes in the form of resuscitation courses are used worldwide to attempt to optimise standards of clinical practice in resuscitation management, minimise error and decrease patient morbidity and mortality. Most often, SRT programmes are evaluated at a local level in terms of participant's enjoyment and engagement. The most important question, however, must be whether these programmes are effective. To date, there has been no cross disciplinary systematic review investigating whether this is the case.

SRT programmes differ in their content and target audience (e.g. the Adult Life Support, Advanced Paediatric Life Support and Neonatal Life Support). However, many aspects are similar, such as the delivery of lectures, use of simulation (often low fidelity) and assessment. Resuscitation governing bodies in different countries (e.g. the Resuscitation Council in the UK) have attempted to standardise each type of course. Courses generally take place over one day and on each, candidates are assessed in relation to their knowledge and skills in resuscitation. If successful, candidates are issued a certificate, which is usually valid for four years.

The Resuscitation Council (UK; 2010) oversees SRT for many adult and paediatric (including neonatal) specialities in the United Kingdom. There is a European Resuscitation Council, which coordinates and oversees SRT programmes in Europe and an International Liaison Committee on Resuscitation whose aim is to provide a forum for liaison between principal resuscitation organisations worldwide. A central feature of these SRT courses is that attendees are from a variety of backgrounds (medical, nursing, etc.), which helps to replicate the multidisciplinary involvement in resuscitations (Resuscitation UK 2010).

Table

Practice points

Ensure all staff in a healthcare organisation attends an SRT programme pertaining to their speciality. Ensure that any reassessment of staff clinical practice skills takes place in an authentic, as well as a simulated, clinical learning context. Assess staff for competency in resuscitation skills three to six months after the SRT. Provide regular booster or refresher sessions three to six months after the SRT.

Some training programmes are mandatory requirements for healthcare professionals, and are thus funded by employers as part of a professional update. Others, however, are attended voluntarily by healthcare professionals who want to further their clinical skills. In the latter case, candidates usually pay an attendance fee, and the course must often be attended in the candidates own time, which may potentially result in barriers to learning. For the purposes of this review, courses, whether mandatory or not, were included as long as they fulfilled the definition of an SRT programme as mentioned earlier in the text.

Because of the financial constraints facing most UK National Health Service organisations, especially in training budgets, organisations are developing their own in-house advanced, immediate and neonatal life support courses. Despite this resolving a problem in the short term (the training and updating of healthcare workers), it may, unfortunately, have implications regarding the quality and standardisation of resuscitation training provision (Resuscitation UK 2010).

An SRT programme for the purposes of this review was defined as a resuscitation training curriculum (not necessarily accredited) delivered to a group of learners over any reported finite period of time in a predefined, structured manner. SRT programmes have been developed around the world to train healthcare professionals in adult, paediatric and neonatal resuscitation. A healthcare professional for this review is defined as an individual who as a result of their role, has contact with patients and has direct responsibility for their clinical care.

Learning and SRT programmes

Learning can be defined as changes in knowledge, understanding and skills (Brown et al. 1997). This can occur following organised training similar to that which takes place during SRT programmes or through more casual self-directed activities such as browsing the literature. An SRT course aims to equip the participant with the knowledge and skills to perform optimal resuscitation in their clinical work place. Knowledge is enhanced by the use of lectures and skills by repeated exposure to simulation scenarios. Overall, the SRT ‘experience’ takes the candidate through Kolb's learning cycle: they build on their prior knowledge by learning new skills and after practicing these new skills they reflect on their ‘action’, resulting in behavioural change (learning; Kolb 1984).

Simulation is specifically used in SRTs and incorporates many of the attributes that have been reported to facilitate learning. These are: appropriate use of feedback, engagement in repetitive practice, the simulator being embedded in a controlled environment and permitting individual learning and learning outcomes being clearly defined. It is also important that the simulator being used is a valid (high-fidelity) approximation of clinical practice (Issenberg et al. 2005).

Tight (2002) suggests that although adults have considerable experience of education, for some, this will have been largely confined to childhood. The concept of andragogy encompasses the idea of how adults learn. This places a greater emphasis on what the learner is doing (Reece & Walker 2000), as opposed to pedagogy, which, as it highlights the teacher dominating and leading the session completely, is used more in the teaching of children. Adults have reached a stage of independence and are, therefore, successfully able to undertake self-directed study (Knowles 1984). Prior to their attendance on an SRT course, learners are encouraged to read and digest the manuals to assist with their learning experience on the day of the training. Prior knowledge and exposure also seem to be key factors influencing learning (Marton et al. 1997). All candidates attending SRTs have had either, as undergraduates, some prior theoretical exposure, or as postgraduates, practical exposure to resuscitation.

Most SRT courses utilise a visual and kinaesthetic approach to learning enhanced by a behaviourist approach to learning based upon repeated practice, where students learn mainly through association. The SRT courses are designed to give candidates the skills to provide effective resuscitation, partially through an approach of repetitive practice during the training. The principle of the educator acting as the facilitator (Dunn 2000) stems from a belief that human beings have a natural eagerness to learn, thus learners become more empowered to take responsibility for their own learning when facilitated to do so by an expert. On SRT courses, candidates are encouraged by instructors to share their knowledge and experiences with their peers during the various simulation scenarios. Burns (2000) suggests that the majority of ‘competency-based’ training is founded upon the theory of reinforcement to strengthen behaviour. It works on the premise that the learner will repeat the desired behaviour if positive reinforcement follows the behaviour. This is used by faculty on SRT programmes repeatedly: candidates are frequently praised and given positive feedback when they perceive that a candidate has shown evidence of knowledge acquisition or improved their skills.

Knowledge and skill acquisition and retention

Most individuals can pass resuscitation courses by achieving a certain mark in a written examination together with demonstrating ability to carry out predetermined tasks on a simulator. The degree of knowledge and skill acquisition may vary (Wynne 1986). Furthermore, the assessment of the magnitude of any transfer of knowledge and skills into the clinical setting may be difficult owing to ethical difficulties observing participants in an acute real-life resuscitation scenario and the lack of any validated measures to do so.

In the context of SRTs, behavioural change (achievement of resuscitation competency) may not be permanent: it is possible that learning can be exhibited in the assessment process following an SRT but there may be factors other than the SRT, which are responsible for the medium or long-term sustenance of the learning (maintenance of competency; McGaghie et al. 2010). One of these may be combining simulation-based medical education as on an SRT with deliberate practice – thus ensuring mastery at a particular skill (Ericsson KA 2006; McGaghie et al. 2011). However, those individuals who are not frequently exposed to resuscitation situations after an SRT may still lose skills and/or confidence quickly. This problem is illustrated by David and Prior-Willeard (1993) who assert that survival to hospital discharge depends greatly upon the initial treatment a patient receives during resuscitation, yet they suggest that, based on a clinical assessment of doctors about to take their MRCP exam, the basic life support skills of many doctors, nurses and medical students (who have previously received resuscitation training) is of poor quality.

Review aims

To determine:

1. Whether after attending SRT programmes, the participants have a sustained retention of resuscitation knowledge and skills after their initial acquisition.

2. Whether participants attending SRT programmes exhibit behavioural change in the work setting.

3. Whether there is an impact on outcome for patients and/or their healthcare organisation after the institution of an SRT programme.

Review methodology

Group formation

A systematic review group was formed of staff from different disciplines working at the Liverpool Women's Hospital Foundation Trust. All group members (two consultant neonatologists (B.N.S. and C.D.), an advanced neonatal nurse practitioner (C.M.) and a hospital librarian (S.M.)) attended a one-day training course on how to conduct a BEME review. After this, individual roles were defined within the group and a timeline set for completion of the study.

Search strategies

A search strategy was developed by the group led by C.M. (see Appendix 1 for the search terms). The following databases were searched by S.M.: Medline, CINHAL, Pub Med and the Cochrane Database of systematic reviews. This search was confined to the English language literature as there is no evidence of a systematic bias from the use of language restrictions in systematic reviews (Morrison A et al. 2009) and to avoid the long potential time delay that obtaining translations may have entailed. Two search updates were performed over the two years of conducting the review to allow for the inclusion of new publications.

All articles that described an SRT, as previously defined, were identified by the presence of one or more of the key words from Appendix 1 in the title.

The majority of reference titles obtained clearly had no relevance to the review (for example, those related to basic science or animal work). In order to streamline the process, the decision was taken for one group member (C.D.) to discard those which unambiguously had no relevance. The abstracts of the remaining articles (where the article was of relevance or where there was uncertainty from just reading the title) were then distributed throughout the group. Each abstract was initially read by one of the group members who then decided on whether the article was likely to fulfil the inclusion criteria, and if it did, allocated a provisional Kirkpatrick (1994) level (see details in Box 1).

Box 1. Possible levels of outcome of articles (Modified from Kirkpatrick, 1994)

The Kirkpatrick system below was modified from Kirkpatrick's 1994 model of outcomes at four levels. Articles were allocated a Kirkpatrick level according to the outcomes described – some articles described outcomes relating to more than one level in which case they were included in the analysis for each outcome level.

Kirkpatrick Level 1 Reaction to learning experience

Evidence of learners’ views on the overall learning experience and its inter-professional nature including the training programme, rather than any specific learning outcomes.

Kirkpatrick Level 2a Modification of attitudes and perceptions

Evidence of documented changes in reciprocal attitudes or perceptions between participant groups and possible changes in perception or attitude towards the value and/or use of team approaches to caring for a specific client group.

Kirkpatrick Level 2b Acquisition of knowledge and skills

Evidence of knowledge and skills acquisition immediately following completion of a SRT.

Kirpatrick Level 2c

Evidence of the retention of knowledge and/or skills over a period of time after the SRT.

Kirkpatrick Level 3 Behavioural change

Evidence of transfer of learning to clinical practice.

Kirkpatrick Level 4a Change in organisational practice

Evidence of changes within the organisational practice and delivery of care after the SRT.

Kirkpatrick Level 4b Benefits to patients/clients, families and communities

Evidence of documented impacts in the health or well being of patients/clients, families and communities after the SRT.

All abstracts were subsequently reviewed blindly by C.M. in order to confirm that the provisional Kirkpatrick level had been appropriately assigned and that the article should be included in the review, pending receipt of the full article, or otherwise. If there was disparity between the coder's Kirkpatrick level, and/or disagreement whether the article should be included, further discussion took place between the two coders in order to agree these issues by consensus.

The full article of each included study was then requested. When received, each article was categorised according to discipline (adult, paediatric and neonatal) and assigned a unique reference number. Each article was read by C.M., and the provisional Kirkpatrick level was again reviewed and confirmed or changed accordingly. The full text of all the articles identified for provisional inclusion together with allocated Kirkpatrick levels were then distributed to a second reviewer in the group for confirmation of the Kirkpatrick level allocation and final decision regarding inclusion.

The bibliographies of all articles to be included in the review were also searched to capture any further relevant articles which were categorised and coded as mentioned earlier.

Quality assessment and final inclusion of articles

Initially, articles were assessed independently by two members of the group (C.D. and C.M.) and scored in relation to two different quality assessments related to level of evidence presented and clarity of methodology and results reported (Appendix 2B and C). There were few randomised trials (7), but the vast majority of studies were cohort studies reporting data of a similar evidence level. All studies had a clarity of results and methodology reporting sufficient to merit inclusion: as a result, it became evident that neither ‘quality’ assessment could be used to define appropriate articles for inclusion. It was, therefore, decided to include articles using all research designs, and a number of criteria for inclusion based on a minimum requirement for results reporting were agreed upon as follows:

• The reported resuscitation training had to have been delivered in a predefined structured manner over a reported finite period of time.

• The participants had to be healthcare practitioners (including preregistration and postregistration, undergraduate and postgraduate).

• Participants had to be assessed by a marked or scored assessment at the end of the training, and the result of this assessment had to be stated.

• If participants were assessed some time after the training, the immediate posttraining assessment result also had to be stated.

• Where there was an improvement in any outcome for patients and/or their healthcare organisation, the magnitude and type of the effect had to be stated.

Any lack of clarity in an article in relation to the above-mentioned criteria was discussed and final agreement of the articles inclusion or exclusion was reached by consensus.

The search process yielded 3781 article titles. Of these, 425 abstracts were reviewed and 196 full articles obtained. Of these, 105 were included as there were 11 duplicate publications identified and 80 did not completely fulfil the results reporting inclusion criteria (Figure 1)

Figure 1. Flow chart of the process for final inclusion of papers in the review.

Coding and analysis

An initial coding sheet was designed by the group and produced in an Access Data Base electronic format. To pilot this, five of the selected articles were coded by two independent coders (C.D. and C.M.), and the sheet was redesigned to exclude any ambiguities. Following this, 20 articles were coded by the same two coders. It was felt that there were too many fields present with irrelevant information in the electronic format, so a simplified (paper) coding sheet was then produced (Appendix 2A). All articles were subsequently coded independently by C.M. and B.N.S., and the results were periodically reviewed to ensure that they were in agreement prior to data being inputted. Very few differences in coding occurred – these were discussed by the two coders in question and agreement by consensus reached.

For ease of reference, the relevant results were displayed on a final coding sheet in tabular form for each Kirkpatrick group for adult, paediatric and neonatal resuscitation separately using a Microsoft word document (Appendix 3).

Data relating to Kirkpatrick level 1 (satisfaction with the SRT) have not been analysed or reported in this study as, although satisfaction with teaching may affect learning, it was not directly relevant to the aims of the review.

Heterogeneity of research designs, educational interventions and outcome measures precluded meta-analysis of quantitative data (for Kirkpatrick level 2c studies, each assessment outcome used a different marking system (Tables A4–A6) and for level 4 studies outcomes were different in many studies (Tables A7–A9)). Qualitative data synthesis of research methods and outcomes was carried out by two members of the group (C.M. and B.S.) independently identifying themes from the interventions and outcomes from studies at each Kirkpatrick level. C.M. and B.S. then discussed these themes and agreed by consensus the key themes that had emerged. The narrative that emerged described the key themes and overall outcomes within groups of studies. This was discussed and refined by the review team who agreed the final narrative findings given below.

Findings

The findings will be presented for each of Kirkpatrick levels 2, 3 and 4, subdivided into adult, paediatric and neonatal resuscitation data. This allows the reader to view data that exists for their own discipline. A description of the studies for each level and each discipline, linked to the tables in Appendix 3 that display the full relevant data for each level, is followed by a description of the themes, which emerged from the data for each Kirkpatrick level.

Kirkpatrick 2A and 2B: Modification of attitudes and perceptions (2A) and acquisition of knowledge and skills (2B)

Neonates (Appendix 3, Table A1)

There were three studies in this category (Ergenekon et al. 2000; Trevisanuto et al. 2005; Cavaleiro et al. 2009). The nature of the SRT offered was a mixture of lectures and simulation, and one study reported an accredited training programme. All three tested knowledge at the end of the training by multiple choice questionnaire (MCQ), and all three demonstrated statistically significant improved knowledge at the end of training (p more significant than <0.01 in all cases). None reported testing skills at the end of training; however, one assessed confidence (Kirkpatrick level 2A) in resuscitation revealing an improvement (Ergenekon et al. 2000). One subgroup of students in one study (Cavaleiro et al. 2009), using self-study alone, showed no improvement in knowledge compared to those receiving a lecture.

Paediatrics (Appendix 3, Table A2)

There were five articles in this category. The nature of the SRT that where stated included lectures and simulation (three reporting an accredited training programme). Three studies tested knowledge at the end of training (two with an MCQ and one with written case scenarios; Quan et al. 2001; Waisman et al. 2002; Gerard et al. 2006). In one study, there was a statistically significant improvement in knowledge (Waisman et al. 2002) and in another one there was no change (Quan et al. 2001). In the third study, knowledge change was not stated (Gerard et al. 2006). Three studies reported testing skills at the end via simulation with or without video, two (Quan et al. 2001; Donoghue et al. 2009) reporting statistically significant improvement in skills (one not reporting outcomes; Gerard et al. 2006). Three studies (Quan et al. 2001; Dobson et al. 2003; Gerard et al. 2006) assessed confidence (Kirkpatrick 2A) by questionnaire and reported statistically significant improvements in confidence score after training. A subgroup of participants in one study who received high fidelity simulation training had improved skills on testing compared to a low fidelity training group (Donoghue et al. 2009).

Adults (Appendix 3, Table A3)

There were 23 articles in this category. The nature of the SRT in most cases included simulation with mannequins combined with lectures (14 used an accredited training programme). Eleven studies reported testing knowledge at the end of training (10 with an MCQ and one with short answer questions; Girdley et al. 1993; Ali et al. 1995; Ali et al. 1996a,b; Ali et al. 1998; Azcona et al. 2002; Tippet 2004; Owen et al. 2006; Aboutanos et al. 2007; Dauphin et al. 2007; Hoadley 2009; Jenson et al. 2009). All of these studies reported statistically significant improved exam scores at the end of the training compared to before training. Eighteen studies reported testing skills at the end of training using simulation mannequins (Ali et al. 1995; Ali et al. 1996a,b; Greig et al. 1996; Bilger et al. 1997; Ali et al. 1998; Marshall et al. 2001; Azcona et al. 2002; Mayo et al. 2004; Cimrin et al. 2005; Devita et al. 2005; Monsieurs et al., 2005; Wayne et al. 2005; Dunning et al. 2006; Owen et al. 2006; Rosenthal et al. 2006; Aboutanos et al. 2007; Hoadley 2009; Jenson et al. 2009). In three studies, the testing took the form of an objective structured clinical examination and in one only telephone skills conveying the severity of the collapse requiring resuscitation to other professionals were tested. Seven studies reported statistically significant improvements in postcourse skill scores compared to those of precourse (Ali et al. 1996a,b; Ali et al. 1998; Marshall et al. 2001; Cimrin et al. 2005; Devita et al. 2005; Dunning et al. 2006; Rosenthal et al. 2006). In addition, four studies reported skill improvement but with no p value reported to indicate whether this was statistically significant (Greig et al. 1996; Azcona et al. 2002; Owen et al. 2006; Jenson et al. 2009), and six studies reported improved scores in skills in a group receiving the training compared with a control group who did not (four of these were randomised controlled trials; Ali et al. 1995; Ali et al. 1996a,b; Bilger et al. 1997; Ali et al. 1998; Mayo et al. 2004; Wayne et al. 2005). Three studies did not report the levels of skill before and after the training despite reporting testing it (Monsieurs et al. 2005; Aboutanos et al. 2007; Hoadley 2009).

Summary of Kirkpatrick 2A and 2B studies

The overwhelming message from these studies is that both knowledge and skills are significantly improved after SRT compared with pretraining levels. This has been confirmed both when individuals are tested pretraining and posttraining and also, in the context of randomised controlled trials, when groups of participants who have been trained are compared with control groups who have not. The assessment of knowledge and skills levels and changes in these were reported using scoring systems, which were unique to each study in most cases thus precluding meta-analysis. There is a suggestion from one study that high fidelity simulation compared to low fidelity may be more effective in improving skills (Donoghue et al. 2009), and that attending a training session compared to self-study might be more effective in improving knowledge (Cavaleiro et al. 2009). There were no clear differences in outcomes between accredited and nonaccredited training programmes. Where reported, confidence at performing resuscitation tasks is universally improved in participants who have undertaken SRT. There is no evidence available to indicate whether the improvement in knowledge and/or skills after SRT results in improved clinical performance immediately after SRT.

Kirkpatrick 2C – Retention of knowledge and skills over a period of time after SRT

Neonates (Appendix 3, Table A4)

There were eight studies in this category. In those studies that stated the nature of the training, all used simulation with mannequins and most used lectures (four described accredited programmes). The number of participants followed up after SRT in the studies ranged from 6 to 166. The period of follow-up ranged from 6 weeks to 12 months. All studies reported knowledge retesting at follow-up with an MCQ and five reported skill testing using mannequins. Four studies reported a decrease in knowledge (Kaczorowski et al. 1998; Curran et al. 2004; Trevisanuto et al. 2005; Duran et al. 2008a,b) and four reported that knowledge did not change at follow-up (Dunn et al. 1992; Levitt et al.1996; West 2000; Skidmore & Urquhart 2001; only two of these, however, reported no statistically significant difference). There did not seem to be any difference with respect to the nature of the training between those studies where knowledge decreased and those where it was maintained. In all but one study, which tested skills (Dunn et al. 1992; Kaczorowski et al. 1998; West 2000; Skidmore & Urquhart 2001; Curran et al. 2004), a significant decrease in skills at follow-up testing occurred. The study where skills were maintained was small (six participants) and skills were tested only six weeks after the training (West 2000).

Paediatric (Appendix 3, Table A5)

There were five articles in this category (two reporting accredited training programmes). The nature of training was variable: in two studies this was unknown, in one it was self-study and in others it was lectures and simulation with mannequins. The period of follow-up testing ranged from 2 to 21 months. All studies reported knowledge testing (three with an MCQ), three demonstrating a decrease in knowledge at follow-up (Spaite et al. 2000; Su et al. 2000; Wolfram et al. 2003) and one demonstrating no change (assessment was by telephone questionnaire and no p value was reported; (Durojaiye and O’Meara 2002). Two reported testing skills at follow-up but did not report any assessment data (Nadel et al. 2000; Su et al. 2000).

Adults (Appendix 3, Table A6)

There were 39 articles in this category. The nature of the training was varied and included lectures, simulation with mannequins and videos (in 18, this was part of an accredited programme). The training was delivered over a period of time ranging from 15 minutes to two-and-a-half days. The period between the training and testing at follow-up ranged from 1 to 60 months. Twenty-seven studies reported testing knowledge at a later date (20 with an MCQ, the others with a variety of written assessments). Sixteen of these studies reported significant deterioration in knowledge at follow-up testing (Gass & Curry 1983; Fossel et al. 1983; Stross 1983; Curry & Gas 1987; Ali et al. 1996a,b; Broomfield 1996; O'steen et al. 1996; Leith 1997; Wenzel et al. 1997; Blumenfeld et al. 1998; Young & King 2000; Ali et al. 2002; Azcona et al. 2002; Boonmak et al. 2004; Tippett 2004; Semeraro et al. 2006) and seven reported no deterioration in knowledge at follow-up testing (Stross 1983; Coleman et al. 1991; O’Donnell & Skinner 1993; Holden et al. 1996; Hammond et al. 2000; Aboutanos et al. 2007; Cooper et al. 2007). With respect to the nature of the training, those groups who received a refresher or booster session (in two randomised trials) maintained knowledge better than those who did not (Stross 1983; O’Donnell & Skinner 1993). There were no other clear differences between those retaining and those deteriorating in their knowledge with respect to the nature of their training. Twenty-eight studies reported a deterioration in skills at follow-up testing (Gass & Curry 1983; Fossel et al. 1983; Stross 1983; Mancini & Kaye 1985; Curry & Gas 1987; Bradley et al. 1988; Plank & Steinke 1989; Yakel 1989; Ten Eyck 1993; Fabius et al. 1994; McKee et al. 1994; Ali et al. 1996a,b; Broomfield 1996; Erickson et al. 1996; Holden et al. 1996; O'steen et al. 1996; Leith 1997; Wenzel et al. 1997; Hammond et al. 2000; Kovacs et al. 2000; Young & King 2000; Ali et al. 2002; Heidenreich et al. 2004; Semeraro et al. 2006; Beckers et al. 2007; Cooper et al. 2007; Spooner et al. 2007; Smith et al. 2008), whereas only nine reported maintenance of skills at follow-up (Coleman et al. 1991; O’Donnell & Skinner 1993; McKee et al. 1994; Kovacs et al. 2000; Ander et al. 2004; Boonmak et al. 2004; Heidenreich et al. 2004; De Regge et al. 2006; Wayne et al. 2006). In the studies where skills were maintained, two (Coleman et al. 1991; Boonmak et al. 2004) reported retesting only a short time period after the SRT (three months), three studies (Ander et al. 2004; Heidenreich et al. 2004; De Regge et al. 2006) reported maintenance of isolated discrete skills within a resuscitation scenario (other skills having deteriorated) and three (O’Donnell & Skinner 1993; Kovacs et al. 2000; Wayne et al. 2006) had as part of their SRT, repeated testing and refresher sessions (all in the context of randomised trials).

Summary of findings from Kirkpatrick 2C studies

It seems that knowledge can be maintained for several months after SRT; however, there is no specific aspect of training that can be identified, which facilitates this. There were no clear differences in outcomes between accredited and nonaccredited training programmes. Skills generally deteriorate from at least three months after SRT. Factors, which may prevent this occurring are, providing refresher or booster sessions after training and possibly identifying discrete actions to be assessed within simulation during training and at follow-up. Skills were all assessed at follow-up using simulation in mannequins and not in real clinical situations making it impossible to know whether the deterioration or maintenance of skills identified was being reflected in clinical practice. Any association with behavioural change and a change in clinical performance in participants in those studies where their retention of skills and/or knowledge was reported is, therefore, unknown. In the context of this review, Kirkpatrick level 3, therefore, relates to retention of knowledge and skills and their application in a simulated environment. There is a need for work to be carried out to explore any association between behavioural change as evidenced by a simulated environment and behavioural change in a ‘real-life’ setting. To our knowledge, investigating and identifying behavioural change in individuals in such a setting has not been systematically investigated.

Kirkpatrick 3: Evidence of transfer of learning to clinical practice

There were no studies in this category.

Kirkpatrick 4 – Evidence of benefit to patients, families and communities after SRT

Neonates (Appendix 3, Table A7)

There were seven studies in this category all following accredited programmes, which included lectures and simulation training. These studies reported outcomes following the introduction of SRT programmes within individual institutions, often over a period of years. Four studies reported a significant impact on patient outcome, (Zhu et al. 1997; Patel et al. 2001; Patel and Piotrowski 2002; Duran et al. 2008a,b) three reporting an improved resuscitation (Apgar) score in babies and one reporting a reduction in neonatal mortality (Zhu et al. 1997). Two studies reported improvement in clinical management with respect to the organisation of clinical resuscitations and interventions during resuscitation (improvement in delivery room preparation and assessment of the baby (Ryan et al. 1999) and reduction in hypothermia and inappropriate use of the drug Naloxone (Singh et al. 2006)).

Paediatrics (Appendix 3, Table A8)

There were two studies in this category. Neither followed an accredited training programme. One study involved weekly simulation scenarios and one involved supervised practice. Neither of these studies reported any impact on patient outcome. One study reported an improvement in clinical management (Losek et al. 1994) and one reported deterioration in clinical management (Lo et al. 2009). The latter study had weekly simulation scenarios as part of the training.

Adults (Appendix 3, Table A9)

There were 13 articles in this category. Programmes, where stated, included lectures and simulation (only two did not follow an accredited programme). The majority of studies compared outcomes following the introduction of training into an institution; however, three studies (Dane et al. 2000; Moretti et al. 2007; Woodall et al. 2007) compared outcomes between groups of individuals who had received training with those who did not within the same institution. Seven studies reported a significant improvement in patient outcome, all of them showing a statistically significant reduction in mortality as well as in some improvement in other patient outcomes (Camp et al. 1997; Dane et al. 2000; Arreola et al. 2004; Van Olden et al. 2004; Moretti et al. 2007; Woodall et al. 2007; Spearpoint et al. 2009). Six studies reported a significant improvement in clinical management (less errors occurring or improved management at specific tasks; Vestrup et al. 1988; Makker et al. 1995; Camp et al. 1997; Arreola et al. 2004; Van Olden et al. 2004; Woodall et al. 2007).

Summary of findings from Kirkpatrick 4

Most of the studies reporting outcomes at Kirkpatrick 4 level were carried out over many years – with a period before SRT being introduced (typically 2–3 years) being compared with one after its introduction. From these, there is overwhelming evidence from the reported studies that the introduction of SRT within an institution has a direct positive impact on mortality and also on clinical management. The majority of SRT that were delivered were accredited programmes, which include a mixture of lectures and simulation. There were no clear differences in outcomes between accredited and non accredited training programmes.

Discussion

This review has described and analysed the evidence available for the efficacy of SRT on acquisition of knowledge and skills, their retention and the effect of SRT on patient care and outcome. This is the first systematic review of the literature investigating these issues. The following section summarises our conclusions regarding this in relation to the review aims and suggests a number of practice points to guide improvement in training resuscitation practice.

After attending SRT programmes do the participants have a sustained retention of resuscitation knowledge and skills after their initial acquisition?

It is clear that immediately after the vast majority of SRT programmes, knowledge and skills assessed by written examination and simulation are significantly improved (all studies where this was reported showed this to be the case). After some SRT, knowledge, assessed by written examination, may be maintained for 3 to 12 months after the initial training. There were no differences with respect to the education provided or assessments used in studies where knowledge had deteriorated compared with those where it was retained. Although it is possible that knowledge retention (given that knowledge is necessary to enable an individual to use their skills in resuscitation) may result in an improvement in clinical resuscitation practice, there is no evidence available that demonstrates this. However, the ability to demonstrate appropriate resuscitation practice in a simulated scenario is more likely than not to deteriorate after SRT as early as three months after training. Therefore even if knowledge retention did improve clinical resuscitation practice, it seems not to result in maintenance of appropriate practical skills in a simulated scenario.

There is no evidence available to assess whether ability in resuscitation procedures in clinical practice changes after SRT in individuals, what the time frame for this change (if it occurs) may be and whether there is any correlation with loss of ability in a simulated environment. Further work needs to be done to investigate this (see subsequent sessions).

Much of the training offered in SRTs consist of lectures with simulation with a mannequin and is thus very similar across accredited training programmes and even in those studies that reported nonaccredited programmes. As previously discussed, educationally this SRT training approach seems to be optimal as it offers experiential learning (Kolb 1984) through practical simulation experiences aimed at supporting experiential and reflective learning (Issenberg et al. 1999) and incorporates many facets within the simulation scenarios, which facilitate learning (Issenberg et al. 2005) although learning was not sustained. There were no characteristics of individual training programmes identified that influenced the retention of knowledge and skills at a later date. Deliberate practice, reported to encourage ‘mastery’ (Ericsson KA 2006; McGaghie WC 2011) does not seem to have been specifically or consistently used in the SRTs reviewed. Incorporating this into SRTs may involve more time and a higher instructor – candidate ratio to ensure that all participants have achieved mastery.

Support for participants after attending SRTs may also be an important focus in order to try and ensure change in clinical practice and maintenance of skills. Some studies reviewed here suggested that factors, which may ameliorate deterioration in knowledge and particularly skills might be the provision of regular booster or refresher sessions and focusing on discrete skills as part of a task during training and at follow-up (O’Donnell & Skinner 1993; Kovacs et al. 2000; Ander et al. 2004; Heidenreich et al. 2004; De Regge et al. 2006; Wayne et al. 2006). As well as further simulation sessions, other work has suggested that ‘reinforcement’ in the clinical area to strengthen behaviour will also improve competence (Burns 2000).

Is there an impact on outcomes for patients and/or their healthcare organisation?

It is clear from data in this review that the introduction of SRTs within institutions, where no previous training existed, has a positive effect on patient outcome and leads to improvement in clinical management. In particular, mortality rates are reduced. There is clearly a ‘group’ or institutional effect of introducing these courses. However, the relative benefits for subgroups of different disciplines of healthcare practitioners is unclear. Given that there was no training before the introduction of SRTs into the institutions who reported improvement, it is likely that resuscitation practice within these institutions was at a low baseline thus making improvement more likely to occur. There is no evidence available to assess whether further improvement might occur in institutions where all staff are trained (i.e. a higher baseline of resuscitation practice) and extra training offered before mandatory updates.

Value for money and practicalities of training

Current mandatory training programmes take place at their most frequent annually, sometimes every two to three years. This review suggests that further, earlier intervention with participants might be appropriate. This not only has cost, but human resource implications. It would not be practical to offer three monthly cycles of booster resuscitation sessions at institutions – rather it might be more feasible to embed aspects of deliberate practice (including resuscitation drills) at staff induction sessions and into daily work.

If institutions are to organise and run their own in-house SRT programmes it is important that they ensure that they incorporate appropriate educational approaches into these.

Further research

Investigation of later clinical performance in individual participants in relation to skills learnt on SRT programmes and whether deteriorations in skills after SRT as assessed by simulation correlates with deterioration of skills in clinical practice are areas that have not been researched. This may be quite difficult to do, possibly involving routine videoing of resuscitation. There are ethical and consent issues surrounding this practice and, at present, there is no validated assessment tool for this. There are also concerns that videos may be used in litigation cases (O’Donnell et al. 2008).The effects of embedding aspects of deliberate practice into routine work and the use of resuscitation drills require further work and the timing and frequency of booster sessions has yet to be determined.

Where staff of all disciplines in a healthcare institution are trained in resuscitation, there is a need for research, which investigates whether the learning that takes place on subsequent resuscitation courses results in improvement in resuscitation management.

Strengths, weaknesses and limitations of the review

This review has systematically obtained literature pertaining to SRTs and their impact. Results have been reported by speciality (adult, paediatric and neonatal), thus, facilitating the readers understanding of the evidence available within each speciality.

The systematic review only considered articles from the English language literature to avoid the long potential time delay that obtaining translations may have entailed. This is often standard practice for systematic reviews, making it possible that articles with relevant data (in another language), which could have contributed to the results may have been overlooked. There is, however, no evidence of a systematic bias from the use of language restrictions in systematic reviews (Morrison A et al. 2009). The nature of the published body of evidence ruled-out a formal meta-analysis for this review. Heterogeneity of research designs and unstandardized outcome measures made a quantitative synthesis of the research evidence impossible. By the nature of qualitative analysis of themes, the quality of the final data collection and analysis depends on the integrity and unbiased approach of the researchers. Bias is possible if the researchers approach the subject with preconceived notions which may affect the findings. In order to minimise this, validation of the analysis was carried out by triangulation of the findings with others members of the review group.

Conclusions

1. SRTs result in an improvement in knowledge and skills in those that attend them.

2. Deterioration in skills and to a lesser extent knowledge is highly likely as early as three months following SRTs.

3. There is a small amount of evidence that booster or refresher sessions may improve an individual's ability to retain resuscitation skills after initial training. However, the timing and frequency of these in different disciplines has yet to be determined.

4. Ensuring clinical staff of all disciplines in a healthcare institution, where no previous training existed, are trained in resuscitation will improve the clinical management and mortality rates after resuscitation attempts.

5. Where staff of all disciplines in a healthcare institution are trained in resuscitation, there is a need for research, which investigates whether the learning that takes place on subsequent resuscitation courses results attended by individuals from these institutions results in further behavioural change in the clinical area (that is a change in clinical practice) thus further improving resuscitation management.

6. There is an urgent need for research to determine whether deteriorations in skills after SRT as assessed by simulation correlates with deterioration of skills in clinical practice.

Acknowledgements

With thanks to Marilyn Hammick for her support and encouragement with this BEME review and Neonatal resuscitation funds, Liverpool Women s Hospital.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Appendix 1: Table to show search strategy

Appendix 2A: Coding Sheet printed computerised version

Appendix 2B: Initial assessment of quality

Table

Criteria		Yes (2/good)	Partial (1/fair)	No (0/poor)	N/A
Study aims
1.	Is the hypothesis/aim/objective of the study clearly & sufficiently described?	Easily identified in introduction/method. Specifies: purpose, subjects/target population, and specific interventions/associations under investigation.	Vague/incomplete reporting or some info has to be gathered from parts of the paper other than intro/background/objective section.	Question or objective not reported/incomprehensible.
Study design & sample characteristics
2.	Is the study design well described and appropriate?(If study question not given, infer from conclusions).	Design easily identified, well described and appropriate.	Design and/or study question not clearly described, or design only partially addresses study question.	Design does not answer study question or design is poorly described.
3.	Is the method of intervention group selection described and appropriate?	Described and appropriate.	Selection methods not completely described, but no obvious inappropriateness. Or selection strategy likely introduces bias but not enough to seriously distort results.	No information/inappropriate information provided or selection bias, which likely distorts results.
4.	Are the characteristics of intervention group clearly described (i.e. age range, occupation)?	Sufficient relevant demographic information. Reproducible criteria used to categorise participants clearly defined.	Poorly defined criteria or incomplete demographic information.	No baseline/demographic info provided.
5.	Have the characteristics of participants lost to follow-up been described?	Losses adequately reported & not likely to affect results.	Losses not well reported, but small & not likely to affect results.	No information or losses large and likely to affect results.	No participants lost to follow-up.
6.	Are educational intervention(s) clearly described?	Defined and reproducible.	Partially defined, but insufficient detail to reproduce design.	Not described.
7.	Is method of delivery of educational intervention and subsequent follow-up clearly defined?	Sufficient relevant descriptive information. Reproducible criteria used to replicate intervention defined.	Poorly defined criteria or incomplete descriptive information.	No criteria/descriptive info provided.
Data analysis and results
8.	Are the main outcomes to be measured clearly described in the introduction/method?	Defined and measured according to reproducible criteria.	Definition leaves room for subjectivity, or not sure (i.e. not reported in detail, but probably acceptable). Or precise definition(s) are missing, but no evidence of major problems. Or instrument/mode of assessment(s) not reported.	Main outcomes first mentioned in results section. Or measures not defined/inconsistent/poorly defined.
9.	If possible, was an attempt made to blind those measuring the main outcomes of the intervention?	Assessor blind to intervention/study group.	Inadequate blinding: i.e. assessor may have been aware of group participant assigned to.	No attempt made to blind assessor.	Not possible/appropriate – e.g. observational/before & after study.
10.	Are population characteristics (if measured & described) controlled for and adequately described?	Appropriate control at design/analysis stage or randomised study with comparable baseline characteristics.	Incomplete control/description. Or not considered but unlikely to seriously influence results.	Not controlled for and likely to seriously influence results.
11.	Are the outcomes chosen to evaluate the intervention appropriate?	Appropriate outcomes selected and reported.	Some outcomes not relevant to assessing appropriateness of intervention.	Outcome measures do not evaluate intervention or poorly reported/not defined/inconsistent.
12.	Are the main findings clearly described?	Simple outcome data (e.g. mean/prevalence) reported for all major findings.	Incomplete or inappropriate descriptive statistics.	No/inadequate descriptive statistics.
13.	Are methods of analysis adequately described and appropriate?	Described and appropriate.	Not reported but probably appropriate or some tests appropriate, some not.	Methods not described and cannot be determined.
14.	Are estimates of variance reported for the main results?	Appropriate estimates provided (SD/SE, confidence intervals).	Undefined or estimates provided for some but not all outcomes.	No information.
15.	In trials/cohort studies, do analyses adjust for different lengths of follow-up, or in case-control studies, is the time between intervention and outcome the same for cases/controls?	Different lengths of follow-up adjusted for (e.g. survival analysis) and adequately described.	Different lengths of follow-up probably adjusted for but not adequately described.	Differences in follow-up ignored.	Cross-sectional design or same length of follow-up.
Conclusions
16.	Are the conclusions supported by the results?	All conclusions supported by data.	Some of the major conclusions are supported by the data; some are not. Or speculative interpretations are not indicated as such.	None/few of major conclusions supported by the data.

Appendix 2C: Final quality assessment criteria

Methodology

1. Randomised control trials

Individuals are randomly allocated to a control group and another group who receive a specific intervention- groups are identical for significant variables.

2. Cohort study

Groups are selected based upon their exposure to something and followed up for a specific outcome.

3. Case control studies

Cases with the condition/subject of interest are matched with ‘controls’ without

4. Cross sectional surveys/studies

Interview/questions are of a sample of the population of interest at a certain point in time

5. Case study report

A report based upon a single patient

Quality score

4. Results from this are clear with good methodology.

3. Results are unclear with good methodology

2. Results are clear but with poor methodology

1. Results are unclear and specific to the individual study.

Appendix 3

Table A1.  Kirkpatrick 2A and 2B neonates

Author	Accredited training programme	Nature of the training	Tested knowledge at the end of the training	How was knowledge tested	Knowledge significantly improved at the end of the training	Tested skills at the end of the training	How were skills tested	Skills significantly improved at the end of the training	Confidence assessed (if so how)	Significant improvement in confidence at the end of the training
Cavaleiro et al. 2009	No	50 min Lecture then self study or Simulation (RCT)	Yes – after lecture and again after self study/simulation	MCQ	Yes after lecture (p < 0.0001) No after self study/simulation	No	N/A	N/A	No	N/A
Ergenekon et al. 2000	No	8 hours Lectures simulation	Yes	MCQ	Yes (mean score precourse 9.5 vs. post course 14.2) (p = 0.001)	No	N/A	N/A	Yes, assessed in evaluation form at the end of the course	72% felt more confident at the end
Trevisanuto et al. 2005	Yes NRP	2 courses 2 day Lectures Simulation	Yes	MCQ	Yes Both courses (52% to 85% and 64% to 94%) p ≤ 0.01	No	N/A	N/A	No	N/A

Notes: MCQ- multiple choice questionnaire, NRP- Neonatal Resuscitation Programme, N/A – Not applicable, RCT – randomised controlled trial.

Table A2.  Kirkpatrick 2A and 2B paediatrics

Author	Accredited training programme	Nature of the training	Tested knowledge at the end of the training	How was knowledge tested	Knowledge significantly improved at the end of the training	Tested skills at the end of the training	How were skills tested	Skills significantly improved at the end of the training	Confidence tested at the end of the training (if so how)	Significant improvement in confidence at the end of the training
Dobson et al. 2003	No	Six hours lectures simulation	No	Not tested	Not tested	No	Not tested	Not tested	Yes – Likert scale	Yes in all 13 areas tested (p ≤ 0.002)
Donoghue et al. 2009	No	Simulation – Hi fidelity vs low (RCT)	No	Not tested	Not tested	Yes	Simulation with two different manikins	Both groups improved scores but P value not stated (High fidelity group improved more than low fidelity) (P= 0.007)	No	N/A
Gerard et al. 2006	Yes PALS	Web based course vs traditional PALS	Yes	MCQ	Not stated	Yes	Video of performance	Not stated	Yes	Scores improved from 3.77 to 4.28 in one group and 3.57 to 4.24 in other group (p value not stated but 95% confidence intervals indicate significant)
Quan et al. 2001	Yes PALS	Two days Not stated	Yes	Written case scenarios	No	Yes	Video simulation	Yes P=<0.05 and <0.01 depending upon skill	Yes questionnaire	Mean confidence score increased from 1.9 to 6.2 (p value not stated but 95% confidence intervals indicate significant)
Waisman et al. 2002	Yes PALS	Not stated	Yes	MCQ	Yes – proportion passing exam increased from 62% to 84% (p ≤ 0.001)	No	N/A	N/A	No	N/A

Notes: MCQ – multiple choice questionnaires, PALS- Paediatric Advanced Life Support, RCT – randomised controlled trial.

Table A3.  Kirkpatrick 2A and 2B adults

Author	Accredited training programme	Nature of the training	Tested knowledge at the end of the training	How was knowledge tested	Knowledge significantly improved at the end of the training	Tested skills at the end of the training	How were skills tested	Skills significantly improved at the end of the training	Confidence tested at the end of the training (if so how)	Significant improvement in confidence at the end of the training
Aboutanos et al. 2007	No	Lectures simulation	Yes	MCQ	Score increased from 72% to 79% (p = 0.032)	Yes	OSCE	Not known – no precourse score	No	N/A
Ali et al. 1995	Yes ATLS	Not stated (RCT of course vs not course)	Yes	MCQ	Improved compared with control group (p < 0.01)	Yes	OSCE	Improved compared with control group (p < 0.01)	No	N/A
Ali et al. 1996a,b	Yes ATLS	Not stated (RCT of course vs not course)	Yes	MCQ	Improved compared with control group and precourse scores (no p value)	Yes	OSCE	Improved compared with control group and precourse scores (p < 0.05)	No	N/A
Ali et al. 1998	No PHTLS	Not stated	yes	MCQ	Improved compared with control group and precourse scores (p < 0.05)	Yes	Simulation	Improved compared with control group and precourse scores (p < 0.05)	No	N/A
Bilger et al. 1997	Yes AHA	Model telephone simulation (RCT – phone vs no phone)	No	N/A	N/A	Yes	Use of phone	Improved in group taught with model phone (p < 0.01)	No	N/A
Cimrin et al. 2005	No	Lectures simulation	No	N/A	N/A	Yes	Simulation	Improved from score of 11.2 precourse to 15.6 postcourse (p < 0.001)	No	N/A
Azcona et al. 2002	Yes ATLS	Not stated	Yes	MCQ	Improved from 0% to 100% pass	Yes	Simulation	Improved from 5/16 to 16/16 passed	No	N/A
Dauphin et al. 2007	Yes ALSO	2 days Lectures simulation	Yes	MCQ	Improved from mean score of 55% precourse to mean 86% postcourse (p < 0.01)	No	N/A	N/A	Yes questionnaires	Eight of nine felt more confident
Dunning et al. 2006	No	Lectures simulation	No	N/A	N/A	Yes	Simulation	Improved times in most tasks (all p < 0.05)	No	N/A
Featherstone et al. 2005	Yes ALERT	Not stated	No	N/A	N/A	No	N/A	N/A	Yes questionnaires	Confidence improved in many areas (p < 0.01)
Girdley et al. 1993	Yes ATLS	Lectures simulation	Yes	MCQ	Improved from mean score of 28.3% precourse to mean 34.5% postcourse p = 0.0001	No	N/A	N/A	No	N/A
Greig et al. 1996	Yes BLS	Not stated	No	N/A	N/A	Yes	Simulation	Yes (six weeks later) – p value not stated	No	No
Devita et al. 2005	No	Lectures, simulation, debriefing	No	N/A	N/A	Yes – as a team	Simulation	Improved survival and task completion after training (p < 0.002)	No	N/A
Hoadley 2009	Yes ACLS	Lectures simulation	Yes	MCQ	Improved score from mean precourse of 80% to postcourse mean of 89% (p < 0.001)	Yes	Simulation	Not known	Yes	N/K
Jensen et al. 2009	Yes ALS	Lectures simulation RCT	Yes	MCQ	Improved scores of means of 73 and 70% precourse to 85 and 83 % postcourse (no p value reported)	Yes	Simulation	Yes (combined score with MCQ)	No	N/A
Mayo et al. 2004	No	Two groups – one received training the other not simulation	No	N/A	N/A	Yes	Simulation	Improved in most areas in group receiving training (p < 0.001)	No	N/A
Monsieurs et al. 2005	No	Not stated RCT two different bagging systems	No	N/A	N/A	Yes	Simulation	Not stated – automatic bagging system better than manual p = 0.0001)	No	N/A
Marshall et al. 2001	Yes ATLS	Not stated	No	Not sated	N/A	Yes	Simulation	Skills improved in all areas postcourse (p < 0.002)	Yes survey	Increased from mean score of 5.8 to 8.1 (p < 0.01)
Murphy & Fitzsimons 2004	Yes ILS	Not stated	No	N/A	N/A	No	N/A	N/A	Yes – qualitative	Improved (qualitative data)
Owen et al. 2006	No	Simulation	Yes	MCQ	Yes p = 0.001	Yes	simulation	Improved (p value not stated))	Yes – questionnaire	p ≤ 0.001
Rosenthal et al. 2006	No	Simulation	No	N/A	N/A	Yes	Simulation	Improved in nearly all areas (at 6 weeks after)from pre-course score (p<0.0001)	No	N/A
Tippet 2004	Yes ATLS	Not stated	Yes	Short answers	Improved from mean 61% to mean 83% (p = 0.006)	No	N/A	N/A	No	N/A
Wayne et al. 2005	No	Simulation RCT	No	N/A	N/A	Yes	Simulation	Improved scores – 38% higher than controls with no training (p<0.0001)	No	N/A

Notes: ACLS – Advanced Cardiac Life Support, AHA – American Heart Association, ALERT – Acute Life-Threatening Events: Recognition and treatment, ALSO – Advanced Life Support for Obstetrics, ATLS – Advanced Trauma Life Support, BLS – Basic Life Support, MCQ – multiple choice questionnaire, N/A – not applicable, OSCE – Objective Structured Clinical Examination, PHTLS – Prehospital Trauma Life support.

Table A4.  Kirkpatrick 2C neonates

Author	AP	Nature of the training	No of participants Followed up (D=doctors, N=nurses, S=students O=other)	When tested at follow-up	Components of ability tested		Knowledge change	Notes	p	Skill change	Notes	p
					K	S
Curran et al. 2004	No	Computer manikin	60 (D)	Four and eight months (one group with booster)	Yes MCQ	Yes Simulation	R then NC	RCT – Decreased knowledge at four months in both groups then remained same at eight months	<0.0001 then Not SIGNIF	R	RCT – Both groups at eight months	<0.0001
Dunn et al. 1992	No	One day Lectures Demo	166 (N)	Six months	Yes MCQ	Yes Simulation	NC	Mean MCQ score from 91% to 85%	Not SIGNIF	R	All passed after training – All failed at follow-up	N/K
Duran et al. 2008a,b	Yes NRP	Lectures Simulation	42 (D)	6 and 12 months	Yes MCQ	No	R	Mean MCQ score from 94.5% to 59.2% after 6 months and 93.2% to 58.3% after 12 months	N/K	N/A		N/A
Kuczorowski et al. 1999	Yes NRP	Video Practical	44 (D)	6–8 months	Yes MCQ	Yes Simulation	R	RCT – All passed after training –at follow-up 26 (56%) passed in control group and two other groups who had booster	N/K	R	All passed after training – all failed at follow-up	N/K
Levitt et al. 1996	Yes NRP	N/K	10 (D)	6–9 months	Yes MCQ	No	NC	Mean MCQ score from 86.4% to 75.4%	Not SIGNIF	N/A		N/A
Skidmore & Urquhart 2001	No	Lectures Simulation	62 (D and N)	6 months 12 months	Yes MCQ	Yes Simulation	NC		N/K	R	After six months (but not back to pretraining score)	N/K
Trevisanuto et al. 2005	Yes NRP	2 days Lectures Simulation	25 (D)	6 months	Yes MCQ	No	R	Mean MCQ score from 94.1% to 62.7%	<0.0001	N/A	N/A	N/A
West 2000	No	Two hours N/K	6 (N)	6 weeks	Yes MCQ	Yes Simulation	NC		N/K	NC		N/K

Notes: AP – Accredited programme, Knowledge change:, I – Increased, NC – No change, p – p value, MCQ – Multiple-Choice Questionnaires, Not SIGNIF – not significant, N/A – Not applicable, N/K – Not known, NRP – Neonatal Resuscitation Programme, R – Reduced.

Table A5.  Kirkpatrick 2C paediatrics

Author	AP	Nature of the training	No of participants Followed up (D = doctors, N = nurses, S = students, O = other)	When tested at follow-up	Components of ability tested		Knowledge change	Notes	p	Skill change	Notes	p
					K	S
Durojaiye and O’Meara 2002	Yes PLS	N/K	23 (D)	2 weeks and 2 and 4 months	Yes phone questions	No	NC		N/K	N/A		N/A
Nadel et al. 2000	No	Eight hours Lectures Simulation video – Then booster in one group	57 (D)	Approx 12 months	Yes MCQ	Yes Simulation video	N/K	Did not report change in knowledge but group who received booster did better than control group	N/K	N/K	Did not report change in skills over time but group who received booster did better than control group	N/K
Spaite et al. 2000	No	Self-study	11 (O)	Four months	Yes ‘test’	No	R	Mean score in test fell from 13.04 to 11.59	<0.01	N/A		N/A
Su et al. 2000	No	16 hours Lectures Simulation Subgroups had simulation or knowledge exam at 6 months	43 (O)	12 months	YesMCQ	Yes simulation	R	Scores in reduced between 19.7% and 22.3% at follow-up – no difference between groups randomised to have test at six months and controls	<0.05	N/K		N/K
Wolfram et al. 2003	Yes PALS	N/K	99 (O)	Mean 21 months	Yes MCQ	No	R	25% passed exam at follow-up	N/K	N/A		N/A

Notes: AP – Accredited programme, Knowledge and Skill change: I – Increased, NC – No change, MCQ – Multiple-Choice Questionnaires, N/A – Not applicable, N/K – Not known, p – p value, PALS – Paediatric Advanced Life Support, PLS – Paediatric Life Support, R – Reduced.

Table A6.  Kirkpatrick 2C adults

Author	AP	Nature of the training	No of participants Followed up (D = doctors, N = nurses, S = students O = other)	When tested at follow-up	Components of ability tested		Knowledge change	Notes	p	Skill change	Notes	p
					K	S
Aboutanos et al. 2007	No	Lectures simulation	12 (D)	Two years	Yes MCQ	No	I	Mean score increased from 65% to 77%	<0.05	N/A		N/A
Ali et al. 2002	Yes ATLS	Lecture Simulation	144 (D)	Six months, 2, 4, 6 and 8 years	Yes MCQ	Yes OSCE Simulation	R	High and low trauma-exposed groups reduced scores from 83.9% to 74.8% and 81.9% to 74.6%, respectively, at six months. After this, no group passed MCQ	N/K	R		N/K
Ali et al. 1996a,b	Yes ATLS	N/K	60 (D)	Six months, 2,4 and 6 years	Yes MCQ	Yes OSCE	R	Scores after ATLS 85.3 – 87.7% in four groups. At six months = 77.8% (50%) pass, at two years 70.6% (0 passes), at four years 69.4% (0 passes), at six years 68.9% (0 passes)	N/K	R	Score after ATLS 16.6. Score at six months = 16.8, at two years = 13.9, at four years 12.0, at six years 11.9	N/K
Ander et al. 2004	No	Four hour lectures two hours and Simulation	40 (D)	Six and 12 months	No	Yes Simulation	N/A		N/A	I	Two of three skills improved at follow-up	N/K
Azcona et al. 2002	Yes ATLS	Not stated	59 (D)	Less than two years (38) and more than two years (21)	Yews MCQ	No	R	8/38 and 2/21 passed at follow-up	N/K	N/A		N/A
Beckers et al. 2007	No	15-minute lecture	59 (S)	Six months	No	Yes Simulation	N/A		N/A	R	Time to first shock elongated from mean of 56.5 seconds posttraining to 59.9 seconds at follow-up (but not back to pretraining level)	<0.01
Blumenfeld et al. 1998	Yes ATLS	N/K	220 (D)	Three to 60 months	Yes MCQ	No	R	Mean score 84% post-course and 66% at follow-up. 50% participants scored above 80% by 180 weeks	N/K	N/A		N/A
Boonmak et al. 2004	No	One-hour lecture, one-hour Simulation	30 (N)	Three months	Yes MCQ	Yes Simulation	R	Mean score fell from 75.4% to 60.5% at follow-up (back to pretraining levels)	N/K	N/C	(Mean skill score after training 79.7, at follow-up 75.7)	Not SIGNIF
Bradley et al. 1988	No	10-hour and four-hour lectures and Simulation	51 (O)	18 months (after Six month test at follow-up)	Yes MCQ And written	Yes Simulation	?R	RCT Proportion of failures may have increased in both groups at follow-up – no formal analysis	N/K	?R	Proportion of failures may have increased at follow-up – no formal analysis	N/K
Broomfield 1996	Yes ENB	Three hours Lectures Simulation	19 (N)	10 weeks	Yes MCQ	Yes Simulation	R	Mean score 23.9 postcourse and 19.4 at follow-up (higher than pretraining)	<0.0001	R	Mean score 7.2 postcourse and 5.1 at follow-up (higher than pretraining)	<0.0001
Coleman et al. 1991	No	Four hours of either lectures, discussion, handouts and simulation or e-learning	49 (S)	Three months	Yes MCQ	Yes Simulation	NC	Maintained scores in both groups	Not SIGNIF	NC	Maintained scores in both groups	Not SIGNIF
Cooper et al. 2007	Yes ILS	One-day lectures Simulation	29 (D, N, O)	Six months	Yes MCQ	Yes Simulation	N/C	Mean score 82% postcourse and 80% at follow-up	Not SIGNIF	R	Mean score 99% postcourse and 85% at follow-up (higher than pretraining)	0.02
Curry and Gas 1987	No	N/K	85 (D and N)	Six and 12 months	Yes MCQ	Yes Simulation	R	Doctors mean score 89.6% postcourse, 84% at 6 months and 83.4% at 12 months Nurses mean score 92.3% postcourse, 82% at 6 months and 79.4% at 12 months (Both back to pretraining levels)	N/K	R	Both for doctors and nurses – numerical data not reported (Both back to pretraining levels)	N/K
Curry and Gas 1983	Yes CPR	N/K	12(N) 13(D) 12(N) 6(D)	Six months 12 months	Yes MCQ	Yes Simulation	R	D and N had decreased after six months – back to pretraining levels by 12 months	<0.05	R	D skills decreased after six months, N by 12 months – both back to pretraining by 12 months	<0.01
De Regge et al. 2006	No	Simulation	Two groups of 16 (N)	Three and six months	No	Yes Simulation	N/A		N/A	N/C	Two groups with different resuscitation bags – Efficiency of ventilation stayed the same in both	Not SIGNIF
Erickson et al. 1996	No	30-minute lectures and Simulation	11 (D, N, O)	Two months	No	In clinical area	N/A		N/A	R	Airway and trauma skills decreased at follow-up (to pretraining levels)	N/K
Fabius et al. 1994	No	Computer demo and Simulation	54 (N, O)	Six months	No		N/A		N/A	?R	RCT – two groups – 1 person in each group ‘passed’ at follow-up compared with six and 34, respectively, immediately posttraining	N/K
Fossel et al. 1983	No	Simulation	41 (S)	One year and two years	Yes MCQ	Yes Simulation	R	Score lower at one and two years compared with those followed up at 2-3 weeks after training	<0.001	R	CPR performance lower at one and two years compared with those followed at 2-3 weeks after training	<0.05
Hammond et al. 2000	Yes ALS	Two days Lectures Simulation	40 (N)	18 months	Yes MCQ	Yes Simulation	N/C	Mean score 81.7% post course, 83.8% at follow-up	Not SIGNIF	R	75% percent passed at follow-up	N/K
Heidenreich et al. 2004	Yes AHA course	25 minutes Instruction Video Simulation	28 (S)	Six and 18 months	No	Yes Simulation	N/A		N/A	R and N/C	In standard CPR – reduction percentage correct chest compressions from 54 to 35 then 32. In uninterrupted chest compressions stayed the same	<0.02
Holden et al. 1996	No	N/K	55 (N)	Six months	Yes MCQ	Yes Simulation	N/C	Number achieving more than 50% correct answers did not deteriorate	N/K	? R	21 (38%) passed skills test at six months	N/K
Jensen et al. 2009	Yes ALS	Lectures simulation	Two groups (immedaitely and 6 months after qualification)	Six months	Yes composite	Yes composite with MCQ	N/K		N/A	N/K	Mean scores reduced from 85% and 83% post course to 82% and 78% at follow-up	N/K
Kovacs et al. 2000	No	One-hour lecture, five hours simulation and half-an-hour/week for three weeks	84 (S)	16, 25, and 40 weeks	No	Yes Simulation	N/A		N/A	N/C And R	RCT – groups with practice and feedback maintained skills. Control group and group receiving feedback alone deteriorated by 16 weeks (Mean score 45.2 to 34 and 45.2 to 35.4, respectively)	<0.05 in D groups
Leith 1997	No	‘Defibrillation training’	10 and 10 (N)	Six and 12 months	Yes write exam	Yes Simulation	R	Mean score reduced from 89% post course to 76% at 6 months and 70% at 12 months	N/K	R	Pass rate of practical test decreased from 70% to 0% at 6 and 12 months	N/K
Mancini & Kaye 1985	Yes AHA	N/K	33 (D)	Mean eight and 22 months	Yes Assessment and action	Yes Simulation	?R	Less than 100% of candidates responded correctly in all aspects except for assessing unresponsiveness (presume all needed to be correct immediately post-training)	N/K	?R	Less than 100% of candidates performed correctly in all aspects except for ventilating (presume all needed to be correct immediately post-training)	N/K
McKee et al. 1994	No	Lectures Simulation	50 (N)	One week, one, three and six months	No	Yes Simulation	N/A		N/A	R and N/C	Delay to defibrillation increased – resuscitation score deteriorated then improved to post training score at 6 months	<0.05 and Not SIGNIF
O’Donnell & Skinner 1993	No	20-minute lecture and Simulation	44 K 60 S	Six months	Yes Questionnaire	Yes Simulation	N/C and R	N/C in the 2 groups with refresher sessions and D in group with no refresher prior to follow-up	Not SIGNIF and <0.05	I	Group with monthly refresher sessions improved in ‘pass rate’ for performing cardiac massage from 39 to 69%	<0.05
O'steen et al. 1996	Yes ACLS	N/K	40 (N)	Mean 344 days (0–1034)	Yes MCQ	Yes Simulation	R	After 12 months – no further deterioration after	<0.05	R	After 12 months – no further deterioration after	<0.05
Plank & Steinke 1989	No	Two hours Lecture and simulation vs video	36(N)	6–8 weeks	Yes MCQ	Yes simulation	N/K	Pass score not stated	N/A	R	26 failed at follow-up	Not stated
Semeraro et al. 2006	Yes ALS	Lectures Simulation	47 (D)	Six months	Yes MCQ	Yes Simulation	R	Mean score 85.9% post-course and 79.5% at follow-up	<0.001	R	All passed postcourse and 30 passed at follow-up	<0.001
Smith et al. 2008	Yes ACLS BLS	N/K	133 (N)	Three, six, nine and 12 months	Yes MCQ	Yes Simulation	N/K	Results not reported	N/K	R	Four groups – all deteriorated – some to pretraining	N/K
Spooner et al. 2007	Yes BLS	eight hours Lectures simulation	66 (S) RCT standard vs feedback mannequin	Six weeks	No	Yes simulation	N/A		N/A	R	Fail rate increased from 16% and 25% to 10% and 38%, respectively	N/K
Stross 1983	Yes ACLS	N/K	132 (D)	One year	Yes ECG recognition and mock arrest	Yes Simulation	N/C and R	RCT. All three groups maintained ECG recognition but deteriorated in mock arrest – the two groups receiving booster sessions performed better at follow-up	N/K	R	39% performed successful ventilation and 47% external cardiac massage compared with all having passed ACLS course	N/K
Ten Eyck 1993	No	Four hours lectures and Simulation	48 (O)	Six months	No	Yes simulation	N/A		N/A	R	Five failed resuscitations and others did not perform other required aspects of simulation	N/K
Tippett 2004	Yes ATNC	21/2 days Lectures simulation	14 (N)	Three months	Yes Short answers	No	R	Mean score 83% postcourse and 73% at follow-up (back to precourse levels)	<0.05	N/A		N/A
Wayne et al. 2006	No	4 × 2 hours teaching and HI fidelity Simulation	38 (D)	Six and 14 months	No	Yes Simulation	N/A		N/A	N/C	In context of RCT previously over six months which involved three lots of testing	Not SIGNIF
Wenzel et al. 1997	No	Two hours Instruction Simulation	113 (S)	Six months	Yes MCQ	Yes Simulation	R	Mean score 6.4 postcourse and 6.2 at follow-up	<0.05	R	Five of nine skills deteriorated significantly	<0.0001
Yakel 1989	Yes 2 BLS courses	45 minutes or eight hours Lectures Simulation	81 then 86 (N)	Four months and eight months	No	Yes Simulation	N/A		N/A	R	Mean score 55 postcourse and 38 at 4 months. Improved to 42 at 8 months (p < 0.001) – four month test acted as booster (remedial training given). Longer course did better at follow-up (p < 0.05)	N/K
Young and King 2000	No	N/K	10 (N)	Six and 12 weeks	Yes Oral questions	Yes Simulation	R	Five failed at six weeks, five failed at 12 weeks	N/K	R	Five failed at six weeks, six failed at 12 weeks	N/K

Notes: AP – Accredited programme, ACLS – Advanced Cardiac Life Support, AHA – American Heart Association, ALS – Advanced Life Support, ATLS – Advanced Trauma Life Support, BLS – Basic Life Support, CPR – Cardio-pulmonary Resuscitation, ILS – Immediate Life Support, Knowledge and Skill change D – Decreased, MCQ – Multiple-Choice Questionnaires, N/A – Not applicable, NC – No change, N/K – Not known, Not SIGNIF – not significant, p – p value, R – Reduced.

Table A7.  Kirkpatrick 4 neonates

Author	Accredited programme	Nature of the training	Period studied	Significant impact on patient outcome	Nature of the impact on patients (data)	Significant increase in survival rates (data)	Significant improvement in clinical management	Nature of the improvement (data)
Boo 2009	Yes NRP	Lectures Simulation	Five years pretraining and 8 years posttraining	Not stated	Numerical decline in perinatal and neonatal mortality but no p values	? Less mortality	Not stated	Not stated
Duran et al. 2008a,b	Yes NRP	Not stated	Over three-year period including pre and post implementation of training	Yes	After training: significant increase in one minute Apgar score (5.43–6.5 – p = 0.01) Babies with ischaemic changes on CT reduced from 91% to 62% (p = 0.02) Reduction in inpatient stay from 12 to 6.1 days p ≤ 0.05	No statistically significant difference	Not stated	Not stated
Patel et al. 2001	Yes NRP	Not stated	Before training (1985–1988), during transition (1989–1990) and after training (1991–1995)	Yes	Fewer babies with low one and five minutes Apgar scores posttraining p ≤ 0.001	Not stated	Not stated	Not stated
Patel & Piotrowski 2002	Yes Neonatal resuscitation programme (NRP)	Not stated	Before NRP – 1985–1988, after 1991–1995.	Yes	Higher one minute Apgar score (7–10) (24% pre vs 31% post NRP – p = 0.001) and higher five minute (53% vs 65% – p < 0.001). More changed from low one minute to high five minute after NRP (39% to 49% – p < 0.001)	Not stated	Not stated	Not stated
Ryan et al. 1999	Yes NRP	Lectures Simulation	51 deliveries before and 51 deliveries after the training (1994–1995)	Not stated	Not stated	Not stated	Yes	Improvement in delivery room preparation (p = 0.01), management (p = 0.01) assessment (p = 0.02) and interventions (p = 0.02)
Singh et al. 2006	Yes NLS	Lectures simulation	Data collected precourse in 1990–1994 and postcourse in 1997 and 2003	Not stated	Not stated	Not stated	Yes for par of findings	Inappropriate use of naloxone fell from 75% to 10 % (p = 0.0001). Total use of naloxone fell from 13% to 0.5%, incidence of hypothermia fell from 9% to 2.3% (both not statistically significant)
Zhu et al. 1997	Yes NRP	Not stated	Pretraining 1992, posttraining 1993 to 1995	Yes	3× reduction in neonatal mortality (9.9 to 3.4 per 1000) p < 0.001	Yes

Notes: NLS – Neonatal life support, NRP – Neonatal Resuscitation Programme.

Table A8.  Kirkpatrick 4 paediatrics

Author	Accredited programme	Nature of the training	Period studied	Significant impact on patient outcome	Nature of the impact on patients (data)	Significant increase in survival rates (data)	Significant improvement in clinical management	Nature of the improvement (data)
Lo et al. 2009	No	Simulation Weekly scenarios	23 weekly training sessions of approximately 30 minutes	Not documented	N/A	Not documented	No	Median time for chest reopening significantly longer- p = 0.002. Longer to give medication p = 0.002
Losek et al. 1994	No (PALS for emergency medicine in State)	Lectures as before but with additional supervised practice from 1986	Patients 0–18 years from January 1990 to December 1991 compared with data from January 1983 to June 1985	Not documented	N/A	Not documented	Yes	<18-month-old improved intubation p = 0.000008 and vascular access p = 0.000003 Older child improved vascular access p < 0.05

Notes: N/A – not applicable, PALS – Paediatric Life Support.

Table A9.  Kirkpatrick 4 adults

Author	Accredited programme	Nature of the training	Period studied	Significant improvement in patient outcome	Nature of the impact on patients (data)	Significant increase in survival rates (data)	Significant improvement in clinical management	Nature of the improvement (data)
Arreola et al. 2004	Yes PHTLS, BLS and ALS (some in house)	Not stated	Three ambulance services October to December 1994, pretraining and January to June 1995 posttraining. January to September 2000 pretraining and October 2000 to June 2001 posttraining	Yes	Not stated	Improvement in survival in those patients transported in one intervention centre (p = 0.04)	Yes	Improved airway management in the two intervention centres (p < 0.001) Improved spinal immobilisation in one intervention centre p < 0.001 Some improved iv fluid administration in two intervention centres (p < 0.001)
Camp et al. 1997	ACLS	Not stated	1980 to 1984 pretraining and from 1985 to 1990 posttraining	Yes	Increased ‘death events’ reversed by intervention increased from 2% to 11% (p < 0.001) posttraining	Yes	Yes	Increased intervention at ‘death events’ posttraining (from 5% to 37% – p < 0.001)
Curry and Gas 1987	No	Not stated	1981–1985 – one hospital received training – the other did not	No		No	No	No difference in death rates between trained and untrained staff
Dane et al. 2000	Yes ACLS	Not stated	1996 and 1997 Compared resuscitation outcome of nurses ACLS trained with those not	Yes	4× more likely to survive when treated by trained nurses (38% to 10%) p = 0.02	Yes	Not stated	Not stated
Makker et al. 1995	Yes ACLS	Not stated	1991 – 225 cardiac arrests	No		No	Yes	Certified doctors made less errors in first semester after training (5.9%) compared with second semester (14.7%) (p = 0.05)
Moretti et al. 2007	Yes ACLS	Two-day course Lectures Simulation	January 1998 to March 2001 Compared resuscitation outcome of personnel ACLS trained with those not	Yes	Increase in return of spontaneous circulation with trained versus non-trained (49/113 vs 16/59 p = 0.04)	Better survival in ACLS trained group: at 30 days (27% versus 6% – p = 0.02) and 1 year (22% vs 0% p = 0.002)	Not stated	Not stated
Murphy & Fitzsimons 2004	Yes ILS	Not stated	1999–2000 and 2001–2002	Not stated	Not stated	Not stated	No	No difference in personnel who inserted mask or defibrillated
Seidelin & Bridges 1993	No	Six hours simulation	Six month period Aug 1987 to January 1987 compared with 1985	No		No	No
Spearpoint et al. 2009	Yes ILS	One day Lectures simulation	January 2002 to December 2007 – training ongoing during this period	Yes	Increase in survival to return of spontaneous circulation p ≤ 0.005	Reduced deaths at cardiac arrest over time period (p < 0.0002) Survival to discharge after emergency call increased to 39% (2007) from 28% (2004) p < 0.005	Not stated	Not stated
Van Olden et al. 2004	Yes ATLS	Not stated	May 1996 to September 1997 precourse and December 1997 to April 1999 postcourse	Yes	Significant reduction in mortality 24.2% to 0% in first 60 minutes following resuscitation	Yes	Not stated	Not stated
Van Olden et al. 2004	Yes ATLS	Not stated	Compared period pretraining (June 1996 to November 1997) to posttraining (January 1998 to July 1999)	Not stated	Not stated	Not stated	Yes	10 (of 14) procedures were performed better posttraining and management scores increased from 4.2 pretraining to 5.8 posttraining (p < 0.0001)
Vestrup et al. 1988	Yes ATLS	Not stated	Compared periods of pretraining (April 1983 to March 1984) to post training (April 1985 to 1986)	Not stated	Not stated	Not significantly different	yes	Significant increase in rectal examinations for trauma patients p = 0.03
Woodall et al. 2007	Yes ACLS	Lectures clinical placement	January 2000 to December 2002 Compared resuscitation outcome of paramedics ACLS trained with those not	Yes	Pulse on admission more likely in ACLS trained (21%) compared with nontrained (8.5%) p = 0.0001	Yes Increased survival to discharge in ACLS trained (6.7%) versus nontrained (4.66%) p = 0.03	Yes	Quicker mean time to first shock in ACLS trained (9.44 minutes) vs nontrained (10.07 minutes)

Notes: ACLS – Advanced Cardiac Life Support, ATLS – Advanced Trauma Life Support, ILS – Immediate Life Support, BLS – Basic Life Support, PHTLS – Pre-Hospital Trauma Life Support Course