Clinicians prefer simple educational tools for implementing practice change

Abstract

Background: Successful implementation of practice change requires educational tools that engage and motivate clinicians.

Aim: To examine clinician usage and preference for different educational tools when a multifaceted strategy was used for implementation of new recommendations for managing newborns at risk of sepsis.

Methods: Seminars, web-based tutorial, handouts, pocket cards and web-based management algorithm were used to educate health professionals. Ninety-two clinicians attended seminars and completed feedback questionnaires that included three questions assessing knowledge of the recommendations. After 3 months, an electronic survey containing the same questions was sent to 41 key stakeholders (staff neonatologists, trainee physicians, nurse practitioners and respiratory therapists) who provided patient care during the implementation period. Compliance with recommendations was assessed by chart audit.

Results: Seminar content was helpful to 97% of participants and 88% were comfortable using the recommendations. Response rate for the 3-month survey was 80%. The most frequently used and useful tools were pocket cards (76%) and seminars (76%); 79% continued to use the card. Only one respondent used the web tutorial and four used the algorithm. There was no significant difference in percent correct responses to the questions between the two timepoints (p > 0.05). Compliance with the recommendations was 83%.

Conclusions: When provided with different educational tools, clinicians prefer pocket cards and seminars – tools that are simple and readily accessed.

Introduction

Clinical practice guidelines are systematically developed statements based on evidence and expert consensus intended to assist clinicians, patients and policy makers make appropriate health care decisions. New guidelines are developed or revised based on new knowledge and almost always lead to changes in practice. In 2006, Graham proposed the ‘knowledge-to-action’ process as a model for knowledge translation (Graham et al. 2006). One of the important steps in the knowledge-to-action cycle is implementation – the process of transfer and dissemination of information and its application in practice. Successful implementation of new guidelines and practice change requires educational tools that engage and motivate busy clinicians. This may be a particular challenge for inter-professional clinical teams, as the stakeholders are from different health care professions and at differing levels of training and competence. Increasingly, investigators are focusing on effective methods of dissemination and implementation of new recommendations. A variety of studies as well as systematic reviews have shown that multi-faceted educational interventions using active rather than passive educational strategies are most effective in changing practice (Grimshaw et al. 2001; Gross et al. 2001; Grimshaw et al. 2004; Jain et al. 2006). However, studies have not focused on the methods which clinicians prefer and use most frequently when multi-faceted approaches are used, in particular what methods are used by clinicians who work in inter-professional teams.

In 2007, the Canadian Paediatric Society (CPS) published new evidence-based recommendations for the investigation and management of newborn infants at risk of infection (Fetus and Newborn Committee, CPS, 2007). Early-onset sepsis (sepsis occurring in the first 72 hours of life) in newborns is an important clinical problem with an incidence of 1–2 / 1000 livebirths and case fatality rate of approximately 3% in term newborns (Moore et al. 2003; Puopolo, 2008). The presenting signs and symptoms are usually subtle and the consequences potentially devastating and therefore many non-septic infants are investigated and treated based on the presence of risk factors. These new recommendations were implemented in the Department of Paediatrics at Mount Sinai Hospital, Toronto, in 2009. Implementation provided an opportunity to explore clinician usage and preference for a variety of educational tools used as part of the dissemination strategy.

Table

Practice points

Multifaceted educational strategies are most effective when implementing practice change. Educational strategies should target key stakeholders. Educational tools that are simple and easily available are used most frequently. Educational tools that require time commitment and self-directed learning are used less frequently. Simple tools are effective in leading to practice change.

Methods

Mount Sinai Hospital is a tertiary perinatal centre located in a large urban centre. It is one of two perinatal referral centres for a region with approximately 80,000 births annually. At the hospital, there are approximately 6500 births annually, of which 1/3 is considered low risk, 1/3 at-risk and 1/3 high risk (Ontario Medical Association). All situations in which maternal or infant risk factors for neonatal sepsis are identified and a member of the paediatric health care team assesses the infant at the time of birth. A plan for investigation and management of the infant is then made. Until July 2009, clinicians used a hospital guideline based on the recommendations of the Society of Obstetricians and Gynecologists of Canada (Money and Dobson 2004) and the Centre for Disease Control (Schrag et al. 2002) for management of mothers and infants at risk of Group B streptococcal sepsis. The CPS recommendations were implemented during July, August and September of 2009.

Prior to implementation of the CPS recommendations, all stakeholders involved in the care of infants at risk of sepsis were identified – staff neonatologists, paediatric residents and neonatal – perinatal medicine subspecialty residents and fellows working in the neonatal intensive care unit (NICU), neonatal nurse practitioners, NICU respiratory therapists working in the labour and delivery suite, staff paediatricians, family physicians caring for newborns, midwives and nurses working in the NICU, labour and delivery suite and mother and baby (post-partum) unit. The first four groups of health professionals (neonatologists, residents and fellows, nurse practitioners and respiratory therapists) were the individuals most responsible for initial identification and management of the infants and therefore were considered key stakeholders.

Potential barriers to dissemination and implementation of new recommendations were considered. These included the ability of staff to commit time for education, frequent rotation of residents and fellows through the NICU on a monthly basis, availability of staff (most paediatricians, midwives and family practitioners had offices off-site) and differing roles in the care of these infants. As residents and fellows were in-house at all times, they were responsible for initially assessing at-risk infants and determining the appropriate investigation and treatment plan, under the supervision of the neonatologist. This task was shared with nurse practitioners during the day. Paediatricians and family physicians were responsible for the subsequent care of stable infants admitted to the mother and baby unit, including discharge planning. NICU respiratory therapists working in the labour and delivery suite were available for initial stabilization of infants when factors associated with increased need for newborn resuscitation were present and therefore identified many infants with septic risk factors.

Based on a literature review to identify effective strategies for guideline implementation and knowledge translation, a multi-faceted educational intervention was developed. The most in-depth educational initiatives focused on key stakeholders. A series of interactive seminars describing the new recommendations were provided for neonatologists, residents and fellows, nurse practitioners and labour and delivery room respiratory therapists. These individuals also received written information and a laminated pocket card which summarized the recommendations. Additional seminars were conducted for post-partum ward nurses. Information summarizing the new recommendations and practice changes, and the CPS position statement were sent electronically to paediatricians, family physicians and midwives. Two web-based tools were developed. The first was an algorithm which could be used to determine the appropriate investigation and management recommendations for individual infants. The second was a self-directed tutorial which included content about neonatal sepsis, described the recommendations and provided self-assessment questions with feedback on the correct responses. Both of these tools could be accessed from any computer within the hospital and were initially pilot-tested with a group of six residents. The link was sent to all stakeholders. Notices and reminders were posted in the labour and delivery suite and NICU. Pocket cards were available for all clinicians.

The recommendations were implemented during a 3-month period in 2009. Following this implementation period, new staff continued to receive written information and a pocket card.

Evaluation of the educational strategies was conducted during and after the 3-month implementation period. At the end of each seminar, participants were invited to complete a feedback form, which assessed their knowledge and satisfaction. Knowledge was assessed with three multiple choice questions related to three clinical situations about infants at risk of sepsis. Three months after the implementation period had ended, an electronic survey was sent to the 41 key stakeholders who had had patient care responsibilities during the implementation period, identified using staff schedules. The survey explored three domains – what educational tools had clinicians used, what tools did they find most useful and assessment of knowledge, using the same three questions asked on the seminar feedback form. Compliance with the new recommendations was determined by chart audit, carried out for 3 months subsequent to the implementation period.

Descriptive data was compiled using Microsoft Office Excel. Responses to the knowledge assessment upon completion of the seminar and after 3 months were compared using Fisher's exact test; a p value of ≤0.05 was considered to be significant.

This project was approved by the Research Ethics Board, Mount Sinai Hospital.

Results

Ninety-two health professionals, including seven neonatologists, eight residents, 20 subspecialty residents / fellows, three nurse practitioners, 17 respiratory therapists, 34 nurses and three unidentified individuals, attended 10 seminars and completed a feedback form (completion rate 100%). Fifty -five belonged to one of the four key stakeholder professions. Information was sent electronically to 100 paediatricians, family physicians and midwives. Seventy-five pocket cards were distributed during the 3-month implementation period.

Participant satisfaction responses from the seminar feedback form are shown in Figure 1. More than 80% of participants responded positively with regard to the content and presentation of the seminars. Perhaps most importantly, 88% indicated that they would be comfortable using the new recommendations.

Figure 1. The graph shows the responses of seminar participants to items on the seminar feedback questionnaire. Eighty-eight percent of respondents agreed that they would be comfortable using the new recommendations after attending the seminar and 97% found the content helpful.

Thirty-three individuals responded to the 3-month electronic survey (response rate 80%). The characteristics of the responders are shown in Table 1. Figure 2 describes the educational tools used by these clinicians. To learn about the new recommendations, 76% of respondents had attended a seminar and 76% used the pocket card. The pocket card was used most frequently by the nurse practitioners (100%) and residents and fellows (86%). Approximately 50% of all respondents had used the CPS statement itself or other written materials. Both the web-based algorithm and tutorial were used infrequently, as shown in the graph. Of these tools, the pocket card continued to be used most frequently after the implementation period, with 79% indicating that they continued to use the card. Again, the most frequent users were the nurse practitioners, residents and fellows.

Figure 2. Information about the use of educational tools from 33 key stakeholders who completed the 3-month electronic survey is shown. The first panel indicates that the most frequently used educational tools were the interactive seminars and pocket cards. As shown in the second panel, almost 80% continued to use the pocket card after 3 months.

Figure 3. This figure summarizes the responses of 33 key stakeholders who completed the 3-month electronic survey to questions about the usefulness of each educational tool. Over 80% of respondents found the interactive seminars and pocket card very useful.

Table 1  Demographics of survey respondents (n = 33)

Characteristics of respondents	% of Respondents
Resident or fellow	47%
Staff neonatologist / paediatrician	30%
Respiratory therapist / nurse practitioner	23%
Work primarily in the NICU	67%
Familiar with the previous hospital guidelines for infants at risk of sepsis	73%
Previously familiar with 2007 CPS statement	60%

Respondents were asked to indicate how useful they found the tools that they had used, using a 4-point scale ranging from not useful to very useful and this data is summarized in Table 3. Although all tools were found to be very useful by some, only the pocket cards and the seminars were reported to be very useful by the majority. Less than fifty percent found the two web-based tools very useful.

Following the implementation period, only 15% of respondents indicated that they were very uncomfortable using the new recommendations and always needed to refer to written information such as the pocket card or ask for advice. Forty-nine percent were reasonably comfortable but still occasionally used a written reference or asked for advice and 36% were very comfortable, rarely using a reference.

There were no significant differences in the percent of correct responses to the three knowledge assessment immediately following the seminar (n = 92) and after 3 months (n = 33) (question 1, 91% versus 96%; question 2, 80% versus 67%; question 3, 73% versus 71%), p > 0.05). The correct response to question 2 probably represents the most variance from previous hospital guidelines and this may account for the decrease in percent correct responses after 3 months.

Based on chart audit, compliance with the new recommendations was 83%.

Discussion

The multifaceted educational approach used for dissemination and implementation of a practice change (new recommendations for investigation and management of newborns at risk of sepsis) that involved different health care professions in a single hospital site was effective, as indicated by the high rate of compliance. This result is consistent with that reported for similar multifaceted strategies, although post-intervention compliance rates are not as high in some studies (Gross et al. 2001; Jain et al. 2006) as in our study. Based on responses to the three questions, there did not appear to be a significant decline in knowledge about management of infants at risk of sepsis 3 months after the implementation period.

Knowledge translation literature emphasizes that dissemination and implementation of new knowledge requires methods that are active, involve stakeholder participation and are integrated into current processes. We therefore chose seminars as one of the prime strategies for educating key stakeholders. Seminars were designed to be interactive and were conducted at times convenient for participants such as scheduled meeting and seminar times or shift change. Seminar content included background information about neonatal sepsis and reference to the evidence supporting the recommendations as well as review of the recommendations themselves. The seminars addressed some of the barriers to implementation of practice guidelines that have been identified, including awareness of guidelines, lack of understanding of guidelines rationale and clinician time (Grol 2001).

Pocket cards appeared to be a popular method of learning and reinforcing the practice change. This is consistent with previous literature demonstrating the effectiveness of pocket cards as part of educational strategies for implementation of new clinical practices and to improve health care provider knowledge (Blanco et al. 2005; Richardson et al. 2006; Mikhael et al. 2008). Pocket cards are inexpensive, easily implemented and readily available educational tools.

The most striking finding in this study was lack of use and lack of preference for electronic clinical and learning tools. In our centre, use of the web-based algorithm required that clinicians log on to a computer and go through several steps to access the algorithm. Using a pocket card was likely simpler and faster. Provision of the algorithm as an application for a personal digital assistant or mobile phone may have increased its utility and popularity. Using a before and after study design, Roy and colleagues showed that use of a handheld decision-support system improved diagnostic decision making for patients with suspected pulmonary embolism when compared to posters and pocket cards (Roy et al. 2009).

The past decade has witnessed a plethora of electronic and web-based learning tools, courses and curricula. Although these methods may have some advantages such as facilitation of distance learning, provision of immediate feedback and delivery of curricula to large numbers of learners, there is still not much evidence to support the notion that they are more effective than conventional learning strategies (Cook 2006). Although web-based learning may be a useful adjunct to conventional learning, some authors suggest that learners prefer conventional methods that involve personal interactions compared to electronic formats (Turner et al. 2006; Kerfoot 2008). One study does report physician acceptance of and some self-reported changes in practice following online problem-based learning modules based on clinical care guidelines (Robson 2009). However, there was little documented change in knowledge and participants volunteered. In our project, almost all clinicians who responded to the survey had attended a seminar and therefore may have felt that additional self-directed learning was not necessary.

There are several limitations to this study. This is a single-centre study with a relatively small sample size and therefore results may not be generalizable for implementation of recommendations on a larger scale. The study relies on self-reported data; actual use of the different tools could not be verified. Both the seminar feedback form and the electronic 3-month survey were completed anonymously; therefore it is uncertain whether the same individuals answered the knowledge assessment questions. Knowledge assessment was conducted 3 months after the implementation period and we did not determine whether there was retention of knowledge past this period. Some work has suggested that retention of knowledge declines after 6 months (Semerara et al. 2006).

In summary, clinicians from different health care professions involved in the care of newborn infants preferred seminars and pocket cards when learning about and using new clinical recommendations. The high degree of compliance with the practice change resulting from the new recommendations and evidence of knowledge retention suggests that these tools were effective. Educational strategies that required self-directed learning were used infrequently and this may be because of the time required to undertake these activities. Those involved in the implementation of new clinical guidelines should consider educational strategies that are simple and easily available.

Acknowledgements

Philip Ye provided assistance with the statistical analysis. The Ontario Ministry of Health and Long Term Care and the CIHR Team in Maternal-Infant Care Research (MiCare) supported the study. This study received financial support from the Academic Health Sciences Centre Alternate Funding Plan Innovation Fund, Ontario Ministry of Health and Long Term Care.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.