SAFE QI – a framework to overcome the challenges of implementing a quality improvement curriculum into a residency program

Abstract

Quality improvement (QI) is an essential component of medical practice. Medical students and residents must learn the skills to conduct clinical QI during their educational programs. Medical educators must create and implement a curriculum in QI to empower their students to develop this skill and knowledge. However, developing and implementing a QI curriculum may be challenging for some residency programs. Residency programs with a relatively short duration of training – for example, only 2 years – may be unable to implement an extensive QI curriculum without siphoning away time for other learning objectives. Small residency programs may lack faculty with expertise to teach this topic. Residency programs with only a few residents may find it difficult to evaluate the success of a QI curriculum using robust statistical analysis. These residency programs need a QI curriculum with several features. The curriculum must be deliverable in a short period of time. There must be tools to assess the residents’ attainment of the curricular objectives. The curriculum must give the residents practical skills to develop their own QI initiatives. Finally, there must be simple methods to evaluate the curriculum’s effectiveness. To address these goals, we developed the SAFE QI (QI curriculum which is short, assessed, functional, and effective) framework for the 2-year subspecialty respirology residency program at the University of Alberta. There are 2–3 entrants per year for a total of 4–6 residents. This framework helps medical educators overcome the challenges of implementing a QI curriculum into their educational programs. This article illustrates how this framework was used to develop and deliver an institution’s own QI curriculum.

Keywords:

• curriculum development
• quality improvement
• residency education
• clinical practice audit

Background

Postgraduate medical residents must learn skills to implement quality improvement (QI) into their medical practices.^1,2 To accomplish this, medical educators must develop and implement curricula in clinical QI into their residency training programs.

For example, some residency programs have used an intensive 12-week process to teach QI.³ Other residency programs require their residents to study high-risk cases and document their assessment in a database which is reviewed by faculty.⁴ Mann et al found large variability in QI teaching amongst the pediatric residency programs they surveyed.⁵ QI curricula have been successfully integrated into many other residency programs including psychiatry,⁶ geriatrics,⁷ obstetrics-gynecology,⁸ radiology,⁹ and surgery.¹⁰

However, despite the success of many residency programs, a curriculum in QI may still be difficult to implement for certain residency programs.^3,11 Some residency programs have a relatively short duration of training, for example, only 2 years. Their residents might have limited time to complete a QI project. Also, educators may want to keep the QI curriculum relatively short so that it does not consume too much of the total training period and siphon away time to learn other curricular objectives of the residency program. Some residency programs are small and have only a few residents. Compared to larger residency programs, smaller residency programs may have fewer faculty members. Thus, some residency programs may lack the faculty with the expertise to teach or demonstrate this topic.³ Quantitative curriculum evaluation, with robust statistical analysis, is often difficult to implement with small residency programs.¹² They are often unable to accrue large numbers of subjects (i.e., residents) to study. Rather, curriculum evaluation often tends to focus on qualitative data collection. These data may include feedback from the learners about the quality of instruction or opinions from the faculty on the success of the QI initiatives.

To resolve these problems, some residency programs might combine their instructions with other residency programs to pool resources.¹³ However, this can lead to other problems.¹⁴ For example, the instruction may be too generic and less relevant for the residents’ needs. Or, the residency program may choose to rely solely on didactic teaching without giving the residents experiential learning to develop their own QI projects.

In 2011, the 2-year respirology residency program sought to overcome these challenges in implementing our QI curriculum at the University of Alberta. Two to three residents enter this program each year after completing 3 years of core internal medicine. There are a total of 4–6 subspecialty residents at a given time. This article reflected on the challenges for a small residency program and turned limitations into advantages.¹⁵ To do this, we first conducted a needs analysis. This involved surveying faculty, as well as past and current residents, to determine what we required to teach a QI curriculum. Based on this, we determined that our QI curriculum needed four essential features.

First, the curriculum needed to be short and focused. This was done so that residents could attain these learning objectives during their 2-year residency, yet leave sufficient time to teach the rest of the specialty’s training objectives. Second, we needed to assess whether the residents had achieved the curricular goals. This meant developing assessment tools for this purpose. Third, the curriculum needed to teach residents skills in QI that were functional. In other words, we deemed it insufficient to provide didactic instruction alone. Our residents needed experiential learning.^7,16,17 We aimed to provide this hands-on learning with feedback so our residents could acquire the skills to complete QI projects in their future practice with the least waste of time and resources. Fourth, the curriculum delivery needed to be effective, achieving results that were valued and intended.^18,19 Furthermore, our small residency program needed to use simple measures of effectiveness. In the case of QI, we sought to determine whether the residents’ QI initiatives led to improved clinical care, whether their work was valued by faculty, and whether the residents retained the skills to complete QI projects after residency.

We incorporated these four essential features into a pilot framework which we named SAFE QI. This acronym describes our QI curriculum which is short, assessed, functional and effective. The remainder of this article illustrates how other medical educators can use the SAFE QI framework to overcome the challenges of implementing their own QI curriculum into their training programs.

SAFE QI curriculum

The four elements of the SAFE QI curriculum are summarized in Table 1.

Table 1 Components of the SAFE QI curriculum

Short	Keep the QI curriculum comprehensive but short so that it does not siphon away too much time to teach other educational objectives in the residency program.
Assessed	Use tools to assess the residents’ skills in developing quality improvement programs.
Functional	Provide learning opportunities that give residents functional skills that they can use in their own clinical practice.
Effective	Develop a curriculum that is effective, achieving results that are valued and intended.

Abbreviation: SAFE QI, quality improvement curriculum which is short, assessed, functional, and effective.

Short

SAFE QI is a QI curriculum that can be delivered in a relatively condensed period. To do this, educators need to provide focused teaching. We chose to focus our teaching on the principles of conducting a clinical practice audit.^20–23 This would give our residents the knowledge and skills to initiate QI in their own practices. QI can also examine large systems of practice, such as within a hospital or entire health care region. However, these types of projects require considerable time and labor to plan and implement. It would be difficult for our residents to complete such large-scale projects within a 2-year residency.

We delivered the curriculum in two stages. The first stage occurred near the beginning of residency. To maximize learning, we provided the initial instruction using a 1-hour buzz group where the larger group was split into pairs to discuss the topic, and then reformed for a whole group discussion.²⁴ Residents participated in this small group session and learned the 14 steps of performing a clinical practice audit, such as choosing a topic, choosing a criterion standard, and writing out the primary and secondary audit questions.²⁰ We delineated each of these steps using post-discharge care of patients admitted with exacerbation of COPD as an example. We then used brainstorming to apply these steps to other clinical activities.

In the second stage of curriculum delivery, we gave the residents tools to complete their own clinical practice audit. We created a checklist of items that residents needed to include in their audit, as well as a template for writing their report. The report template described the format we expected, including the background, methods, subjects, data collection, results, and discussion. The template also specified the contents which should be included, such as the primary and secondary audit questions, the criterion standard against which current clinical practice was being compared, and the data collection methods. Residents were given up to 15 months to develop and complete their project, and to submit their report. They were allotted 3 hours of scheduled time per month to work on their projects. Every 3 months, the residency program director met with the residents to review their progress. During these meetings, the residents gave updates on the number of steps they had completed in their project and described any barriers they encountered toward completing these steps.

Assessed

During curriculum development, educators need to plan how to assess whether the residents have attained the curricular objectives. According to Miller’s pyramid,²⁵ this can include examinations to test their knowledge and/or understanding or observations of their performance. Alternatively, work-based assessment can involve analyzing the residents’ outcomes, process of care, or volume.²⁶

Rather than simply assessing their recall of our instruction, we chose to assess the residents’ skills in performing an actual clinical practice audit. To achieve this, we created a scoring sheet and rating scale to assess the quality of those audits (Table 2). The scoring sheets assessed whether the audits contained all the requisite elements including the use of an appropriate criterion standard, a clear primary question, clear secondary questions, collection of appropriate data, identification of any gaps between actual and ideal practice, and feasible recommendations to ameliorate those gaps. Each item would then receive a rating. For example, lack of an appropriate criterion standard would receive a rating of “Failed to Meet Expectations” for that item. Providing an appropriate criterion standard would receive a rating of “Met Expectations”. Cogently explaining the choice of that criterion standard, and providing sound reasons for not choosing other criterion standards, would receive a rating of “Exceeded Expectations”. The scoring sheet also contained a global rating score to assess the overall quality of the audit. After grading the projects, we provided teaching feedback.

Table 2 Quality improvement project assessment form

The resident demonstrated the following objectives:
Learning objectives	Could not assess	Failed to meet expectations	Met expectations	Exceeded expectations
1. Chose an appropriate topic
2. Used an appropriate criterion standard
3. Wrote a clinical practice audit question +/–secondary questions
4. Chose appropriate data to answer the question(s)
5. Designed a useful data collection form
6. Chose an appropriate number of charts/patients/studies to review
7. Established an appropriate method to obtain the data
8. Successfully obtained ethics approval, if needed
9. Answered the audit question
10. Effectively presented the results in a written report
11. Determined if there were gaps between actual and ideal performance
12. Recommended practical changes to narrow the gaps (if any) between actual and ideal performance
Global rating scale
		Failed to meet expectations	Met expectations	Exceeded expectations
Overall, this quality improvement project met the training objectives

Comments:

Functional

Educators should design and deliver a curriculum that provides functional knowledge and skills. That is, the residents should be able to take what they have learned and apply it in clinical practice. Unfortunately, this does not always occur.^27–29 In the case of QI, residents should attain the competencies to complete their own project, with little wasted time and resources. To accomplish this curricular goal, we did two things.

First, we demonstrated how to identify areas in need of QI. Our department compiled a list of clinical service areas that might benefit from a clinical practice audit. These clinical areas included outpatient clinics, inpatient wards, and diagnostic and therapeutic services such as the bronchoscopy unit, pulmonary function laboratory, and sleep study facility. From this list, we brainstormed ideas with the residents to identify specific topics that could be suitable for an audit. For example, topics included complication rates after bronchoscopy, accuracy of pulmonary function test interpretations compared to established guidelines, and appropriate use of pulmonary rehabilitation and medications on discharge after admission to hospital for an exacerbation of COPD. By having faculty and residents work together in this process, we promoted the hidden curriculum³⁰ that QI was valued. We then asked the residents to choose a topic that interested them or to develop their own. To efficiently use their time and resources, we encouraged them to do retrospective analysis of data that had already been collected.

Second, while residents were conducting their clinical practice audits, we identified and corrected a few practical barriers hindering their progress. For example, our residents found it difficult to access clinical data during office hours while they were on other busy clinical rotations. Therefore, we gave them after-hours access to the data and protected time off their rotations during the day. Some hospital-based topics needed ethics approval before the projects could proceed. We discussed the reasons for this^31,32 with our residents and helped them obtain approval from our research ethics board when needed. We initially met with each resident individually every 3 months to review his or her progress, but later found it helpful to add group meetings with all residents every 6 months to discuss common barriers hindering their projects.

Effective

Before curriculum implementation, medical educators should plan as to how they will determine if the curriculum delivery is effective.³³ In other words, the curriculum should successfully produce the desired outcomes, and these outcomes should serve a valued purpose. Some residency programs need simple measures of effectiveness as they lack sufficient numbers of residents to perform robust, quantitative statistical analysis. For our QI curriculum, we chose to evaluate three outcomes.

First, we wanted to ensure that the residents’ projects were useful. We sent the residents’ clinical audit reports to the faculty engaged in the clinical activity that was being audited. We then asked for the faculty’s feedback. Specifically, we asked if the residents’ reports were clear and if their recommendations for improvement were feasible. Second, we examined whether the residents retained confidence in their QI knowledge after finishing their residency. Third, we wanted to analyze whether the residents’ clinical practice audits improved clinical care. We retained all clinical practice audit reports. After spending 4 years to implement the recommendations, we have asked new, incoming residents to repeat the audits to determine if improvement had occurred.

Curriculum evaluation

After we introduced our QI curriculum, we evaluated the curriculum’s effectiveness over a 4-year period. During this time, 11 residents completed our residency program. All of them completed a QI project for a total of seven projects. Some of the residents worked in groups while others did projects alone.

Using a five-point Likert scale (1-strongly disagree, 2-disagree, 3-neither agree nor disagree, 4-agree, 5-strongly agree), we surveyed the nine faculty members who worked in the clinical areas that were audited by the residents. Of them, seven faculty responded to the survey. When asked if the residents’ audit reports were clear, three faculty responded “agree” and four faculty responded “strongly agree”. When asked if the residents’ recommendations for improvement were feasible, one faculty responded “agree” and six faculty responded “strongly agree”.

A year after finishing their residency, we surveyed the nine residents who had completed the residency program in the first 3 years out of our 4-year evaluation period. We received eight responses to our survey from these former residents. When asked if they felt confident implementing a QI initiative in their own practice, five respondents answered “agree” and three respondents answered “strongly agree”. In addition, four of these former residents indicated that they had already started QI initiatives in their own clinical practices.

It has been 4 years since we implemented our SAFE QI curriculum. We are now asking incoming residents to repeat all the earlier projects done by previous residents. Over the next 4 years, we plan to analyze the data to determine the extent to which the residents’ QI projects have led to improvements in clinical care.

Conclusion

Clinical QI is a vital component of medicine. Medical educators can use our SAFE QI framework to implement a QI curriculum into their own educational programs. We successfully implemented this into our small subspecialty residency program. In doing so, we efficiently satisfied our national accreditation requirements for residency training and taught our residents a useful skill for their own clinical practices. Our residents can now use their basic knowledge to collaborate with other health disciplines and participate in system-wide institutional QI initiatives. We are repeating all our QI projects to determine if clinical care has improved. Other small medical or surgical residency programs can also use this framework to achieve these goals.

Further work can be done to generalize the applicability of the SAFE QI curriculum into larger residency programs that possess greater numbers of residents and faculty. In so doing, residents in programs with a longer duration of training can experience the entire spectrum of QI from start to finish.

Acknowledgments

The author would like to thank the Respirology Residency Program Committee at the University of Alberta for approving the curriculum.

Disclosure

The author reports no conflicts of interest in this work.