Quality education: A pilot quality improvement curriculum for psychiatry residents

Abstract

Background: A series of Institute of Medicine's reports have highlighted the need for greater quality improvement (QI) training in medical education; however, few formal QI curricula for medical trainees have been described in the literature.

Aim: The objective of this study was to develop a contextual QI curriculum involving a QI workshop and longitudinal QI projects (QIPs) for psychiatry trainees.

Methods: We examined psychiatry residents’ attitudes on QI training following their exposure to a physician-manager curriculum using focus group methodology. Focus group data were used to inform revisions to the QI curriculum. Following the curriculum revisions, we administered a resident questionnaire to elicit resident perceptions on the modified QI curriculum.

Results: Focus group data from 40 psychiatry residents at the University of Toronto identified the following themes: challenges with QIP workload, difficulties of QI workshop integration into the curriculum, and value of the experiential component of the QIP. Of the 26 residents, 18 completed the resident questionnaire on the revised curriculum and reported an enhanced appreciation of QI in their current clinical practice.

Conclusion: The study results suggest that this experiential format warrants further exploration as a model for QI training in medicine.

Background

Over the last 20 years, gaps in health care quality have led to a renewed focus on continuous quality improvement (QI) in medical education (Berwick 1989, 1996). In 2001, the Institute of Medicine's report, one of many independent reports, identified large discrepancies between the potential standard of care and the actual care provided to the patients (Committee on Quality Health Care in America 2001). In the report, QI was defined as the process of improving “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with professional knowledge.” As a result of the emergence of quality care as a health care priority, medical training has required integration of QI principles into undergraduate and postgraduate medical education in order to prepare the next generation of practicing physicians.

In response to these reports (Helpern et al. 2001), both the Accreditation Council for Graduate Medical Education (ACGME) in the United States and the Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDs (2005) framework have mandated that QI be considered a key training competency during residency training. Within the ACGME Outcome Project (2005), the systems-based practice and the practice-based learning and improvement competencies identify QI as an essential component to residency training. Moreover, the RCPSC CanMEDs manager role specifies that “physicians are able to participate in systemic quality process evaluation and improvement, such as patient safety initiatives.” Despite the focus on establishing postgraduate trainee competency in QI, a paucity of formal QI curricula has been found in the literature.

Several studies have outlined the need for curricular reform in medical education to address the gaps in QI training. A survey of family medicine graduates indicated that trainees felt least prepared for the systems-based practice and practice-based learning ACMGE competency areas (Lynch 2003). In a Canadian survey of 495 psychiatry residents, the respondents’ perceived knowledge in QI was the lowest in comparison to 11 other administrative knowledge domains (Sockalingam et al. 2008). This is concerning given the requirement for physicians at all levels of the health care system to participate in improving the quality of patient care and safety (Becher & Chassin 2002).

Efforts to prepare faculty for teaching QI to residents primarily consisted of workshops, with few reports describing integrated QI curricula in postgraduate programs in medicine. Preliminary literature suggests that experiential training opportunities, such as QI projects (QIPs), may reinforce learning and retention of QI knowledge longitudinally in trainees (Davis et al. 1992; Mazmanian & Davis 2002). Additional QI education initiatives have been reported in internal medicine and surgery residency programs (Allen et al. 2005; Canal et al. 2007; Wong et al. 2008). In an attempt to reflect future professional practices, programs have piloted interdisciplinary QI curricula that offer a solution to bridging the QI training gaps between various disciplines (Singh et al. 2005; Varkey et al. 2006). However, no research has elicited trainee perspectives on QI training following the exposure to a pilot administrative curriculum, including teaching on QI principles and experiential learning through QIPs.

In keeping with our continued development of a physician-manager curriculum, we elicited the psychiatry residents’ preferences on the QI component of the physician-manager curriculum during their residency training. The results were used in collaboration with the opinions of faculty experts in QI and education to revise the QI curriculum prior to the second year of the QI curriculum. Potential participants had exposure to administrative and leadership training within the overall physician-manager curriculum and were able to use this experience to inform their feedback. In Phase 2 of the curriculum development, we sought to determine trainee feedback on the educational experience of the second iteration of the curriculum by administering an online resident questionnaire. Given the limited data on longitudinal QI training experiences that are clinically relevant to residents during their training, we aimed to utilize this data to develop a novel QI curriculum that could be generalized to other medical specialties.

Table

Practice points

Formal curricula in QI are lacking in medical education. Longitudinal QIPs can provide residents a practical and clinically relevant QI experience. Time commitment and workload are potential challenges to implement a QI curriculum for residents. Future studies examining additional methods for delivering QI education are needed.

Methods

Setting

In response to the CanMEDs and ACGME recommendations and identified gaps in administrative training in Canada (Somers et al. 2004), we developed and piloted a physician-manager curriculum at the University of Toronto for postgraduate year (PGY) 2 to PGY4 psychiatry residents in the 2006–2007 academic year (Stergiopoulos et al. 2009). During the 2006–2007 academic year, the Department of Psychiatry, University of Toronto, had 126 psychiatry residents enrolled in the residency program. Psychiatry residents at the University of Toronto undergo core residency training at seven major teaching hospitals and no physician-manager curriculum or formal training was offered at these sites prior to 2006. The development of the physician-manager curriculum is described in detail elsewhere and incorporated feedback from educational experts, content experts, and needs assessment data (Maggi et al. 2008; Stergiopoulos et al. 2009).

A QI workshop was delivered as one of eight half-day workshops in the first 2 years of the curriculum. All psychiatry residents were required to attend a centralized QI workshop during their PGY2 training. Based on the needs assessment data for the overall physician-manager curriculum (Sockalingam et al. 2007), the QI workshop employed didactic, case-based, and small-group learning methods. This is in keeping with the literature supporting the effectiveness of active and experiential learning in comparison to didactic teaching in producing lasting changes in physician practice (Mazmanian & Davis 2002).

A designated physician-manager curriculum committee comprising education experts, administrative psychiatry content experts, and psychiatry residents, developed the learning objectives for the pilot QI workshop, which are described in detail in a previous publication (Stergiopoulos et al. 2009).

During the second year of the physician-manager curriculum, residents were required to participate in longitudinal QIPs during the course of their PGY3 year. Although QIPs could be focused on any clinical area of psychiatry, PGY3 residents tended to focus on topics related to their core PGY3 training rotations, specifically rotations in child psychiatry, care of patients with chronic mental illness or emergency psychiatry (as part of on-call duties). QIPs related to core rotations were developed in collaboration with rotation or clinic supervisors in order to address active quality concerns and to promote sustainable changes within the service or program. Therefore, some projects were aligned with routine QI, whereas other QIPs were new initiatives within the programs.

PGY3 QIPs were introduced in a centralized QI educational seminar and the residents were given a QIP package, supervisor list, and their group assignments. The QIPs utilized the plan-do-study-act (PDSA) cycle as the framework for understanding and conducting QI (Table 1) (Berwick 1998). Each resident group was instructed to meet with their QI supervisor at least five times during the academic year before presenting their project at departmental rounds or research day. In order to foster project site support and participation, PGY3 residents were required to incorporate a multidisciplinary approach to their project and to include relevant nonmedical staff in the process.

Table 1.  PDSA cycle (Berwick 1998)

• Plan how you will implement the intervention

• Do it (carry out the change, preferably on a small scale)

• Study the process to see whether your intervention has made an improvement

• Act on what you learned (implement a change on larger scale or start a new area of improvement)

Methods for eliciting resident perceptions

We approached the QI curriculum development in two phases. First, we conducted a sub-analysis of QI-specific content obtained from focus groups examining residents’ perceptions on the overall physician-manager curriculum. Second, we collected resident evaluations of the QI experience after implementing changes to the QI curriculum based upon focus group feedback.

Phase 1: Focus groups

In Phase 1, 40 PGY2–PGY5 psychiatry residents at the University of Toronto participated in one of five focus groups conducted at five main clinical sites as part of an initial study examining residents’ perspectives on the physician-manager pilot curriculum. Participants in focus groups ranged from 4 to 10 at each site. Due to limited resident placement at two hospital sites, we opted to focus on the five larger training sites. Moreover, PGY1 psychiatry residents were excluded due to their limited exposure to psychiatry rotations and the physician-manager curriculum.

Participation in the focus groups was voluntary and there was no pre-selection of participants. Focus groups were facilitated by a research assistant with training in focus group facilitation and used questions from a standardized interview template asking the participants about their understanding of the physician-manager role, barriers to teaching and learning this role during residency, and resident perspectives on administrative curriculum content and format. Participants were informed that the focus groups would be audio-recorded and the investigators were not involved in the audio transcription process. All participants provided informed consent prior to participating in the focus group.

In keeping with grounded theory methodology, the focus group data were entered and analyzed using NVivo 8 qualitative analysis software to guide the discovery of themes pertaining to the physician-manager curriculum. We conducted a sub-analysis of data pertaining to the QI curriculum from a larger dataset discussing the overall physician-manager curriculum. The generation of themes and categories using a constant comparative method allowed for the development of higher order categories. This method of qualitative analysis was used to limit the influence of researcher bias in creating themes and categories. Line-by-line analysis of focus group transcripts and coding of themes was conducted independently by two of the authors (S.S. and V.S.). Any discrepancies in coding were discussed and themes were revised and refined on a second reading. Unifying conceptual themes were used to create higher order categories.

Phase 2: Resident QIP questionnaire

Prior to the second year of the QIPs, the QI curriculum was revised based on the focus group feedback and recommendations from faculty involved in the first year of the QI curriculum. As per focus group feedback, the QI workshop was moved from PGY2 to PGY3 to decrease the time between the workshop and the initiation of QIPs. Furthermore, the revised QI workshop included examples of previously completed QIPs and an opportunity for residents to break out into groups to brainstorm about their QIPs with expert faculty guidance. Residents continued to receive longitudinal QIP supervision by faculty supervisors throughout their PGY3 year. Residents were evaluated on their QIP by their participation in QIP supervision and by faculty evaluating the final QIP presentation. Examples of QIPs completed during this academic year are outlined in Table 2. Copies of residents’ QIP evaluations were included in their central psychiatry residency training file and QIP evaluations were used as part of their physician-manager assessment during their PGY3 clinical rotations.

Table 2.  Examples of resident team QIPs

• The use of chart audits to improve adherence to atypical antipsychotic metabolic monitoring guidelines on an Assertive Community Treatment team

• The development of a metabolic monitoring protocol for youth taking antipsychotic medication in a residential treatment program

• Implementation of a communication flow guideline to reduce psychiatric consult times on a consultation-liaison psychiatry service

• Medical clearance in the psychiatric emergency setting: the role for standardized physical examination forms

• A QI study of nicotine replacement practices in a short-term emergency psychiatry unit

• Improving the fragmentation of care between primary care physicians and psychiatrists with a consultation template for ongoing psychiatric care

• The impact of psychiatric medication information sheets on patient satisfaction with outpatient psychiatric treatment

• The role for standardized forms for emotional disturbed persons in the emergency department to improve communication between psychiatry teams and police officers

Many residents in the first iteration of the QIPs presented their projects at psychiatry educational rounds at their respective hospital sites. Faculty involved in the pilot QI curriculum suggested developing a specific QIP presentation venue to consolidate resident QI training. As a result, residents were encouraged to present their projects at a newly developed QIP Research Track created within the annual departmental research day during the second iteration of the QI curriculum. All QI presentations were evaluated on this research day by faculty and a “Best QI Project” award was given to the project with the highest ratings. Table 3 summarizes the key components of the QIP evaluations.

Table 3.  QIP presentation evaluation domains

Project should define a QI issue and be supported by a literature review

Clear identification and incorporation of key stakeholders related to QIP

Selected clear measures/processes/tools related to QIP

Baseline measurements of outcomes were conducted, and where applicable, an intervention was proposed and analyzed

QIP presentation was organized in a clear manner and reviewed with supervisor prior to presentation

A total of 26 psychiatry residents participated in the revised QI curriculum during their PGY3 academic year. In an attempt to elicit further feedback on the QIP experience and modifications, all residents were e-mailed an online questionnaire to evaluate their QI educational experience as part of standard postgraduate education procedure. The purpose of the questionnaire was to gather trainee feedback on the changes made to the QI component of the physician-manager curriculum based on the focus group data and the feedback from the faculty involved in the curriculum. The online questionnaire consisted of nine questions including four attitude questions rated on a five-point Likert scale (Table 4) and three open-ended questions. Completion of the evaluation form was anonymous and voluntary. Data obtained from the evaluation forms were analyzed using descriptive statistics.

Table 4.  Trainee reflection on revised QI curriculum

Experience	Mean (SD*)	Number of respondents agreeing with statement
The QIP reinforced important QI theories and principles	4.00 (1.03)	14
The QIP is important to our residency training	3.00 (1.33)	8
The project showed me how to use QI within my daily clinical practice	3.67 (0.84)	10
The QIP was my only formal training exposure to using QI in my clinical practice	3.83 (1.54)	12

Results

Focus group results

A total of 40 psychiatry residents participated in the physician-manager curriculum focus groups. The training levels of respondents were as follows: 16 PGY2, 6 PGY3, 10 PGY4, and 8 PGY5 residents. Nineteen participants (45%) were males. The following sections describe the QI-specific themes emerging from focus group data sub-analysis of the QI content.

Barriers to teaching and learning QI

Psychiatry residents identified workload as a major barrier to the QIPs. Projects were perceived as time consuming and a challenge to integrate in the midst of other educational and clinical duties. One respondent commented on the struggle with additional project hours:

“The quality improvement is a lot of work. … I probably spent over 30 hours on it on top of my work.” (Group 1)

“… All the readings for the presentation and stress of doing rounds. Like the data collection alone … having to go through all the charts… it adds up.” (Group 1)

Residents emphasized the need for clearly allocated educational time to complete QIPs and offset the workload demands.

Position of QI teaching within the overall curriculum

Focus group participants identified a need for improved integration of the QI workshop within the overall physician-manager curriculum. The curriculum involves eight workshops including the QI workshop. However, the residents in the focus group highlighted the need for QI education proximal to starting the QIPs. The distance between the PGY2 QI workshop and the PGY3 QIP was identified as an obstacle to the educational process. Furthermore, the residents proposed having more QI seminars during the PGY3 year to supplement the development and learning associated with the QIP:

“We could have gone through the exercises like how to do the quality improvement project in like maybe three to four seminars.” (Group 1)

“Maybe it would be worthwhile to just shift the quality improvement talk from PGY2 to right before PGY3 when we’re ready to do a project and a discussion about it would be more salient.” (Group 3)

Integrating QI teaching into clinical rotations

Several residents discussed the need to make the QIPs relevant to their clinical rotations in order to reinforce the use of QI principles in daily practice. Participants talked about the benefit and ease of developing a QIP based on their current psychiatric residency rotation. For example, one participant discussed the utility of discussing the QIP with the director of the inpatient ward during their inpatient psychiatry rotation, in order to select a project that would have clinical impact that would be observable during the rotation.

“If you’re asking about how we could implement the things you learn, well it would make sense that in PGY3 we’re actually trying to create some sort of small program or research that is service based.” (Group 3)

The project was also perceived to be beneficial in terms of evaluating residents on their QI knowledge and skills. Participants found it useful experience to think about quality issues during their clinical rotations. (Group 4)

“I think getting something experiential is actually quite helpful.” (Group 1)

Although many groups were limited to the plan-do-study components of the PDSA cycle, they were able to make recommendations to the clinical programs that were implemented by program or clinic administrators. Nonetheless, some groups were able to demonstrate changes in the quality of clinical care including a project improving the quality of referral note content for an inpatient psychiatric consultation service.

QIP resident questionnaire

Upon completion of the second iteration of the QI curriculum, a total of 18 of 26 psychiatry residents completed the online QI curriculum questionnaire yielding a 69% response rate. Only one QIP was an individual project. The perceptions of resident respondents are summarized in Table 4.

The respondents felt that the positive aspects of the revised QI curriculum were its team approach, opportunity for a formal QI training experience, the use of reflection and change management during the QI process, and being able to choose an area of interest for QI. One resident wrote, “We had a chance to really understand the principals and implement a change.” Another resident commented on the rewarding features of the project: “We were able to help the institution where we worked with our findings of our project.”

Respondents identified some weaknesses of the revised curriculum, specifically on balancing the time commitment for the project with other educational obligations. One resident wrote, “there should also be a web-based bank of QI projects that have been done including the names of residents, supervisors and include the slides of all projects to enhance continuity and as examples rather than re-inventing the wheel each year.”

Recommendations for reinforcing QI education during psychiatry residency included monthly QI case studies or rounds, increased number of QI seminars throughout their residency training, extending the QIPs over 2 years, and examples of faculty-developed QI initiatives. The need for greater faculty mentoring and faculty development was thought to be paramount to longitudinal learning and reinforcement of QI principles.

Discussion

This article reinforces the idea that formal QI training can be integrated into a medical residency training programs in a clinically relevant and practical manner. Although a larger physician-manager curriculum was established at our institution, the implementation of QI-specific curriculum within the larger curriculum provided challenges in terms of time commitment and integration. Resident focus groups identified the need for allocated time to reduce QIP workload, the need for QI education proximal to the initiation of QIPs, and the importance of linking their QIPs and workshop to quality issues encountered in their daily clinical rotations. We were able to implement changes in the timing of the QI workshop within the QI curriculum and continued to reinforce the clinical focus of QIPs. Due to competing educational reform within our department, we were unable to extend the amount of academic protected time for QIPs at the present time. Nonetheless, residents were exempted from their clinical responsibilities during the scheduled QI supervision.

Despite the early challenges of the curriculum, the preliminary feedback of residents participating in the revised QI curriculum was favorable. The resident workload associated with the QIPs continued to be an ongoing challenge and the residents perceived their QI education as being less important in comparison to their other clinical duties. However, questionnaire respondents indicated that the projects reinforced QI principles and was their only formal training exposure to QI. Nearly 90% of the respondents reported that they would be more likely to consider QI issues in their future practice as a result of this curriculum. Increased resident awareness of QI issues is paramount to creating future physicians skilled in participating in future QI. Moreover, the resident QIPs generated QI initiatives within the clinical settings studied and further reinforced the impact of QI on daily clinical practice.

The shift in resident perception of QI training with the use of QIPs and the revised QI curriculum is likely a result of specific curriculum revisions, such as the shift of the QI workshop to PGY3 and the creation of the QIP Research Track at our annual research day. The integration of the QI workshop into the PGY3 year reduced the lag time between residents learning QI principles and developing their QIPs, thus reinforcing QI training within their daily clinical activities. Furthermore, residents attended the QIP Research Track and allowed for reinforcement of QI issues within a variety of clinical contexts as highlighted by their peers’ presentations.

Despite residents’ positive experience of the QIPs, we were faced with the challenge of overcoming residents’ resistance to this additional educational requirement. The residents’ resistance to the projects were addressed by clear communication of QIP requirements by our psychiatry postgraduate department, support from the Chair of our department, and the creation of protected academic time for QIP teaching and supervision. The group experience of the projects provided additional peer support during the process. Finally, the QI supervisors and the curriculum coordinators participated in regular QI curriculum teleconferences to identify and resolve the emerging challenges during the QIP supervision process.

Our study is limited by our use of focus groups, which may be susceptible to selection bias and may not have captured all participants’ views on the QI curriculum. Second, the study was conducted at single university site and may limit generalization to other residency programs. However, nearly 70% of residents training in the Department of Psychiatry at the University of Toronto graduated from medical schools other than Toronto and represent a diverse resident cohort (Sockalingam et al. 2007). Although the 69% response rate for the QIP feedback questionnaire is comparable to other surveys administered to medical professionals (Cummings et al. 2001), there is a possibility of a response bias and residents with negative views of the QIPs may not have been adequately captured. Replication of this study at other educational institutions is needed to determine the generalizability of our findings.

Moreover, it is unclear if smaller residency programs would be able to deliver the proposed QI curriculum. Our QI curriculum was embedded within the larger physician-manager framework; however, the QI component consisted of an independent workshop supplemented by resident QIPs. Limited resources and faculty content experts in the area of QI may limit transferability of the curriculum to smaller residency training programs. A sufficient number of QI supervisors are needed for the longitudinal resident QIPs and argue for increased efforts in faculty development in this area. Collaboration with hospital quality and safety initiatives may be one strategy for smaller programs to deliver this type of training to their residents.

In addition, we are encountering challenges in adhering to QIP timeliness due to the delays associated with the need for Institutional Review Board (IRB) approval in specific circumstances. The participating hospital sites have had varying perceptions on the need for IRB approval for QIPs, and we have made efforts to establish an IRB consensus among the hospital sites.

Conclusion

This study underscores the value of an integrated QI curriculum during psychiatry residency training. The overall mission of this QI curriculum is to prepare future physicians to be agents of change and assume much needed leadership roles within the health care system. In keeping with the QI initiative, several iterations of a QI curriculum may be needed prior to producing a final model curriculum for QI training that can be used for trainees across medicine. Future quantitative studies are needed to establish if QI knowledge and skill gaps are narrowed by QI workshops and projects. Moreover, the curriculum was enhanced by the combination of an interactive QI workshop supplemented with experiential learning projects that reinforced the relevance of QI in residents’ daily clinical practice. This multifaceted approach is contingent on faculty development in QI and will need to be formally addressed as part of future QI training initiatives.

It is hoped that this QI curriculum will provide the foundation for our future physicians to assume QI roles within the current health care system. Early resident QI experiences provide an opportunity to engage our future health care leaders and to provide them with the administrative skills needed to succeed in today's complex health care system.

Acknowledgments

The authors would like to thank Dr Don Wasylenki, Chair, Department of Psychiatry, University of Toronto, for his support for this project. We thank our two research assistants, Dr Saima Khan and Ms Tutsirai Makawaza, for their efforts throughout this project. This study was funded by the 2007 Royal College of Physicians and Surgeons CanMEDs Development Grant.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Sanjeev Sockalingam

SANJEEV SOCKALINGAM is a postgraduate psychiatry coordinator and an assistant professor in the Department of Psychiatry, University of Toronto.

Vicky Stergiopoulos

VICKY STERGIOPOULOS is the medical director of Inner City Health Associates, Toronto, Ontario. She is also an assistant professor in the Department of Psychiatry, University of Toronto.

Julie Maggi

JULIE MAGGI is a postgraduate psychiatry coordinator and an assistant professor in the Department of Psychiatry, University of Toronto.

Ari Zaretsky

ARI ZARETSKY is a postgraduate director and an associate professor in the Department of Psychiatry, University of Toronto.