Facilitators and barriers to a nationwide implementation of competency-based postgraduate medical curricula: A qualitative study

Abstract

Background: Postgraduate medical education (PGME) curricula are being redesigned across the western world.

Aim: This study examined the implementation process (what works where and why) of new competency-based PGME curricula and relevant factors influencing this process.

Methods: In a nationwide project (2006–2010) in the Netherlands, competency-based PGME curricula were implemented for residents in Pediatrics and Obstetrics & Gynecology. The authors conducted 25 semi-structured interviews and used a multi-level theoretical framework to guide coding.

Results: The implementation process proved to be highly dynamic, non-linear, and influenced by many factors. These could be divided into attributes of the innovations/adopters, the implementation process, and the organization. The context determined the speed, quality, and direction of the process and how a factor affected the process.

Conclusions: We identified specific features of PGME innovation: the challenge of implementing other competencies than that of the medical expert; the importance of regional implementation strategies and educational support; the balance between training and patient care; and the need for regional inter-organizational networks of hospitals. The authors recommend: design the curriculum with the needs of the users in mind; facilitate knowledge sharing; organize educational support; translate the national curriculum to the local workplace; and promote regional inter-organizational networks between hospitals.

Introduction

In the past decade, postgraduate medical education (PGME) programs are being redesigned across the western world. Teaching philosophies that were already being used for a long time in undergraduate medical education have now been embraced by policymakers of PGME programs as well. Generally speaking, these philosophies concern outcomes-based education. This is a learner-oriented philosophy that focuses on the learner performance, or outcomes, instead of the resources available to students, or inputs. One approach to outcomes-based education which has become popular in medical education in the western world is competency-based education (Frank et al. 2010a). This can be defined as “an outcomes-based approach to the design, implementation, assessment, and evaluation of medical education programs, using an organizing framework of competencies” (Frank et al. 2010b). In the context of medical education, a competency can be defined as “an observable ability of a health professional, integrating multiple components such as knowledge, skills, values and attitudes” (Frank et al. 2010b). In the United States, the Accreditation Council for Graduate Medical Education started the outcome project in 1998, to develop and implement general competency areas in all PGME curricula (Swing 2007). Another popular approach toward competency-based education is the Canadian Medical Education Directions for Specialists (CanMEDS) framework (Frank & Danoff 2007), which has been used as a basis for reforming PGME curricula in Canada, Australia, and in several countries in Europe.

Although it is universally agreed that implementation of competency-based PGME curricula involves major challenges (Ringsted et al. 2003, 2006; Ten Cate & Scheele 2007; Scheele et al. 2008; Lurie et al. 2009; Wasnick et al. 2010), the implementation process and the factors influencing it have received little attention (Lillevang et al. 2009). In Denmark, the renewal of PGME curricula has been documented. While stakeholders were positive toward the task of developing new curricula, they found this task quite difficult and gained insufficient support during the process (Lillevang et al. 2009). Despite these interesting findings, the Danish study lacked an underlying theoretical framework, and the clinicians responsible for implementing and using the PGME curricula in practice were not interviewed.

The Royal Dutch Medical Association (RDMA) adopted the CanMEDS framework in 2004 for all PGME curricula in the Netherlands. Every medical scientific society in the Netherlands was requested to design a competency-based PGME curriculum according to the seven roles and the required assessment instruments (Table 1) (Jippes et al. 2010).

Table

Practice points

Despite the increasing attention to competency-based PGME curricula, research on the implementation process of these curricula is lacking. The implementation process of competency-based PGME is highly dynamic, non-linear, and influenced by many factors. The local/regional context determines the speed, quality, direction of the implementation process, and how specific factors affect this process. The implementation process is largely dependent on the quality of the initial design of the curriculum; we recommend designing the curriculum with the needs of the primary users (program directors, supervisors, and residents) in mind and align the benefits directly with the outcomes of health care processes, such as patient safety, to make it more attractive for the users to adopt the innovation(s). For implementing competency-based PGME, it is crucial to translate the national curriculum to a regional and local curriculum that specifies the CanMEDS roles, procedures for work-based learning, and assessment in the context of patient care. For organizing the implementation process, it is important to create coordination, establish goals and a timeframe, and monitor the results. This coordination needs to balance “letting it emerge” and “making it happen.” Organizational/educational support and using residents as change agents are both very helpful in accelerating the implementation process.

Table 1  Changes introduced into the Dutch PGME curricula

The RDMA introduced in 2004 the following changes into PGME in the Netherlands:

• The CanMEDS framework. The entire curriculum, including the learning goals, building blocks, assessment strategy and instruments, teaching activities, and materials, should be oriented to a set of seven roles for practicing specialists, derived from societal needs

– medical expert

– communicator

– health advocate

– collaborator

– manager

– scholar

– professional

• The mini clinical evaluation exercise (Mini-CEX); frequency – 10 observations each year

– method to assess competencies in real-life clinical practice (Norcini et al. 2003)

– consists of a short observation by a qualified medical specialist of a resident demonstrating clinical skills, using a pre-defined scoring format, followed by a structured feedback conversation (Pendleton et al. 2003)

• The portfolio, including periodical interviews between the program director and residents about the portflio; frequency – 1st year five meetings, 2nd and 3rd year – three meetings, and 4th and 5th year – two meetings

– in the portfolio, residents collect evidence documenting personal development according to the CanMEDS roles

– the results from the assessment instruments and evidence of acquired knowledge and skills are assembled, along with results of repeated reflection sessions on acquired competencies and a personal strengths–weaknesses analysis (Scheele et al. 2008)

• The critical appraised topic; this includes the systematic search of scientific evidence for a clinically relevant question; frequency – twice each year

• Knowledge progression tests; frequency – each year

From 2006 to 2010, as a pilot implementation project for all medical specialties, the revised competency-based PGME curricula for Obstetrics & Gynecology (O&G) and Pediatrics were the first to be implemented in the Netherlands through a nationwide project called “In VIVO.” Supported by a national project team (national level), the implementation process was guided by dedicated regional implementation teams in each of the country's eight teaching and training regions (regional level). Residents – medical specialists in training – in the O&G and Pediatrics programs follow their training in a teaching and training region; these consist of university medical centers and general hospitals, which collaborate in providing PGME. Program directors in approximately 70 training units were responsible for the implementation of the new curriculum in their departments (local level). Figure 1 presents the organization of the In VIVO implementation process.

Figure 1. Organization of the In VIVO implementation process.

One of the major conclusions of an extensive literature review into diffusion of complex innovation in health service organizations was that the current literature lacks in-depth empirical assessments and descriptions of complex implementation processes in health care (Greenhalgh et al. 2004, 2008, 2010). Other reviews confirmed the need for a research shift from “what works” to “what works where and why” (Damschroder et al. 2009). We believe that the In VIVO project can be viewed as a complex implementation process (Damschroder et al. 2009). Many stakeholders with different backgrounds were involved, including health care professionals, educationalists, medical scientific societies, the government, the RDMA, and the hospitals. The implementation concerned two different medical specialties and different levels (nationwide, regional, and local) of organizing the implementation process. Finally, the implementation of PGME was closely connected to health care processes, because residents learn while working as a medical doctor in the hospitals. Implementation of a renewed PGME curriculum, therefore, requires changes in the way health care processes are structured and vice versa, increasing the complexity of the process (Damschroder et al. 2009).

To understand the implementation of a competency-based PGME curriculum and to provide recommendations for health care professionals and policymakers to tailor their interventions in these complex implementation processes, we conducted a qualitative study evaluating the process in the In VIVO project. A qualitative approach was chosen in order to allow a detailed and context-rich description of the complex implementation process at the national, regional, and local levels, along with a description of the factors influencing this process. Our research question was: Which conditions promoted and impeded for what reason the implementation process on the different levels of the project?

Methods

Over a 6 month period in 2010, we conducted 25 semi-structured, in-depth interviews with participants from the national, regional, and local levels of the project. The respondents were strongly encouraged to state their opinions freely and comprehensively. Theoretical sampling was used to select the respondents (Kennedy & Lingard 2006). The sampling procedure and the interview topics are listed in Table 2.

Table 2  Sampling procedure and interview questions

	Goals	Respondents	Interview topics/questions
National level	• Gain overall view of the project • Gain insight into effects of national and regional implementation strategies and processes, and the factors influencing them • Investigate different implementation strategies and regional implementation teams	Interviews with members of national project team (N = 3) • project leader • educationalist • researcher	• What were the goals and results of the project • Was the project structure effective in achieving the goals and results • What factors positively and negatively influenced the implementation process • What were the differences between regions in terms of the implementation strategy
Regional level	• Gain insight into effects of national and regional implementation strategies and processes and the factors influencing them • Investigate whether some program directors and departments in the region were more innovative and active in the project	Interviews with members of two regional implementation teams (N = 10) • project leader in O&G • project leader in Pediatrics • two residents • educational specialist Regional implementation team A • Highly structured regional implementation process; regional implementation meetings were organized with all program directors in the region • Located in low-density population of people and hospitals Regional implementation team B • Weakly structured regional implementation process; no structural regional implementation activities were executed • Located in a dense population of people and hospitals	• What were the goals and results of the project • What was the implementation strategy in your region • What factors positively and negatively influenced the implementation process • What are the main differences between before and after the project • How has the project contributed to these differences • Were there differences between the departments in your region with regard to innovation and activity
Local level	• Gain insight into effects of regional and local implementation strategies and processes and the factors influencing them	Interviews with members of two departments in the two regions (departments A and B) (N = 12) • program director • supervisor • resident The two departments differed in their innovation and activity with regard to the regional implementation team and project	• What were the goals and results of the project • What was the implementation strategy in your department? • How do you perceive the new curriculum and assessment instruments • What are the main differences between before and after the project • How has the project contributed to these differences • What factors positively and negatively influenced the implementation process

Established ethical standards were used to guide the research procedure (Eva 2009; Ten Cate 2009). All participants gave informed consent. All interviews were recorded and transcribed verbatim (with participants identified by their roles). We used directed content analysis to analyze the interview texts (Hsieh & Shannon 2005). In contrast to grounded theory approaches, in which the coding schemes are developed during or after data collection, directed content analysis uses a theory-based coding scheme that is constructed before data collection. Directed content analysis is considered appropriate when useful theoretical models are available and the purpose of the study is leaning more toward application of existing theory and less to building new theory. To guide the coding of interview data and to report our results, we used a multi-level theoretical framework of complex innovation in health services organizations, constructed by Greenhalgh and colleagues. This framework is the result of the most extensive systematic review currently available in the field of complex innovation in health care. After searching 6000 sources from different research traditions, nearly 500 sources of empirical evidence were included (Greenhalgh et al. 2004, 2008, 2010).

The first two interviews were coded independently by four researchers, and differences in the coding scheme were resolved by discussion and consensus. Subsequent interviews were coded independently by two of these researchers, and all differences in coding interpretation were resolved in the same manner. The principal investigator used these codes to construct a framework describing the implementation process and the factors influencing it. The quotations and the procedure revealed a highly consistent pattern of factors influencing the implementation process. After a group discussion, the final framework was approved by all authors (Hsieh & Shannon 2005).

Results

Following from the directed content analysis, the factors influencing the implementation process of the PGME curricula found in this study were divided into three categories: attributes of the innovations and adopters, attributes of the implementation process, and attributes of the organization (Greenhalgh et al. 2004). In the following sections, we address these factors and discuss in more detail whether these factors had an impeding (barrier) or a promoting (facilitator) effect in the project (Table 3).

Table 3  Main attributes found to be important for PGME innovation

	Facilitator to implementation (what and why it worked as a facilitator)	Barrier to implementation (what and why it worked as a barrier)
Attributes of the innovation and adopters
Relative advantage of:
CanMEDS roles^a	The adopters attached some advantages to the use of this innovation. More attention to generic competencies in the assessment of residents was expected to have positive impacts on the improvement of health care quality and processes, like patient safety, collaboration, and communication and organization. Furthermore, the use of the CanMEDS roles provided a language between program directors, supervisors, and residents to discuss the progress of the residents	The program directors, supervisors, and residents attached clear disadvantages to the use of this innovation: The CanMEDS roles proved difficult to operationalize, implement, and assess – especially the non-medical competencies – in workplace-based training. This was caused by the abstract nature of competencies, a tendency to focus on easily assessable competencies and lack of time, training and evidence
Behavioral changes supervisor: structured feedback^a	The program directors, supervisors, and residents attached clear advantages to the use of this innovation. More structured feedback led to more balanced (both positive and negative comments), efficient, and effective feedback encounters between supervisor and resident	The program directors, supervisors, and residents attached some disadvantages to the use of this innovation. Giving and receiving feedback was perceived as difficult: there was fear of damaging the relationship and fear of negative feedback. Also, the quality of feedback by supervisors was sometimes perceived as worrisome by residents
Behavioral changes resident: pro-active attitude^a	The adopters attached clear advantages to the use of this innovation. The residents became more aware of their learning objectives	The residents attached some disadvantages to the use of this innovation. The residents felt sometimes disappointed because their learning objectives could not be accommodated due to the complexity of work schedules. Some supervisors considered the learning goals far too specific
Assessment instruments^a	The adopters attached some advantages to the use of these innovations. The use of the Mini-CEX produced an improved, standardized, and more objective assessment of clinical skills that allowed more explicit, focused, and balanced reflection on the clinical behavior of residents in specific situations. The portfolio enhanced the opportunity of residents to reflect on their own development and to construct their own training	The program directors, supervisors, and residents attached some disadvantages to the use of this innovation. The accomplishment of a certain number of assessments did become a goal in itself, although it should be a means to facilitate feedback. Organizing a Mini-CEX took a considerable time and effort. The potential of the portfolio was limited due the subjective nature and the limitations of the residents’ ability to shape their own training
Changes educational program^a	The adopters attached some advantages to the use of this innovation. More continuity in the internships led to more opportunities of supervisors to observe residents and it led to steeper learning curves of the resident	The program directors, supervisors, and residents attached some disadvantages to the use of this innovation. Implementation was sometimes difficult due to understaffing of supervisors and residents
Simplicity and compatibility	The innovations “structured feedback by supervisor” and “pro-active attitude resident” were fairly simple to comprehend and they were compatible with the values, norms, and perceived needs of the adopters. These innovations were easily implemented because relatively few additional resources were required to implement them in the work-based training	The innovation's “assessment instruments,” “changes educational program,” and “CanMEDS roles” were fairly difficult to comprehend and they were not so compatible with the values, norms, and perceived needs. These innovations were less likely to be implemented because they required a lot of effort to organize and implement and they intervened with daily health care processes
Attributes of the implementation process
External developments	The forces that were outside of control of the national project were made to work for the implementation process, rather than allowing them to work against the process. The increased attention to educational quality allowed the national project team to start pilot projects introducing an educational quality care system into PGME	Allowing the forces that were outside of control of the national project to work against the implementation process. The introduction of a national medical training fund generated skepticism among the program directors, as it remained unclear how the quality was supposed to be measured. This had a negative impact on the implementation process, because adopters associated this process with the allocation of governmental funds and it consumed a lot of valuable time. The reduction in the number of pediatrics training positions during the project decreased motivation and increased the workloads of supervisors and residents, thus allowing less time for the implementation of the innovations
Appropriate change model	“Letting it happen” was facilitated by the national project team by stimulating regional implementation teams and program directors to entrepreneurship. Entrepreneurship caused regional implementation teams and program directors to take creative action and to let them feel responsible for the implementation process	Too little “making it happen” by the national project team. The program directors, supervisors, and residents perceived that they received too little guidance on the implementation process from the national project team in the beginning of the project. The adopters felt that they were turned loose and that is was unclear what was expected. This slowed down the process
Good project management	The flexibility in the project management by the national project team and the absence of objectives to be reached at certain points in time. The organic implementation process allowed for the incorporation of relevant developments. This increased the acceptance among the adopters because they perceived as having control over the process	The absence of explicit goals, a clear time frame, and monitoring of the results by the national project team. The adopters felt that they were turned loose and that it was unclear what was expected
National and regional implementation strategies and activities^a	Well-organized national and regional meetings/activities with clear goals and relatively high returns for their time investment. Such meetings and activities generated knowledge sharing and a mutual understanding of the process	The absence of or poorly organized national and regional meetings/activities. The absence of such meetings slowed down the implementation process because opportunities to share knowledge were missed. Poorly organized meetings left adopters frustrated because they experienced relatively low returns for their time investment
Educational support^a	The presence of well-organized educational support and competent educationalists. Educational support and educationalists significantly accelerated the implementation process by their specific expertise and knowledge, by providing concrete instructions for the use of the assessment instruments, helping to translate the national curriculum to workplace-based training, organizing the process, assisting in setting up regional meetings and activities, and exchanging best-practices with their colleagues in other hospitals and regions. This support and expertise complemented the expertise of the busy physicians	The absence of well-organized educational support and competent educationalists. The absence slowed down the implementation process. It meant that all support for the implementation process had to be delivered by busy physicians who are in general not trained in educational and change management issues
Human resource factors: training	The provision in the early phases of the implementation process of well-organized training preferably for all adopters in a department aimed at implementing the innovations. Training generated a more positive attitude toward the innovations, led to knowledge sharing, enhanced the educational knowledge and teaching behavior of the doctors, and caused improvements in the clinical learning climate. Entering the training into the early phases allowed the adopters to pick up the innovations early and more easily	The absence of training, the provision of training to some adopters in a department, or the provision of training in the later phases of the implementation process. The absence of training or the provision of the training in the later phases slowed down the process. Providing training to a subset of the adopters in a department caused a sub-optimal learning climate
Interpersonal influence	Facilitate knowledge sharing between program directors, effectively use social networks to spread the innovations and use residents as change agents. Knowledge sharing between program directors led to mutual understanding of the process and led to the exchange of best-practices. The effective use of residents as change agents enhanced their motivation and efforts of knowledge diffusion to their fellow residents	The absence of or suboptimal knowledge sharing and use of residents as change agents. Suboptimal knowledge sharing between program directors slowed down the process because opportunities to exchange best-practices were missed. Not using residents as change agents by their program directors caused these residents to feel left-out, it reduced their motivation and it reduced their efforts to diffuse knowledge of the innovation to their fellow residents, or in a worse case scenario, it caused them to diffuse negative opinions of the innovations to their fellow residents
Attributes of the organization
Leadership and management	The presence of good leaders and management at all levels of the projects. The implementation process was accelerated by the presence of good leaders (project leaders and program directors); they were perceived as role models, they inspired, were ambitious, experienced high intrinsic motivation, were entrepreneurs, and got things done	The absence of good leaders, the lack of “substitute” leaders, and the lack of investment in new leaders at all levels of the projects. The implementation process was slowed down by the absence of good leaders (project leaders and program directors): relatively few initiatives and creative action were undertaken and supervisors and residents lacked good role models to inspire them
Slack resources and support management	The support of management (on all management levels) in providing slack resources, especially educational support. Program directors, supervisors, and residents highly appreciated the positive reinforcement by management, such as the Board of Directors. The implementation process was significantly accelerated if the management was willing and able to provide resources, especially educational support	The absence of support of management (on all management levels) in providing slack resources, especially educational support. The implementation process was significantly slowed down if management was unwilling or unable to provide resources. The adopters were less likely to invest into the process without the positive reinforcement of management
Effective data capture and feedback systems	The application of questionnaires by the national project team to monitor the implementation process. These proved useful for providing feedback on the implementation process and provided a new push for the process	The questionnaires provided only quantitative data (mostly five-point scale). The lack of qualitative data made it hard to interpret the information acquired
Tension for change and balance between supporters and opponents	High value attached to medical education, in general, and high tension for urgency and need to implement the innovations more specifically, as perceived by preferably all adopters in a department. Adopters were more likely to implement the innovations if they attached high value to medical education. “True believers” in the new learning philosophy experienced high intrinsic motivation and functioned as role models for their fellow supervisors and residents. Residents experienced higher motivation and had more fun when program directors and supervisors also felt responsible and took initiative to execute the instruments	Low value attached to medical education in general, and low tension for urgency and need to implement the innovations more specifically, as perceived by all or some adopters in a department. If some or many supervisors and residents in the department considered education less important than research or medical care, adopters were less motivated to implement the innovation. The extent to which implementation was rewarded, supported and expected within the organizations was found to be a driver of adoption of the particular innovation
Difference between general and university hospitals^a	Small departments from general hospitals with cohesive communication and decision-making structures on the one hand and effective regional collaboration structures between general and university hospitals on the other hand. Communication and decision-making processes in the smaller departments were more efficient, and members felt more responsible for implementing the innovations than did their counterparts in the larger university hospital departments. University hospitals were perceived as being more in the lead in terms of the regular regional educational collaboration between program directors, the regional allocation of resident training positions, and educational expertise. The implementation process was accelerated by effective regional collaboration structures between program directors that consolidated the strengths of both types of hospitals	Large departments from university hospitals with fragmented communication and decision-making structures on the one hand and the absence of or ineffective regional collaboration structures between general and university hospitals on the other hand. Communication and decision-making processes in the bigger departments were less efficient. The implementation process was slowed down by the absence of or ineffective regional collaboration structures between program directors of general and university hospitals. Knowledge sharing and best-practices exchange opportunities were missed, which slowed down the process
Balance training and patient care^a	Departments that found a good balance between the personal learning objectives of the resident and the workload of patient care. Preferably, with the renewed curriculum, every resident should have received a tailor-made training according to the required learning objectives. This ambition was limited by the high workload of patient care and the size of the resident group. Furthermore, working under high pressure also provided learning opportunities. A good balance had a facilitating effect on the implementation process	Departments that had a misbalance balance between the personal learning objectives of the resident and the workload of patient care. In most cases, the misbalance is hanging toward doing patient care. Residents missed valuable learning opportunities and were not accomplishing their personal learning objectives. This slowed down the implementation process

Note: ^aSpecific feature of PGME innovation within the Greenhalgh framework, as described in the discussion section of this paper.

Table 4  Recommendations to tailor interventions in the context of PGME innovation

Level^a

The challenge of implementing the CanMEDS roles • Translate the national curriculum to a regional and local curriculum that specifies the CanMEDS roles, procedures for work-based learning, and assessment in the context of patient care • Design the curriculum with the needs of the primary users (program directors, supervisors, and residents) in mind to make it more attractive for them to adopt the innovation • Align the benefits of the renewed curricula directly with the outcomes of health care processes, such as patient safety and team collaboration and communication Regional implementation strategies and educational support • Create coordination of the implementation process (by specialty) • Establish goals, establish a timeframe, and monitor the results in concordance with the professional • Balance “letting it emerge” and “making it happen,” depending on implementation progression • Use training as an implementation tool (to raise awareness among users and to learn the required skills and knowledge) • Facilitate knowledge sharing through meetings and social networks • Use residents as change agents • Identify and reward good leaders and good teams; ensure continuity • Organize logistic organizational and educational support Need for regional inter-organizational networks of hospitals • Build strong regional collaboration networks between university and general hospitals Balance between training and patient care • Achieve a careful balance between education (work-based learning) and patient care

N, R, L N, R, L N, R, L N, R, L N, R, L N, R, L R, L N, R, L R, L N, R, L N, R, L R, L L

Notes: ^alevel at which it is important to apply the practice points: N, national; R, regional; and L, local.

Attributes of the innovation and adopters

The interviewed subjects identified the following (sub-)innovations as the four most important: the explicit use of the CanMEDS roles (Table 1), a behavioral change of supervisors (structured feedback) and residents (pro-active attitude), the introduction of new assessment instruments, and the modified and more explicit structure of local training programs. We report the (sub-)innovations in terms of their relative advantage, simplicity, and compatibility with values, norms, and needs of the adopters.

Relative advantage

Although they did acknowledge that the new curriculum prompted more attention to competencies other than medical skills, respondents also expressed the view that the intended full operationalization, implementation, and assessment of all the CanMEDS roles was only in its initial stage. Reasons included the abstract nature of competencies (e.g., health advocacy (Ringsted et al. 2006)), a tendency to focus on easily assessable competencies, lack of time and lack of training.

SV (PD = program director, SV = supervising medical specialist, RE = resident, NPT = national project team): I think that it's more an impression of whether all of these factors are decisive, or whether a person appears somewhat professional …  It's not just in boxes of “Okay, this, this, this, and then” – all seven competencies in a row – it just doesn’t work that way.

Skeptical adopters also mentioned the lack of evidence for the CanMEDS roles. The adopters’ perception of the quality and validity of the evidence supporting the innovation can influence their adoption decision (Damschroder et al. 2009).

PD: But I keep wondering who has actually proved that this really does make things better. Is this really a positive innovation, or could it cause problems for us in five or ten years?

For supervisors, the most striking behavioral change was that they shifted from providing implicit or no feedback to providing explicit, safe, structured, and repeated feedback on the skills and abilities of residents using a new feedback technique (Pendleton's rules (Pendleton et al. 2003)).

RE: … that, at the end of a shift, supervisors were more likely to say, “Okay, that went well, and be sure to think about this next time” … in the past, no news was good news.

Although the interviewed adopters expressed a very positive general attitude toward the new structured feedback process, they also identified some difficulties related to giving and receiving feedback, including fear of damaging the relationship, fear of negative feedback, and the poor quality of their supervisors’ educational skills.

PD: … people who are evaluated by others never like to be called down, regardless of how nicely you might pack it in a Pendleton formula of “What went well and what could be better?”

The most striking change in the residents’ behavior was the development of a pro-active attitude aimed at shaping their own training, instead of passively following it.

RE: … you’re also more conscious about thinking … what would I like to do here in the next four months, and what would I like to retain? I think this makes you more aware of your learning objectives.

It was also acknowledged, however, that the complexity of work schedules and the structure of the local educational program limited the opportunities for the residents to shape their own training.

RE: … this is your schedule, and then you just do what you have to do, whether you’re good at it or not; it doesn’t matter because, well, we can’t schedule somebody else anyway.

The clear advantages that the respondents attached to the behavioral changes (explicit safe, structured, and repeated feedback by the supervisors and pro-active attitude of the residents) had a facilitating effect on the implementation process.

The introduction of structured and scheduled assessment instruments (e.g., the Mini-CEX (Norcini et al. 2003), Table 1) was perceived as important. Most respondents mentioned that the use of these instruments had produced an improved, standardized, and more objective assessment of clinical skills that allowed more explicit, focused, and balanced reflection on the clinical behavior of residents in specific situations.

SV: … because you’re really observing what's going on there, right? … you observe the whole situation, from the quality of eye contact to how calm someone is; you also look at things that aren’t purely medical …

Adopters who were more skeptical expressed concern that the accomplishment of a certain number of Mini-CEX assessments had become a goal in itself, although it should be a means to an end. These respondents also noted that considerable time and effort were involved in organizing a Mini-CEX.

RE: … and then you write an objective for one Mini-CEX per month, even though … it's obviously not all about a Mini-CEX every month; that defeats the purpose …

The introduction of the use of the portfolio generated a variety of comments. Proponents pointed out the enhanced opportunity to reflect on their own development and to construct their own training.

RE: … it really does make you more conscious of “Alright, what can’t I do” or “What would I like to learn more about?” … and you see that you have the opportunity to do certain things during your internship.

Respondents who were more skeptical pointed out the subjective nature of the portfolio and the limitations to learners’ ability to shape their own education.

RE: … you expect inexperienced residents to formulate objectives in areas in which they are not yet comfortable, and I think that's asking too much.

The changes to the structure of the local educational program were considered an important innovation.

RE: And now that we’re working with this model, the program directors have also become more clear about the necessity of continuity in the internships in order to realize this plan.

In summary, respondents mentioned several advantages and disadvantages to the assessment instruments and the changes to the structure of the local educational program, as discussed above. Due to the lack of either a clear advantage or disadvantage, the effect to the implementation process was neutral.

Simplicity and compatibility

The most easily adopted business innovations are those that have a clear advantage, those that are compatible with the values, norms, and perceived needs of the adopters, and those that are perceived as simple (Rogers 2003). Many respondents felt that the implementation could have been more successful if some innovations had been simpler and more tailored to the existing work processes. The high complexity of the innovations, as perceived by the adopters, had an impeding effect on the implementation process.

RE: … it shouldn’t be so grandiose and complex; it ought to be able to fit into the existing structure in some way …

Attributes of the implementation process

External developments

The implementation process was influenced by a number of external developments. The government introduced a national medical training fund and announced that the allocation of funds would be increasingly dependent upon the quality of the PGME provided. This generated skepticism among the program directors, as it remained unclear how this quality was supposed to be measured. On the one hand, this had a negative impact on the implementation process, because program directors, supervisors, and residents associated this process with the allocation of governmental funds. On the other hand, the increased attention to educational quality allowed the national project team to start pilot projects introducing an educational quality care system into PGME. Another external development was the reduction in the number of pediatrics training positions during the project. This decreased motivation and increased the workloads of supervisors and residents, thus allowing less time for the implementation of the innovation. Despite the fact that these factors were very important, the overall effect for the implementation process was neutral.

PD: … many changes at the same time for the same occupational group … well, you can pay attention to only one thing at a time. The discussion about the quality indicators was also really unpleasant, because there was still the feeling that “it was not about quality” …

Appropriate change model

Every situation of change requires a balance between “letting it emerge” and “making it happen.” Approaches oriented toward letting the change emerge (diffusion) are essentially passive, and the key mechanisms are contagion and imitation (Greenhalgh et al. 2004). In contrast, dissemination (“making it happen”) is a planned and active process intended to increase the rate and level of adoption above what might have been achieved by diffusion alone. The interviews showed that the national project team was struggling to find the best way to strike this balance. In retrospect, the supervisors and residents thought that the national project team should have adopted an approach that tended more toward “making it happen,” than “letting it emerge;” therefore, the chosen strategy had a slightly impeding effect on the implementation process.

PD: … another thing that I noticed in the beginning is that we – and I said this to the national project team – that we were turned loose. We were not told in clear terms what was actually expected of us.

The national project team, however, felt unable to adopt a “make-it-happen” approach, as the team had no formal (i.e., hierarchical) authority to do so.

NPT: We have absolutely no power to apply rules and sanctions. We noticed that it was much more effective to stimulate the entrepreneurship of the program directors …

Good project management

An implementation project can benefit from effective project management, including the tasks of setting explicit goals, defining a clear time frame, and monitoring results (Greenhalgh et al. 2004). Although the goals that were set initially remained intact during the project, they were modified and extended by the introduction of quality criteria for PGME.

NPT: Then it occurred to us, “How can we make forces that are outside of our control work for us, rather than allowing them to work against us?” This was already included in the project plan, but it wasn’t worked out until then … How can we guarantee quality, quality checks?

The interviewees felt that the implementation process was not structured with objectives to be reached at certain points in time. Instead, they perceived it more as an organic process, incorporating relevant developments. Although the loose project management might have had this advantage, overall, the implementation process could have benefited from a tighter project management.

PD: I actually thought that the implementation of the In VIVO project would have a clear beginning and end, but that turned out not to be the case. It does have a beginning, but the process is far from complete.

National and regional implementation strategies and activities

Regional activities (such as organizing regional meetings on a regular basis with all program directors, supervisors, and residents to discuss the renewed curriculum, implementation facilitators, barriers, and best-practices) were valued more than national ones; however, both had a facilitating effect on the implementation process. The regional meetings generated knowledge sharing and a mutual understanding of the process:

PD Region A: All of the cluster partners were invited, it was well organized, and we could receive concrete information about things that we could change.

RE Region A: … the fact that we got together as a group and conducted the evaluation department by department, what's good and what could be better. I think that was one of the strongest things that took place.

PD Region B: Well, we were – the regional implementation, we didn’t … really stay involved, so we were intensively involved with the implementation in our own hospital.

Educational support

The ability of the regional implementation teams to execute regional and local activities was dependent on support by educational specialists. More successful innovation processes receive ongoing and adequate support in the form of resources (Gustafson et al. 2003).

PD: He was indispensable for the entire project … He knows a lot, he is an excellent instructor, and he knows how to keep things on the right track. Yes, I think that he played a crucial role …

According to the national project team, implementation could have been more successful if the program directors had received proper organizational support. The implementation process was significantly accelerated wherever such support was arranged.

NPT: Those program directors were not accustomed to the tasks which they were given … And then, of course, we also had a lot of setbacks because the program directors were trying to do everything as well as they could, but were actually not very good at that type of management.

Human resource factors

Well-organized training in educational methods and the use of the assessment instruments proved useful in accelerating the implementation process. Training generated a more positive attitude toward the innovations, and it led to knowledge sharing, enhanced the educational knowledge and teaching behavior of the doctors, and caused improvements in the clinical learning climate (Rubak et al. 2008).

RE: … holding the Teach-the-Teacher course for all faculty and the residents in the early phases of the program … that makes the learning environment much safer and more conscious.

Interpersonal influence in all layers of the project

The national and regional implementation activities were effective mechanisms for communication, interpersonal influence, knowledge sharing and learning, and social networking. This is consistent with previously published work on the importance of communication and interpersonal influence in innovation processes (Rogers 2003), including processes in the context of undergraduate medical education (Bland et al. 2000), and PGME (Lillevang et al. 2009).

PD: … make sure that your counterparts tell you how they are faring with the implementation. Don’t do it alone … try to hear what others have to say, and whether they are referring to the national or the regional meetings. Then you don’t have to think up as much on your own, and you don’t have to try out everything first hand.

Our results illustrate the importance of the change-agent function fulfilled by the residents. The effective use of these change agents enhanced the diffusion of knowledge to their fellow residents and acted as facilitators to the implementation process. This provides support for research on the effectiveness of peer opinion leaders, who exert influence through their representativeness and credibility (Fitzgerald et al. 2002).

PD Region A: … one thing that I really appreciated was that it was truly carried out together with the residents. I think that if program directors had just thought this up without them, it would not have been as good, and it would not have been as easy to implement.

PD Region B: I think that the residents were not as motivated as they could have been, because we ignored them to some extent. … And that would have been quite different if we had started two years ago …

RE Region B: … the program directors expected more input from the residents, and then I thought, “Yeah, right. You’ll be happy if we just take the minutes.”

Attributes of the organization

Leadership and management

Our results support the importance of leadership and management (Gustafson et al. 2003) to innovation in health care and undergraduate medical education (Bland et al. 2000). Among other positive attributes, good leaders are role models: they inspire, are ambitious, experience high intrinsic motivation, are entrepreneurs, and get things done. The implementation process was accelerated wherever such support was present.

PD: In my opinion, you have to set an example in this respect; you have to do it yourself, you have to show that it's fun, that it's good, so that the residents will think it's fine. So you do have to create an atmosphere of enthusiasm.

Instead of searching for a program director who possesses all these qualities, these qualities should be present in the team of supervisors.

PD: Don’t be the only one who wants it, because then it won’t work. Let everyone evaluate the residents; let everyone conduct feedback meetings …

Slack resources and the support of management

The support of management (on all management levels), especially in terms of providing educational input (a slack resource), was helpful for accelerating the implementation process (in agreement with, e.g., Gustafson et al. (2003)).

PD: … the educational support is possible because our Board of Directors considers education important … otherwise, we wouldn’t have had that educational capacity.

Effective data capture and feedback systems

Good teams reflect upon their actions (Gustafson et al. 2003; Damschroder et al. 2009). The application of questionnaires by the national project team to monitor the implementation process proved useful for providing feedback and had a facilitating effect on the implementation process.

NPT: So all of those moments when something happens again – when you make something visible – they provide a new push.

Tension for change and balance between supporters and opponents

Every supervisor and resident in the team should ideally feel the urgency and need to implement the innovations (Gustafson et al. 2003). The need for change has been found to be an important factor in the success of curricular changes in medical schools (Bland et al. 2000). Every team needs at least some adopters who are “true believers” in the new learning philosophy and who experience high intrinsic motivation, as studies have shown that such “true believers” are more likely to adopt the innovation (Gladwin et al. 2002).

PD: I really liked that, because I stand behind the objectives of the project: the implementation of modernization – which I consider useful. I think that the doctors and society will ultimately benefit from it – from doctors who finish their education more quickly.

One barrier to change was the feeling that many colleagues in the department considered education less important than research or medical care. The extent to which implementation of the new curriculum and use of the innovations was rewarded, supported and expected within the organizations was found to be a driver of adoption of the particular innovation (Gershon et al. 2004).

PD: As long as science and health care are the most important pillars … that is how we are evaluated; your CV doesn’t say much about education; there aren’t any prizes for that … so that's always a struggle.

Residents experienced higher motivation and had more fun when program directors and supervisors also felt responsible and took initiative to execute the instruments. This is consistent with recent work on portfolio mentoring (Dekker et al. 2009).

RE: … in the past, I’ve had a program director who would say, “Oh, yeah. Portfolio. I haven’t read it.” … that doesn’t motivate people to complete them, if it's not going to be read anyway … Then it just feels like you’re doing it for nothing.

Difference between general and university hospitals

Because of the smaller size of their faculty, general teaching hospitals appeared to be at an advantage in the implementation of PGME modernization: communication and decision-making processes were more efficient, and members felt more responsible for implementing the innovations than did their counterparts in the larger university hospital departments.

RE: one advantage of the general hospital is that you’ve got a smaller group of people and … that people feel responsible for actually doing it. And that provides motivation to do it …

Conversely, university hospitals were perceived as being more in the lead in terms of the regular regional educational collaboration between program directors, the regional allocation of resident training positions, and educational expertise (e.g., educationalists). The leading role played by large, specialized organizations in assimilating innovation has been described previously (Greenhalgh et al. 2004). Therefore, this attribute was a barrier and facilitator, according to the loco-regional organization.

PD: The university hospital obviously takes the lead quite often in a number of matters. This sometimes gives those in the general hospitals the impression that they are straggling behind.

PD: One thing that we always project … is that our region should be strong in its postgraduate educational programs. Because we organize our postgraduate educational programs regionally – this doesn’t apply only to us, it applies to all of the specialties as well – our region needs to be so good that, in a few years when the residents can choose, they will say, “I’d like to go to that region, because everything is really well organized there.” In this respect, it's thus more in our own interest to cooperate than it is to compete.

Balance between education and patient care

Preferably, in competency-based education, every resident receives a tailor-made training according to the required learning objectives. However, this ambition is limited by the organization of patient care, high workload, and the size of the resident group. Therefore, good teams strive to achieve a balance between personal learning objectives of the resident and patient care. The departmental culture defined the result of the balance.

RE: You’re obviously always going to be left with the fact that a large group of residents who have to complete the program in six years, and you’ve just got to see lots of patients.

However, working under high pressure also provided learning opportunities.

PD: A part of your program involves learning how to function under conditions of stress, how to take responsibility in such situations, because you’ll have to do that later in practice.

Discussion

This qualitative study describes the implementation process of the competency-based PGME curricula for O&G and Pediatrics. We observed three interrelated groups of factors that functioned as facilitators or barriers to the process: attributes of the innovations and adopters, attributes of the implementation process, and attributes of the organization (Table 3). The factors influencing the implementation process identified in our study were comparable to factors that have been described previously in the Greenhalgh framework of complex service innovation in health care (Greenhalgh et al. 2004). We prefer to concentrate our ‘Discussion’ section on four groups of specific features of innovation in the context of PGME which were not documented before. These features fit within the factors of the Greenhalgh framework; therefore, they are not, in general, to be perceived as new, and rather, their importance concerns the specific description of their application in the context of innovation in PGME. Our results showed the context to be the dominant factor in complex innovations like implementing a new PGME curriculum. In this specific innovation, the local/regional context determined the speed, quality, and direction of the implementation process, the extent to which the innovations were implemented, and how a factor affected the process. All factors identified in our study were to some extent promoting or impeding to the implementation process dependent on the specific circumstances and context (Table 3).

The challenge of implementing the CanMEDS roles

The CanMEDS roles (Table 1) proved difficult to operationalize (or to translate), to implement and to assess – especially the non-medical competencies – in workplace-based training. This can be explained by looking at the attributes of the innovation, implementation process, and organization. First, the innovation that had to be implemented was under construction; in fact, it was a semi-finished product. A national curriculum was written but a blueprint on how to apply the CanMEDS roles in work-based training was lacking. Users attached some advantages to this innovation (e.g., the potential benefits for the improvement of health care and society), but most users perceived the application in work-based training as very difficult, especially the non-medical competencies (e.g., they were perceived as artificial). This caused users to refrain from the implementation of the CanMEDS roles and to concentrate on the most easily adoptable innovation with the highest perceived benefits; these are the behavioral changes by the supervisor (structured feedback) and the resident (pro-active attitude). In general, these behavioral changes were considered as having significant advantages and may therefore be perceived as a positive behavioral change from an educational perspective. Second, the implementation process lacked clear project management to monitor the troublesome implementation of the CanMEDS roles, and moreover the national project team was unable to attach more to a “making it happen” approach. The national project team succeeded in stimulating the regional implementation teams and program directors into entrepreneurship (“letting it happen”); however, this was not enough to counter the implementation problems. Also, the teachers (program directors and supervisors) were not trained in educating the CanMEDS roles; the training they were being provided with was mainly oriented toward providing more structured and safe feedback. Third, from the perspective of the organization, a real sense of urgency to implement the innovations was lacking by the target audience: the program directors, supervisors, and residents. They felt the innovations were important, but in general, short-term issues (e.g., managing day-to-day health care operations and doing research) were considered more important. It proved difficult for adopters to look beyond the short-term issues into the long-term goals of the new curriculum; these are the delivery of high quality and transparent health care provided by competent health care professionals.

Regional implementation strategies and educational support

Our results showed (Table 3) that residents, supervisors, and program directors strongly appreciated any educational (and organizational) support provided, and they attached high value to regional implementation activities. The provision of both educational support and regional implementation activities at the regional and local levels, along with the customization of these activities to the specific needs of the adopters and the situation (current or change) can enhance the success of the implementation process. It has been shown that such targeted support accelerates and facilitates processes of implementation (Rogers 2003). However, in a PGME curriculum reform in Denmark, educational support had in some aspects an impeding impact on the process (Lillevang et al. 2009). As in the Netherlands, in Denmark, the professionals required and requested additional educational support for implementing the renewed PGME curricula. According to the authors, this incongruence may be subscribed to different roles and capabilities of some of the educational advisors in Denmark.

Balance between training and patient care

Although PGME is intertwined with the delivery of health care (residents learn while working), medical education has always been regarded as less important than health care delivery and research. Our results showed (Table 3) that successful implementation required medical professionals to balance the importance of a given innovation against the constraints that it will impose on their other tasks. To do this, organizational changes were needed. Measures some program directors took were: scheduling residents in such a way that they can accomplish their own learning objectives, increasing the duration of clinical rotations to allow for longitudinal observation and steeper learning curves, and scheduling regular feedback encounters between program director and residents.

Need for regional inter-organizational networks of hospitals

Our results showed (Table 3) the importance of collaboration between university and general hospitals in implementing and providing PGME at various levels: between program directors within the same specialty, in providing regional educational support, and with respect to managerial collaboration. The implementation process benefited from strong networks of collaboration at all these levels. Integrative inter-organizational networks with good governance structures and explicit shared values and goals can help disperse innovations among member organizations (Greenhalgh et al. 2004).

Strengths, limitations, and suggestions for further research

The main strength of our study is that it is among the first to provide an in-depth empirical assessment and description of the implementation process of competency-based PGME. The theory-driven selection of respondents of various backgrounds from within the various layers of the project allowed a context-rich assessment and description of the complex implementation process. Although these rich descriptions are highly valued, they are currently lacking in the literature on innovation (Greenhalgh et al. 2004). Our findings thus confirm and extend existing research into complex innovation processes in the context of health care, undergraduate and PGME (Bland et al. 2000; Fleuren et al. 2004; Greenhalgh et al. 2004; Lillevang et al. 2009). The main limitation of our study involves the relatively small number of interviews. For theoretical reasons, we selected two regions, each with two departments. Practical and financial constraints limited the number of interviews to 25. We cannot rule out the possibility that our results may have been different had we sampled more regions, departments, and respondents, including those with other backgrounds (e.g., members of the Board of Directors or governmental policymakers). Nevertheless, the fact that our findings are largely in agreement with previously described principles of health care innovation suggests that these differences would have been minor. As with most qualitative research, however, caution is required when generalizing the findings to other countries, sectors, and types of innovations and innovation processes.

Conclusion and recommendations

This study has shown that PGME innovation is characterized by a highly dynamic and non-linear complex implementation process, which is influenced by many factors. We showed which factors were important in this context of implementation and why these were important. Although the factors influencing the innovation of PGME are largely similar to those influencing other innovations in health care and undergraduate and PGME (Bland et al. 2000; Wartman et al. 2001; Greenhalgh et al. 2004; Lillevang et al. 2009), we identified four specific features of PGME innovation: the challenge of implementing the CanMEDS roles; the importance of regional implementation strategies and educational support; the balance between training and patient care; and the need for regional inter-organizational networks of hospitals. Based on our experience, a number of recommendations can be generated for health care professionals and policymakers to tailor their interventions in the context of implementing PGME curricula (Table 4).

Our results ask for a re-assessment of the implementation of the CanMEDS roles. Additional efforts are required in developing blueprints on how to apply the CanMEDS roles in work-based training in order to achieve the high ambition of training medical professionals who are competent in all roles. Users need to perceive the short- and long-term advantages of the innovation and a real sense of urgency in implementing it. One way to achieve this would be to align the benefits of the renewed curricula directly with the outcomes of health care processes, such as patient safety and team collaboration and communication. The innovation as it was implemented was too much designed from an educational and curriculum development viewpoint. The innovation needs to be re-designed with the needs of the primary users (program directors, supervisors, and residents) in mind to make it more attractive for them to adopt the innovation. In addition, we propose coordination of the implementation process at the national, regional, and local levels. This coordination should involve a careful balance between approaches that “let it emerge” and those that “make it happen,” dependent upon the phase of implementation. At a minimum, coordination at each level should set goals, establish a timeframe, monitor the results, and facilitate knowledge sharing through meetings and social networks. Our results suggest that such coordination requires the possibility of using formal power to enforce the process (e.g., through the accreditation of training facilities). Support for program directors is probably best organized at the regional and local levels. Identifying and rewarding good leaders and good teams, using residents effectively as change agents, balancing training and patient care appropriately, providing proper local educational and logistic organizational support, and organizing strong regional collaboration networks between university and general hospitals can all accelerate the implementation process.

Acknowledgments

The authors thank the program directors, medical specialists, and residents of O&G and Pediatrics for their cooperation in this research. This study was funded by the National Board of Health Care Professions and Educations in the Netherlands.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.