Evaluation of a national process of reforming curricula in postgraduate medical education

Abstract

Context: A national reform of the postgraduate medical education in Denmark introduced (1) Outcome-based education, (2) The CanMEDS framework of competence related to seven roles of the doctor, and (3) In-training assessment.

Objectives: The purpose of the study was to evaluate the process of developing new curricula for 38 specialist training programmes. The research question was: which conditions promote and which conditions impede the process?

Methods: Evaluation of the process was conducted among 76 contact-persons, who were chairing the curriculum development process within the specialties. Quantitative and qualitative data from a questionnaire survey and telephone interviews were triangulated for data analysis.

Results: The response rate of the questionnaire survey was 83% (63/76). Twenty-six telephone interviews were conducted. Identified promoting factors included positive attitude and motivation in faculty and support from written guidelines and seminars. Identified impeding factors included insufficient pedagogical support, poor introduction to the task, changing and inconsistent information from authorities, replacement of advisors, and stressful deadlines.

Conclusions: This study identified promoting and impeding factors in a national postgraduate curriculum development process. Surprisingly the study indicates that pedagogical support provided throughout a process in some aspects might not be useful. General suggestions regarding curriculum reform processes are formulated.

Introduction

Major national reforms of postgraduate medical education have taken place in many countries during recent years. These have been caused by various reasons such as societal needs, politics, lack of specialists and wishes for better and shorter education (Frank et al. 1996; The Royal College of Physicians and Surgeons of Canada 1996; Leach 2001, 2004; ten Cate 2007). In essence these reforms include introduction of outcome-based education, a broader definition of competence and requirements of teaching and assessment strategies.

Comprehensive evaluation of reforms of specialist training usually focus on implementation, operation, effects and interim outcomes of the reforms (The Open University Centre for Education in Medicine 2001). However concerning evaluation of the process of developing postgraduate curricula little research has been reported (Posner 1995). Although many countries have recently reformed their postgraduate education and implemented different new educational paradigms like the CanMEDS framework (Frank et al. 1996; Neufeld et al. 1998; Maudsley et al. 2000), the reform process itself has not been described in literature. In order to be able to plan and support curriculum development processes in postgraduate medical education, knowledge about promoting and impeding conditions is needed.

The purpose of this study was to evaluate the process of developing new curricula according to a national reform of postgraduate medical education. The research question was: which conditions promote and which conditions impede the process?

Table

Practice points

When using a top-down strategy of implementing a reform, it is important that authorities are clear in messages and communication throughout the process. Traditions and values of both pedagogical experts and doctors regarding educational principles must be considered when launching new educational concepts. There is a delicate balance between pushing deadlines in a reform and allowing time to work with and adapt to new educational principles. Initiating part of a reform as pilot studies might be of help in clarifying elements of new concepts and how they work in practice.

Methods

Context of the study

A shortage of consultants and reports indicating insufficient quality of postgraduate education in Denmark induced a national reform of the postgraduate educational system. Representatives from key stakeholders participated in a specialist commission that published a report containing various recommendations regarding postgraduate education in Denmark (Ministry of Health 2000). The reform included some of the predominant trends in medical education during the last decade: (1) Outcome-based education (Harden et al. 1999; Harden 2002), (2) The CanMEDS framework describing aspects of competence related to seven roles of the doctor (Frank et al. 1996; Neufeld et al. 1998; Maudsley et al. 2000; Royal College of Physicians and Surgeons of Canada 2000), and (3) In-training assessment (Ringsted 2004).

The process

The Danish National Board of Health (NBH) issued ‘Guidelines for writing curricula’ (The National Board of Health 2001). According to these all of the 38 medical specialities should revise their curriculum indicating learning outcome, teaching strategies, and in-training assessment strategies related to each of the seven CanMEDS roles. Each specialty should appoint two contact-persons to be responsible for the task of developing a new curriculum for the specialty. Staff from NBH including doctors and educationalists was appointed to support the process. Each specialty was assigned an advisor from NBH responsible for supporting the work in the specialty. These advisors were all medical doctors. For political reasons the process was subject to rather tight deadlines.

Evaluation of the process

Quantitative and qualitative methods were included in the evaluation of the process by use of a questionnaire survey and an elaborating telephone interview. The contact-persons were asked their opinion about elements of the reform and specifically about the task of developing the new curriculum in their specialty.

Questionnaire survey

The questionnaire survey was conducted among the specialties’ contact-persons 6 months after the start of the process. The questionnaire was sent by mail to all of the 76 contact-persons. Persons not answering within 1 month received a new questionnaire. In order to detect possible changes during the process, the survey was repeated 1 year later among those 53 persons of the 76 contact-persons who had answered the first questionnaire and who were still listed as contact-persons.

The questionnaire was developed in several steps. First a list of topics that were expected to influence the process was developed combining input from literature and relevant stakeholders (Posner & Rudnitsky 1997; Leach 2001; The Open University Centre for Education in Medicine 2001). These topics were discussed with three persons from the population of contact-persons representing different types of specialties, and the list of topics was reviewed accordingly. The topics covered three themes:

1. The setting and the support in the process: Guidelines from NBH, advisors from NBH, seminars arranged to support the process, pedagogical support from NBH, time available and organisation of the process.

2. Personal issues concerning the contact-persons: motivation for the task, educational experience, position and attitudes regarding education and the task of developing curricula.

3. Activities taking place within the specialties: establishment of committees for developing curricula, debate in the specialties and number of persons involved in each specialty.

The questionnaire consisted of 28 questions to be answered on a 5-point Likert scale (disagree, partly disagree, neutral, partly agree, and agree). In addition demographic questions about the contact-persons and the process in their specific specialty were included.

The questionnaire was pilot-tested for validity and feasibility by having three persons from the population of contact-persons representing different types of specialties answering the questions while thinking aloud. They were asked specifically about how they understood each question and how they would answer it. Comments were used to adjust three of the 28 questions.

Statistics

Questionnaire response data were analysed using SPSS 13.0 software. Paired-Samples t-test was used to compare results from the first to the second round. The effect size (ES) for the differences was calculated using Cohens's d, with ES 0.2, small; 0.5 medium; 0.8 large (Hojat & Xu 2004).

Telephone interviews

Semi-structured telephone interviews were performed 1½ years after the start of the process. The contact-persons were called in random order until saturation in the responses was reached (Kvale 1996).

The interview guide included five open-ended questions probing for in-depth considerations of various issues regarding the process. The interview guide was validated by pilot-interviews with three of the contact-persons. The three persons were each asked specifically to think aloud both about how they understood each question and how they would answer it. Comments were used to make a few changes in the wording of a couple of questions. The questions are listed in Table 1. The interviews were alternately conducted by one of two researchers (GL and LB). One of the researchers (GL) was employed by the NBH during the first 6 months of the study and that might introduce bias in the responses. Hence data achieved by each of the two interviewers was subsequently compared regarding positive/negative and critical/non-critical responses.

Table 1.  Interview guide for the telephone interviews

Interview guide for telephone interviews
Open-ended questions
The NBH^1 has defined requirements regarding content and form of the curricula and communicated these requirements via guidelines and different amendments to them.
How did these requirements influence the curriculum development process in your specialty?
The NBH^1 has set the frame for the specialties’ work with the curriculum development, i.e. regarding economy, time and deadlines, information, advisors at NBH^1 and other support, seminars, etc.
How did this frame influence the development process in your specialty?
Please describe the curriculum development process in your specialty?
Did educational culture or attitudes to specialist education change in your specialty during the process of curriculum development?
If yes, which change did you experience?
Please describe the most important positive and negative experiences you had in relation to the process.

¹NBH = The National Board of Health.

The content analysis of interviews was performed using a method involving an editing organising style in the interpretation (Crabtree & Miller 1999). Two independent researchers (GL and HB) listened to each interview identifying essential messages. From the essential messages categories were identified in a process where resembling essential messages were put together. For each of the identified categories typical indicator quotations were chosen.

The two researchers reached a consensus about a final list of 43 categories. Subsequently the researchers listened to the interviews again, the interviews were coded by each of the researchers and consensus on the categorisation was reached.

Data analysis

The data collected via questionnaires and telephone interviews were triangulated and analysed in an inductive analytical process, which means that the data were used to inform themes of significance to the research question.

Results

For the questionnaire survey a response rate of 83% (63/76) was reached in the first round, and 98% (52/53) in the second round. There were no significant differences between the answers from the first and second round except for four of the questions. For these questions the effect size was only small to medium and hence of no practical importance. The results are listed in Table 2.

Table 2.  Opinions (mean, SD) about the elements of the reform and the process of developing new curricula expressed in the two rounds of the questionnaire survey

	Round 1 (N = 63)	Round 2 (N = 52)	Paired samples t-test	Effect size
	Mean (SD)	Mean (SD)	P	ES
Opinions about the concepts introduced by the reform
My specialty will benefit from the increased focus on educational aspects induced by ‘Guidelines for writing curricula’	3.8 (0.9)	3.9 (1.0)	0.272
It is an advantage that the new curricula will include minimum required competences	3.7 (1.3)	3.9 (1.2)	0.233
It is an improvement to the education that all learning goals have to be assessed	4.0 (1.1)	4.3 (0.9)	0.140
It is an improvement to the education that assessment will have to be performed for all learning goals regarding the seven roles of the doctor	3.5 (1.3)	4.0 (1.1)	0.006	−0.44
It is an improvement to the education that assessment in the future will have consequences	4.4 (0.9)	4.7 (0.6)	0.047	−0.30
I would myself have difficulties to end a young doctor's career because of lacking fulfilment of learning goals	2.8 (1.4)	2.6 (1.2)	0.366
Considerations of the learning environment are relevant for taking into account in the development of the new curriculum	4.0 (1.0)	4.2 (1.0)	0.245
Opinions about the task of developing curricula
The specialties are the appropriate ones to take care of developing the curricula	4.8 (0.5)	4.9 (0.3)	0.110
One of the biggest challenges in the curriculum development process is to formulate strategies for learning and assessment	3.8 (1.3)	3.8 (1.2)	0.960
One of the biggest challenges in the curriculum development process is to define an appropriate number of learning goals for the specialty	3.8 (1.1)	3.4 (1.2)	0.018	0.33
One of the biggest challenges in the curriculum development process is to define an appropriate level of detail for each learning goal	4.1 (0.8)	3.9 (0.8)	0.093
One of the biggest challenges in the curriculum development process is to formulate learning goals within all seven roles of the doctor	3.4 (1.1)	3.2 (1.0)	0.057
One of the biggest challenges in curriculum development process is to stick to the scheduled deadlines	3.8 (1.2)	3.3 (1.3)	0.014	0.38
I found that lacking knowledge about the content of ‘The Specialist Commission Recommendation Report’ in my specialty is a problem in relation to the new curriculum	3.3 (1.3)	3.3 (1.3)	0.835
The present educational culture in my specialty will impede the implementation of the educational approach in ‘Guidelines for writing curricula’	3.2 (1.3)	2.9 (1.1)	0.054
It is/will be a big challenge to start the necessary dialogue and change of educational practice in the specialty in relation to implementation of the new curriculum	4.5 (0.9)	4.5 (0.9)	0.589
The new curricula will contribute to showing that education requires resources	4.5 (0.7)	4.4 (0.9)	0.560
Opinions about the support in the development process
The curriculum development process is organised by the NBH^1 in a way that supports the work in the medical specialties	2.4 (1.1)	2.3 (1.1)	0.604
My specialty has had or will have benefit of our advisor at NBH^1.	3.1 (1.1)	3.2 (1.3)	0.781
The educational ambitions in ‘Guidelines for writing new curricula’ are appropriate to my specialty	3.3 (1.1)	3.3 (1.0)	0.740
‘Guidelines for writing curricula’ is written in a way that inspires me	2.9 (1.1)	2.7 (1.1)	0.405
The appendix in the Guidelines regarding learning goals is considerably easier to use than the appendices regarding teaching and assessment	3.2 (1.0)	3.5 (0.8)	0.095
The educational tools in the appendix regarding teaching make me think differently concerning learning methods	3.1 (1.0)	3.2 (1.1)	0.659
The theory regarding teaching in the Guidelines is difficult to understand	3.6 (1.0)	3.7 (1.0)	1.000
The theory regarding teaching in the Guidelines is difficult to use	3.8 (1.0)	3.8 (1.0)	0.863
The theory regarding assessment in the Guideline's appendix regarding assessment is easy to understand	3.1 (1.1)	2.8 (1.0)	0.412
The theory regarding assessment in the Guideline's appendix regarding assessment is easy to use	2.7 (1.0)	2.7 (0.9)	0.700
The educational tools in the Guideline's appendix regarding assessment make me think differently concerning assessment methods	3.1 (1.1)	3.3 (1.0)	0.274

Notes: Respondents indicated answers on a 5-point Likert scale (1 = disagree, 2 = partly disagree, 3 = neutral, 4 = partly agree, 5 = agree).

Each telephone interview lasted from 5 to 16 min and saturation in the responses was reached after 26 interviews, representing 1/3 of the population. There were no significant differences in the frequency of positive/negative and critical/non-critical answers achieved by the two interviewers. The predominant comments from the interviews are listed in Table 3.

Table 3.  Results from telephone interviews of contact-persons working with curriculum development (N = 26). Examples of quotes are typed in italics

	Positive comments (n = number of persons)	Negative comments (n = number of persons)
Overall opinion of conditions for doing the work	Were good (13)	Were bad (2)
	‘The conditions have worked quite ok under the given circumstances’	‘The conditions were poor’
Written material including guidelines	Useful, supportive (18)	Difficult, strange terminology, not inspiring (4)
	‘The Guidelines worked a rule of conduct. They clarified the task’	‘It was very difficult for us to understand. It was a completely new and strange terminology’
Information from NBH		Unclear, changing, poor introduction
		(12)
		‘We had some frustrations because of changing directives from NBH. Perhaps not surprising since all this was new for NBH as well’
Advisor at NBH	Helpful	No benefit, inexpedient change of advisor
	(16)	(7)
	‘They were kind and helpful, although they were not always capable of giving useful advise’	‘Along the road we became more experienced than them’
Pedagogical support		Not qualified support
		(3)
		‘Some of the presentations at the seminar were useless theoretical anthropology’
Seminars	Helpful, inspiring	Limited benefit
	(13)	(5)
	‘It provided an opportunity to network and exchange ideas and problems’	‘We did not get any major benefit, it rather served to keep us on track’
Financial support	OK	Insufficient
	(7)	(5)
	‘No problems’	‘The financial support did not at all correspond to the amount of work we made’
Time frame	Deadlines OK	Lack of time, stressing
	(8)	(10)
	‘The tight deadlines were ok. If we had had the double time we would have taken the double time’	‘Tight deadlines made it difficult to do it as good as you wanted to’
Workload		Hard work, few persons
		(9)
		‘The fact that we were only a few persons made us feel overworked’
Dialogue in specialty	Constructive dialogue and back up	Poor back up and dialogue
	(16)	(3)
	‘The specialty has been engaged in the process. People were really supportive and enthusiastic’	‘It has been difficult to engage the specialty society’
Cultural changes in specialty as a result of the reform	Already happened (13)	No changes (yet) (14)
	‘People have realised that a change of attitudes is needed. The process has initiated a lot of reflections about learning and assessment’	‘Not yet. Maybe for a few of us. It will not come until the implementation’
Development in specialty	The process contributed to develop and define the specialty
	(12)
	‘We have had a positive discussion about clarifying the future identity of our specialty’
Impact on quality of education	Positive impact
	(11)
	‘The new curriculum will improve the quality of education’
Change in personal attitude towards education	Positive change of attitude to and thoughts about education
	(10)
	‘I changed my perception of my specialty and what we can expect from a trainee’
Personal experience	Benefit
	(13)
	'Interesting, exciting, funny. I have gained an enormous insight in education’

Most of the specialties established a committee for curriculum development (33/38). A wide variety of activities were planned and conducted within the specialties in order to broaden the debate about the new curricula. The contact-persons taking part in the process of developing new curricula were highly motivated people having some educational experience. In general the contact-persons experienced constructive dialogue and support in their specialties introducing the curriculum.

The contact-persons had a positive attitude towards the concepts introduced by the reform and trusted that the new curricula would improve the quality of the education. The positive attitude applied to both outcome-based education, the description of competence related to seven roles of the doctor and to in-training assessment. In general the results from the telephone interviews performed late in the process seem to be rather more positive than the survey results.

Challenges

Although the contact-persons were motivated to undertake the task of developing the curricula it was clearly a challenge. Contact-persons indicated problems in defining an appropriate number of learning goals and specifying an appropriate level of detail for each learning goal. Formulating strategies for learning and in-training assessment also was found difficult and challenging. Formulating learning goals according to the seven roles was not per se a problem.

Promoting factors

When summarizing the results form the surveys and the interviews, promoting factors in the development process included the written guidelines outlining the requirements for the curricula. The guidelines were of some use in structuring and supporting the work in the specialties, despite the fact that the text and especially the pedagogical terminology was difficult to understand and to relate to clinical work. Additionally the guidelines did not provide much motivation and inspiration to do the work. Three seminars conducted by NBH during the first year to support the process were beneficial for most of the participants from the different specialities in giving possibility for face-to-face discussions and for providing concrete formal and informal help and inspiration for the work.

The results were conflicting regarding the support from the advisors at NBH. According to the questionnaire survey, the benefit from the advisors were rated as neither promoting nor impeding, but in the telephone interviews most of the interviewed contact-persons found the advisors helpful in providing supervision and feedback and answering concrete questions. It was clear from the interviews that some replacements among the advisors were perceived as problematic.

Impeding factors

When summarizing the results from the surveys and the interviews, impeding factors included insufficient introduction in the specialties to the curriculum development task, difficulties getting started with the work in the specialties, changing and inconsistent information and requirements from NBH, and the fact that not all written material, such as guidelines on assessment methods, was launched from the start of the process. The contact-persons in the specialties found the tight deadlines frustrating and counter-productive for the work.

The pedagogical support from NBH provided to the contact-persons throughout the process was in general rated to be insufficient and in some aspects actually had an impeding impact on the development process. The pedagogical support was insufficient to overcome the difficulties in defining appropriate learning goals and in specifying strategies for learning and assessment. The identified problems concerning the pedagogical support included difficulties in understanding and relating the pedagogical thoughts to the context of postgraduate work-based education.

Discussion

The main results of the study show that the contact-persons were positive towards the concepts introduced by the reform, that they found the task of developing new curricula according to these concepts to be quite difficult and that they did not get the necessary support in the process, especially regarding pedagogical problems.

An important promoting condition for the development process was the positive attitude and motivation among the contact-persons. These factors are well known from literature (Gale & Grant 1997; Genn 2001). The other promoting factors identified in the study – the written guidelines and the seminars – provided structure and direction to the development process and indicated the standard for the new curricula and hence could be expected to be supportive.

Surprisingly the results demonstrated that the pedagogical support provided throughout the process was perceived of limited benefit and in some aspects actually had an impeding impact on the process. It is possible that the pedagogical assistants themselves had problems with the new paradigm. A collision between their sociologic-pedagogical traditions (Illeris 2004) and the structured, rational approach to education in the paradigm of outcome-based education and in-training assessment according to the seven roles of the doctor (Ringsted 2004) is likely. In Denmark there is no tradition for assessment of clinical performance, traditionally only theoretical exams in undergraduate education have been used. Hence applying in-training assessment in postgraduate work-based education clearly must be a challenge to the physicians and the educators involved. The new educational trends were more or less directly imported from Anglo-American countries with long psychometric traditions, and at that time the literature provided little information about how to handle such paradigm shifts in a profitable way. Only recently discussions about the transatlantic differences emerged (Hodges & Segouin 2008). All in all, the introduction of the new educational concepts was a challenge for the advisors as well and they could probably not foresee all possible problems and give all the right answers from the beginning of the process. One way to overcome this could be to apply a more open dialogue about the nature of the process and the challenges of the reform and to support commitment to take part of the reform process under the given changing circumstances (Gale & Grant 1997; Leach 2001; Wartman et al. 2001).

The difficulties of defining learning goals identified in the study are similar to challenges using outcome-based education in other countries (Talbot 2004; Huddle & Heudebert 2007; ten Cate & Scheele 2007). Huddle and Heudebert argue that objective assessment based on learning objectives may capture only knowledge and skills that amount to the ‘building blocks’ of competence without elucidating higher-level clinical competence (ten Cate & Scheele 2007). In postgraduate medical education most learning goals represent higher-level clinical competences that might be hard to define.

Some of the impeding factors identified in the study resemble the expectations according to the literature i.e. experience of chaos and confusion at the beginning (Walker 1971) and stress caused by narrow time limits (Thacker 2000). The process investigated was based on a national top-down implementation strategy. The results underline the importance of the authority being clear in messages and communication and of the need to motivate and to help in understanding throughout the process. The decision to include outcome-based education and in-training assessment according to the seven roles of the doctor was made by NBH. However it has subsequently been validated that Danish doctors actually agree with the importance of the seven roles (Ringsted et al. 2006).

Methodological aspects

The high response rate of the first questionnaire survey, 83% of the whole population, was considered appropriate in order to describe the view of the 76 contact-persons. For the telephone interviews, the external validity should be quite good having 1/3 of a homogenous population participating (Kvale 1996). The methods used in the study were time consuming and included thorough validation. As intended, the results provided new information about the process of curriculum development.

The interview data seem rather more positive than the survey data. The retrospective design of the interviews might have caused a general tendency to minimize the actual problems and frustrations. But the difference might also be caused by the different way of gathering data or from the different phrasing of questions. Telephone interviews often tend to reduce resistance to sensitive items (Oppenheim 1993). One of the researchers (GL) was employed at NBH during the first half of the period in which the new curricula were developed, and that could have influenced the data in the opposite direction. However, no major differences were found when comparing the data collected by the two interviewers, and the possible bias hence seems to be of less importance.

The open-ended approach in the telephone interviews complicated the categorisation of data since the interviewed persons often spoke about several things at a time and sometimes dualistically about both positive and negative aspects at the same time. The consensus reached between two independent researchers minimised these interpretation problems.

The results of this study is quite closely linked to a national reform in Denmark during recent years. The study was restricted to the point-of-views of those developing the new curricula. Other stakeholders might add further elements to the research question. However, the results provide a picture of essential promoting and impeding conditions that constitutes the background for general recommendations regarding the process of reforming medical education using a top-down implementation strategy. Although it is impossible to foresee and take care of all kinds of problems in the planning of a process, it is possible that proceeding pilot studies or action research involving key stakeholders (Rapoport 1970) during the process might alleviate the implementation of new concepts and methods introduced by reforms.

Conclusion

This study identifies some promoting and impeding factors in a national postgraduate curriculum development process. Most of the results are in line with what could be expected from the literature, but the study indicates that pedagogical support provided throughout a process is not always useful. The results demonstrate the importance of involving and motivating faculty in reform processes. The results provide a background for general recommendations regarding the process of reforming medical education when using a top-down implementation strategy.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Gunver Lillevang

GUNVER LILLEVANG, MD, MHPE, Center for Clinical Education, Copenhagen University Hospital, Copenhagen, Denmark.

Lasse Bugge

LASSE BUGGE, MD, Surgical Department, Vejle Hospital, Vejle, Denmark.

Henning Beck

HENNING BECK, MD, The National Board of Health, Copenhagen, Denmark.

Jan Joost-Rethans

JAN-JOOST RETHANS, MD, Skillslab, Maastricht University, Maastricht, The Netherlands.

Charlotte Ringsted

CHARLOTTE RINGSTED, Professor, Center for Clinical Education, Copenhagen University Hospital, Copenhagen, Denmark.