Evaluating the effectiveness of curriculum change. Is there a difference between graduating student outcomes from two different curricula?

Abstract

Background: Changing a curriculum raises the question whether the results, new curriculum student outcomes, are different from old curriculum student outcomes.

Aims: To see whether different curricula produce different outcomes, we compared test and questionnaire results of two cohorts. We wanted to know if there is a difference on knowledge and skills test results, and on the number of times students practiced their skills during their final internships.

Method: We used two validated test instruments: the Dutch Progress Test (PT) and the Objective Structured Clinical Examination (OSCE). For reporting their skills practice, we asked students of both cohorts to fill out a basic skills questionnaire.

Results: We found no significant difference between both cohorts on the results of their knowledge test and their report on skills practice. On the OSCE, students from the new curriculum cohort scored significantly higher than old curriculum students.

Conclusion: Curriculum change can lead to improvements in graduating students’ outcome results.

Background

During the last decades, many efforts were made towards evaluating the effectiveness of curriculum change. However, the question whether curriculum changes result in better medical doctors is a complex one and cannot be answered by a single outcome study. Answering more complicated questions often requires a programme of research rather than a single study (Schuwirth & Cantillon 2005).

There are three basic levels in any social practice which can be the focus of some kind of evaluation: the people involved, the processes engaged in, or the products emerging from these processes (Hopmann 2003).

When focussing on the products emerging from a curriculum change, researchers tend to engage in outcome research. Graduating students from one curriculum are tested, students from a different curriculum undergo the same tests, and the results are compared (e.g. Pearson et al. 2002; Jünger et al. 2005; Peeraer et al. 2007). Given that a principal objective of medical education is to educate trainees, it is certainly appropriate for medical education researchers to be keenly interested in trainee performance (Prystowsky & Bordage 2001).

Table

Practice points
Comparative outcome studies are needed to build an evidence base on which decisions of change can be made. Curriculum change can lead to improvements in graduating students’ outcome results. Further research is necessary on what constitutes a good doctor.

Aims

We executed a cohort study, comparing students from both old and new curriculum as part of a research project on curriculum change. We investigated knowledge and basic clinical skill outcomes between graduating old and new curriculum students. For the present study, we investigated both cohorts of students in their final year, and we compared different test and questionnaire results in a triangulation. We wanted to find out whether graduating students from two curricula (old and new) show different scores on knowledge testing, on clinical skills testing and on performing skills activity during fulltime internships.

Method

Educational background

In 1998, the medical school of the University of Antwerp decided to drastically change its curriculum, following the advice of the Flemish Interuniversitary Board (VLIR, Vlaamse Interuniversitaire Raad) who did a quality control study on all Flemish medical schools in 1997. International experts (mainly from the Netherlands) were invited to help design a new curriculum, the first one to be applied in Flanders. The old traditional curriculum was abandoned and replaced by a curriculum based on Harden's SPICES model (Harden et al. 1984):

Table

Student centred	N					O	Teacher centred
Problem based	N			O			Information oriented
Integrated	N						Discipline based
Community oriented			N		O		Hospital based
Elective driven			N	O			Uniform
Systematic		N				O	Opportunistic

(N: New curriculum, O: Old curriculum)

The new curriculum reduced the basic sciences courses to the first 1.5 years. Students start with integrated system-based modules in year 2 and finish them at the end of year 5. A module starts with a few hours on anatomy, physiology and histology, focussed on the module's theme, and then moves on until all necessary pathology is learned. After that, a new module starts. Large group lectures (with problem-based elements) still form the largest part of a module but there are also interdisciplinary lectures, case-based lectures and from time to time, students work in small groups on a specific assignment. During each module, students receive bedside teaching in the hospital, which increases the clinical practice students receive.

Because research had shown that old curriculum students had a lack of clinical skills competence (Remmen et al. 1999), clinical skills training changed completely. Clinical and communication skills training now start in year 1, where old curriculum students received limited formal training in years 4 and 5, but had to learn their skills mainly through internships. Students now have a nursing clerkship at the end of year 1, after having been taught the necessary skills, and from year 2 until year 5 they learn all other skills that have to be mastered when graduating. All training sessions are integrated in the study modules: while students learn about gynaecology, they also perform gynaecological skills training, and so on. The skills are taught in small groups (2–15 students), with a teacher who first demonstrates a skill and then allows students to practice the demonstrated skill on each other, on manikins and on simulated patients. The teacher observes the students, corrects them if necessary and answers their questions. Students document their experiences in a portfolio, which is rated each year by a team of skilled teachers. Students are assessed periodically by means of an Objective Structural Clinical Examination (OSCE), with a chance to redo the test and practice during free hours on mastering the skills if they need to. They are also offered extra opportunities to train during free hours with a teacher standby when they need help. A team of new young teachers was attracted to support all clinical teaching.

For both old and new curriculum, the final last months (12 months for old curriculum, 18 months for new curriculum) consists of fulltime internships.

There was no difference in learning goals or competences on both old and new curriculum regarding clinical skills and medical knowledge, although both were taught in a completely new way.

Subjects

A representative sample of students was compiled for both old and new curriculum. The last students of the old curriculum graduated in June 2004 and were, therefore, tested in 2003 and 2004. The first new curriculum students graduated in 2005 and were tested in 2004 and 2005 For both cohorts, we asked students to volunteer in different tests. All volunteers were paid about 25 euro per test. Out of volunteering students (39 from the old curriculum, 43 from the new curriculum), we composed representative samples based on gender and grades for both the old and new curriculum cohort. Both samples consisted of 25 students (Table 1).

Table 1.  Subjects

	Old curriculum		New curriculum
	Total year group	Sample	Total year group	Sample
	104	25	71	25
N
Females	62%	68%	63%	68%
Male	38%	32%	37%	32%
Pass	65%	64%	25%	20%
Pass distinction	25%	24%	44%	52%
Pass great distinction	8%	8%	27%	24%
Pass greatest distinction	2%	4%	4%	4%
Age	25	25	25	25

Instruments

Assessment of clinical skills: OSCE

As assessment tool, a 15 stations OSCE was used, in which each station took 10 minutes and in 9 stations, simulated patients were used. We used an OSCE that represents the basic disciplines of the 7-year (both new and former) curriculum (gynaecology, locomotor system, neurology, etc.). This OSCE is a barrier examination in the new curriculum, which means that students have to pass it in order to progress to the clerkship year. They have to show that they master the standards on basic clinical skills. These standards are the same standards that were used in the former curriculum. There are no other assessment topics in our OSCE apart from showing the mastering of standards on basic clinical skills.

Students’ performance was rated by trained and experienced observers, who were not explicitly informed that test results would be used for this study. The observers used three global scores, for completeness, systematic approach and proficiency. These global scores were given on a scale from 1–10 (≤5 = fail; 10 = excellent) and were described in the ‘instructions for observers’ issued before the OSCE. The three global scores were averaged to obtain one definite station score. The overall score was the mean score across 15 stations. Mean group scores were calculated for the students from the traditional and the renewed curriculum. Group scores per station were calculated.

New curriculum students are familiar with OSCE testing, while the students of the old curriculum never took this type of testing. We tried to reduce this limitation by offering old curriculum students a 1-hour workshop on OSCE testing previous to the test. All old curriculum students but one attended the workshop.

Old curriculum students took the test in February 2004, new curriculum students in October 2004. Both cohorts took the test right after finishing their fulltime internships.

Assessment of knowledge: Progress test

As assessment tool, the March 2004 version of the Dutch Progress Test (PT) was used. This valid and reliable test is best characterized as a comprehensive final examination in medicine, reflecting the cognitive final objectives of the curriculum. It samples knowledge across all disciplines and content areas relevant for the medical degree. As a result the test has no direct link with any specific course or unit. Questions for the PT are written by faculty members from all departments of the medical school (Verhoeven 2003). The test consisted of 250 items closed format, the (multiple) true/false/do not know format; 11 items were removed because they were specifically designed for the Dutch context (e.g. legal procedures). The two cohorts took the test during their final months before graduation (old curriculum students in May 2004, new curriculum students in May 2005).

Clinical skills: Experience questionnaire

Next to taking the OSCE, students also filled in a questionnaire expressing their self-reported experience with training/performing skills during fulltime internships. We gave students a list of the 349 basic clinical skills students had to be able to perform at the end of their 7-year training. Students filled out the questionnaire just after their 12 months of fulltime internships. They indicated how many times they performed each skill (not seen and not done, seen but not done, performed only once, performed ∼3 times, performed 10 times or more). Old curriculum students reported in December 2003, new curriculum students in September 2004.

Data analysis

For each student, we established one total score per assessment (OSCE and PT assessment) and one total score for the experience questionnaire; this resulted in three variables per student.

For the OSCE, the total score was defined as the mean score of the scores on completeness, proficiency and systematic approach across the 15 OSCE stations. The total OSCE score was expressed as the percentage of the maximum attainable score.

For the PT, test results were calculated by subtracting the number of incorrect answers from the number of correct answers, corresponding to a score of +1 for a correct answer, −1 for an incorrect answer, and 0 if the ‘do not know’ option was chosen. The PT score of a student was expressed as the percentage correctly answered questions minus the percentage incorrectly answered questions.

For the experience questionnaire, we transferred ordinal scores to numerical scores, according to the number of times a student performed a skill:

Table

Level 1: not seen, not done	score 0
Level 2: seen, but not done	score 0
Level 3: performed by student one time	score 1
Level 4 performed by student approximately three times	score 3
Level 5: performed by student 10 times or more	score 10

We then added all scores. The maximum attainable score (10 × 349) was 3490, which would mean that a student performed each skill on the list 10 times or more. The total experience questionnaire score of a student was expressed as the percentage of the maximum attainable score.

The difference of the means of three variables were investigated: OSCE scores and PT scores were expressed by means of percentages. Questionnaire scores (Q) were expressed by the total questionnaire sum. The means of the two cohorts were compared using the t-test for independent samples in the statistical package SPSS (release 12).

To investigate whether a significant results also represents a substantial effect, we calculated the Effect Size (ES) to express the degree to which the findings have practical significance in the study population. The ES is an index that quantifies the degree to which the effects found in a study should be considered negligible or important, regardless of the size of the study sample (Hojat & Xu 2004).

For two independent samples the ES is defined as the difference between the means of the two groups, M₁ and M₂, divided by the pooled within group standard deviation.where n₁ and n₂ are the sample sizes, and σ₁ and σ₂ the standard deviations of the two groups.

Results

Table 2 shows that for the PT and the Q no significant differences were found between the two groups: old curriculum students score a mean of 38.7% on the PT while new curriculum students score a mean of 35.9%. Old curriculum students perform 47.7% of the maximum possibility to perform skills (according to the questionnaire) while new curriculum students score 55.2%.

Table 2.  Overall results from OSCE, questionnaire and progress test

	Old curriculum (n = 25 students)		New curriculum (n = 25 students)		Difference New versus Old
Variables	Mean	SD	Mean	SD	Mnew−Mold	p-value	ES
PT (%correct−incorrect)	38.7	8.7	35.9	8.7	2.7	0.27	–
OSCE (% of maximum)	65.2	7.1	73.3	5.3	8.1	<0.0005	1.3
Q Experience (%of maximum)	47.7	13.3	55.2	13.7	7.4	0.06	–

For the OSCE score a significant difference between the two groups was found. Old curriculum students score a mean of 65.2% while new curriculum students score a mean of 73.3%. With a p-value <0.0005, we computed an ES of 1.3, indicating a large effect according to Cohen's classification (Hojat & Xu 2004).

Conclusion

Whereas many perceive the changes in the curriculum as a deterioration of student knowledge, we have first of all proved that our new curriculum students score the same as old curriculum students on a knowledge test. Second, new curriculum students spend more time on skills training during the first 5 years of their study and they also report to practice their skills more during internships, although this difference was not statistically different compared to old curriculum students. Third, new curriculum students score significantly statistically higher, when measuring by means of an OSCE, indicating an improvement in basic skills performance. These results, however, are based on a cohort of only 25 students for each curriculum, on single test results, with a period of 1 year between the tests. These limitations are important as they prevent us from generalizing our findings to all curriculum change outcomes. For clinical skills competence, we conducted no pre-test for old curriculum students.

Assessment can be looked upon as a subset of evaluation, its results potentially being used as a source of information about the program (Goldie 2006). Given the state of flux of the current educational climate, it is important that more comparative outcome studies are conducted in order to build an evidence base from which important medical school and workforce decisions can be made (Pearson et al. 2002). Rigorous evaluation of curriculum reform and educational interventions will always be required if we are to be certain that we are doing the best we can for our students (Jones et al. 2001) and if we want to gain insight into exactly which elements of education work best (Schuwirth & Cantillon 2005). When looking at Miller's pyramid of competence (Miller 1990), we see that we have tested our students according to the levels 1 (the student's situated knowledge, tested with the PT) and 3 (the student shows how, tested with the OSCE). We did not test on Miller's highest level of competence (level 4), i.e. how a student translates knowledge and competence in clinical practice. This should be the focus of further research.

As for the skills questionnaire, we had to rely on the answers given by students in hindsight. To see whether this represents reality, further research using observation in the clinical setting might be useful.

In almost all contemporary medical curricula, skills lab offer extensive skills training for students. This is beneficial for the student but also for patient care: before students enter the hospital, and engage in real life patient contact, they have proved to be competent in mastering clinical and communication skills. This diminishes the risk for patients falling into the hands of unskilled trainees. Although students engage in many hours of skills training during their first years, they also manage to take in all the knowledge needed to become a knowledgeable graduate.

Independent research at the Faculty of Political Sciences of the University of Antwerp (Cornette 2008) has executed a cohort study with the same groups of students we used in our research, albeit with the entire year group. They questioned the students on feelings of certainty/uncertainty in four domains: medical knowledge, clinical skills, communication skills and EBM & science. According to this study, new curriculum students felt significantly more certain in the domains of clinical and communication skills, whereas there was no difference in the domains of medical knowledge and EBM & science. Most former and new curriculum students are convinced that the new curriculum offers a better training for clinical and communication skills.

It is difficult to answer the question whether we are producing better doctors. There is no solid definition of what constitutes a good doctor. Previous research found different characteristics of ‘a good doctor’ in order to define the ideal end-product of medical education. According to both patients and physicians, all characteristics adhering to the domain ‘knowledge’ were considered to be most important (Fones et al. 1998; Cullen et al. 2003). This means that educational change should not result in loss of knowledge or loss of clinical reasoning, as this is a primary characteristic of a good doctor. According to our study, the radical change in skills training results in graduates who are better trained in the field of basic clinical skills and who hold the same level of ready knowledge. This does not answer the question whether we produce better doctors, but our curriculum change did not lead to a loss of the most important characteristic of a good doctor.