Comparing the educational environment (as measured by DREEM) at two different stages of curriculum reform

Abstract

Background: The medical programme at Lund University, Sweden, has undergone curricular reform over several stages, which is still ongoing. Students have been somewhat negative in their evaluations of the education during this time.

Aim: To find out how the students perceived the educational climate using the Dundee Ready Education Environment Measure (DREEM), and to compare the findings taken at two given points in time.

Method: The DREEM instrument was distributed in semesters 2, 6 and 10 in 2003 and 2005, to a total of 503 students.

Results: The students rated their climate as positive. The total DREEM score (145) was somewhat higher than other published results and in the same range as for other reformed curricula. There was hardly any difference between the genders in their perceptions of the climate. Certain items were rated low and became subject of development between the measurements. These items concerned a perceived lack of a support system for stressed students and a lack of feedback and constructive criticism from teachers. Some improvement was detected in 2005.

Conclusion: The educational climate was high in a reformed curriculum and could be maintained high during on-going curricular reform. Educational development resulted in better results on some items.

Introduction

Educational environment has attracted increasing attention in recent years, especially in relation to the way in which it is measured (Genn & Harden 1986; Genn 2001a, b). Several factors influence the students’ experience, e.g. quality of feedback and social relations with classmates in addition to the more obvious aspects like timetabling, exams and teaching. The Dundee Ready Education Environment Measure (DREEM) instrument has been developed and validated for use regardless of culture and country (Roff et al. 1997). It has been used in many settings and for several purposes, e.g. to compare schools (Roff et al. 2001; Al-Hazimi et al. 2004), to compare sites within a school (Varma et al. 2005), to compare academic achievers and under-achievers (Mayya & Roff 2004), to get a baseline before curriculum reform (Jiffry et al. 2005), to find out how students perceive the ideal educational environment (Till 2005), to look at expectations of climate (Miles & Leinster 2007), and, finally, as an instrument for improvement (Whittle et al. 2007).

The medical school at Lund University, Sweden, introduced major curricular reforms in 1991, with minor revisions over the following years. The initial change was aimed at horizontal integration within three phases, some degree of vertical integration, studies based on problems or cases and early introduction of patient communication. Problem-based learning (PBL) was introduced during the first five semesters. The Harvard case method was adopted and introduced in some clinical courses (Stjernquist & Crang-Svalenius 2007).

Table

Practice points

The DREEM is useful for comparing educational climate across schools and programmes. The DREEM can be used to identify problem areas in the education programme. Students’ negative reactions to ongoing curricular reform do not necessarily result in negative changes in perceptions of climate. Educational development aimed at the problem areas can result in improvements that may be detected with the DREEM.

In local evaluations as well as in reviews by the Swedish National Agency for higher education, students have been critical of various aspects of their education, which have not entirely fulfilled their expectations. About 10 years ago, it became obvious that the curriculum was overloaded, especially during the final years of training. There were also some unwanted discrepancies between the two major teaching hospitals responsible for the teaching. Therefore, the faculty launched another major curriculum reform in 2005 focusing on the core of the curriculum, clinical rotations and a rather large (20%) part of the curriculum was reserved for student-selected components. Furthermore, the assessment system was revised. The reform process between 2003 and 2004 caused further complaints from students who were worried about the student-selected components, and from some faculty members who questioned the rationale behind the integration of clinical disciplines and the necessity of defining a core curriculum.

We decided to use the DREEM in order to compare the climate in Lund with other schools worldwide and to study the effects of the ongoing developments. We were also interested in identifying areas in need of further improvement.

Method

The programme and the students

The medical programme at Lund University is 5½ years with undergraduate entry, and it is followed by an internship comprising 18 months. Usually, 96 students are enrolled twice a year (the academic year has two semesters) and 50–60% of the students are female.

We decided to address students in three semesters. Semester 2, which is in the beginning of the programme containing core pre-clinical studies (mainly anatomy and neurobiology); semester 6, which contains the first clinical rotations, and finally semester 10 where the students are well into their clinical studies and facing graduation. We also decided to repeat the study 2 years later to see if educational developments resulted as more positive educational climate. This means that the study was partly longitudinal since we followed the students from semester 2 in 2003 into semester 6 in 2005, and the students from semester 6 in 2003 into semester 10 in 2005. The students in semester 2 in 2003 are thus essentially the same as the students in semester 6 in 2005, and similarly the students in semester 6 in 2003 are essentially the same as the students in semester 10 in 2005. The cohorts of semester 2 differed between the two investigating periods since the cohort of year 2005 started their medical programme with an 8-week long introductory course focussing on an overview of the programme, PBL training and study techniques. This new course was introduced in order to better prepare new students for the coming semesters.

The DREEM questionnaire

The DREEM inventory was translated into Swedish by a teacher proficient in English and then translated back into English by a professional translator. Differences were discussed between the translator and two of the authors. The instrument was piloted with a group of 20 students in the fourth year of study.

The DREEM contains 50 items, which the students rate on a five-point Likert scale (from 0 = strongly disagree to 4 = strongly agree). Some items are negative and the scores are reversed. The maximum score is 200. Five subscales are measured by the inventory: (1) perceptions of learning (max. score 48), (2) perceptions of teachers (max. score 44), (3) academic self-perceptions (max. score 32), (4) perceptions of atmosphere (max. score 48) and (5) social self-perceptions (max. score 28).

The instrument was administered to all students in semesters 2, 6 and 10, respectively, in 2003 and 2005 at the end of lectures. The cohort of semester 2 in 2003 was essentially the same as the cohort of semester 6 in 2005, and the cohort of semester 6 in 2003 was also essentially the same as the cohort of semester 10 in 2005.

Statistical analyses/data analysis

The chi-square test was used to analyse differences in nominal data, while the Mann–Whitney U test and Kruskal–Wallis one-way analysis of variance test were employed for ordinal and ratio data. Due to multiple comparisons (three groups), a reduced p-value of <0.017 was used to control for the risk of mass-significance (Bland & Altman 1995). In all other cases, p-values of ≤0.05 were considered statistically significant. Cronbach's alpha was employed to assess the internal consistency of the subscales of the instrument (Cronbach 1951). All analyses were performed using SPSS for Windows 14.0.

Results

The overall response rate was 82% (201/246) in 2003 and 75% (194/257) in 2005. Of the students that responded 58.9% were females and 41.1% males, figures which corresponds with the overall male–female ratio.

The total mean score of the inventory was 144/200 in 2003 and 146/200 in 2005. The total score from the six different semesters (141–152) are listed in Table 1. Scores in the range 101–150 are considered as ‘more positive than negative’ and scores above 150 as ‘excellent’ (McAleer & Roff 2001). There was a significant increase in the total score on semester 10 between 2003 and 2005 (Table 1).

Table 1.  Sum scores on the different subscales by semester (S) and year

	2003			2005
Subscale (maximum score)	S2	S6	S10	S2	S6	S10
Perceptions of learning (48)	35	33	32	35	33	35
Perception of teachers (44)	28*	30*	32	29	30*	33*
Academic self-perceptions (32)	21*	24	23	20	23*	24
Perceptions of atmosphere (48)	37	37	37	37	36	39
Social self-perceptions (28)	20	21	20	21	20	21
Total (200)	141	145	144*	142	142	152*

Note: *statistically significant difference between years 2003 and 2005, respectively.

On the five subscales, the following mean scores were obtained: (1) perceptions of learning 34/48 in both 2003 and 2005, (2) perceptions of teachers 30/44 in 2003 and 31/44 in 2005, (3) academic self-perceptions 23/32 in 2003 and 22/32 in 2005, (4) perceptions of atmosphere 37/48 in 2003 and 38/48 in 2005 and finally (5) social self-perceptions 20/28 in 2003 and 21/28 in 2005 (Table 1). All the results are in the upper part of the range indicating a positive perception of the educational environment. There was a significant increase in the scores on perceptions of teachers from cohorts of semester 2 through to semester 10. Cronbach's alpha ranged from 0.6 to 0.84 over all the subscales. The only significant subscale difference in relation to gender was with the perception of teachers where females gave a higher rating.

Scores for the individual items in all semesters are listed in Table 2. There were 11 items indicating the real problems in 2003 and only 4 in 2005 (shown in bold). Items that were consistently low over all semesters should also be taken seriously since they are less likely to be the result of a cohort effect. These were: item 4 (I am too tired to enjoy this course), 11 (the atmosphere is relaxed during ward teaching), 22 (the teaching is sufficiently concerned to develop my confidence), 25 (the teaching overemphasizes factual learning), 27 (I am able to memorize all I need), 31 (I have learned a lot about empathy in my profession), 36 (I am able to concentrate well) and 47 (long-term learning is emphasized over short term).

Table 2.  Mean scores for individual items on all semesters (S) in both years

		2003			2005
		S2	S6	S10	S2	S6	S10
1	I am encouraged to participate in class	3.0	3.1	3.1	3.0	3.2	3.4
2	The teachers are knowledgeable	3.3	3.5	3.2	3.5	3.3	3.5
3	There is a good support system for students who get stressed	1.7*	1.6	1.5	2.1*	1.3	1.7
4	I am too tired to enjoy this course	2.2	2.7	2.7	2.6	2.2	2.7
5	Learning strategies which worked for me before continue to work for me now	2.7	2.9	3.2	2.5	2.5	3.3
6	The teachers are patient with patients	2.4	2.7	3.0	2.5	2.8	3.1
7	The teaching is often stimulating	3.5	3.2	2.6	3.2	3.1	3.1
8	The teachers ridicule the students	3.4	3.1	3.5	3.4	3.2	3.3
9	The teachers are authoritarian	2.5	2.1	3.2	2.3	2.4	2.8
10	I am confident about my passing this year	2.4	3.0	3.0	2.2	2.6	3.0
11	The atmosphere is relaxed during the ward teaching	2.8	2.8	2.7	2.5	2.7	2.9
12	This school is well timetabled	2.6	2.8	2.7	2.7	2.6	3.1
13	The teaching is student centred	2.8	2.6	2.5	2.6	2.5	3.0
14	I am rarely bored on this course	2.7	3.1	1.7*	3.0	2.8	2.5*
15	I have good friends in this school	3.4	3.4	3.2	3.5	3.6	3.5
16	The teaching is sufficiently concerned to develop my competence	3.5	3.4	3.1	3.3	3.2	3.4
17	Cheating is a problem in this school	3.5	3.3	3.6	3.5	3.4	3.5
18	The teachers have good communication skills with patients	2.4	2.8	3.0	2.6	3.0	3.1
19	My social life is good	3.4	3.4	3.4	3.4	3.5	3.4
20	The teaching is well focused	2.8	3.0	2.7	2.9	3.0	3.0
21	I feel I am being well prepared for my profession	2.9	3.1	2.8	3.0	2.8	3.2
22	The teaching is sufficiently concerned to develop my confidence	2.6	2.7	2.6	2.7	2.6	2.9
23	The atmosphere is relaxed during lectures	3.3	3.5	3.2	3.4	3.3	3.3
24	The teaching time is put to good use	2.7	2.2	1.8*	2.6	2.3	2.2*
25	The teaching over-emphasizes factual learning	2.3	2.5	2.7	2.5	2.4	2.8
26	Last year's work has been a good preparation for this year's work	2.1	3.3	2.8	2.6	3.3	2.7
27	I am able to memorize all I need	2.1	2.3	2.2	2.2	2.1	2.1
28	I seldom feel lonely	2.9	3.2	3.1	3.1	2.9	3.4
29	The teachers are good at providing feedback to students	1.8	2.0	2.0	1.8	2.1	2.3
30	There are opportunities for me to develop interpersonal skills	3.2	3.4	3.3	2.9	3.4	3.5
31	I have learned a lot about empathy in my profession	2.7	2.9	2.7	2.1	2.9	2.9
32	The teachers provide constructive criticism here	1.4**	1.9**	2.0*,**	1.4**	2.1**	2.4*
33	I feel comfortable in class socially	3.3	3.3	3.4	3.5	3.5	3.6
34	The atmosphere is relaxed during seminars/tutorials	3.2	3.2	3.4	3.4	3.4	3.4
35	I find the experience disappointing	3.1	3.0	2.8	3.3	2.8	3.1
36	I am able to concentrate well	2.7	2.8	2.9	2.8	2.7	2.8
37	The teachers give clear examples	2.8	2.7	2.9	2.8	2.7	3.1
38	I am clear about the learning objectives of the course	3.0	2.7	2.5	3.1	2.3	2.7
39	The teachers get angry in class	3.6	3.4	3.7	3.6	3.4	3.6
40	The teachers are well prepared for their classes	3.0	2.9	3.0	3.0	2.8	3.0
41	My problem solving skills are being well developed here	2.6	3.0	2.6	2.5	2.9	2.9
42	The enjoyment outweighs the stress of studying medicine	2.9	3.4	3.0	3.1	3.0	3.1
43	The atmosphere motivates me as a learner	2.9	3.0	2.6	2.9	2.8	3.0
44	The teaching encourages me to be an active learner	3.0	2.9	2.7	3.0	2.8	2.9
45	Much of what I have to learn seems relevant to a career in medicine	3.4	3.6	3.4	3.4	3.6	3.5
46	My accommodation is pleasant	3.5	3.6	3.7	3.7	3.8	3.6
47	Long-term learning is emphasized over short term	2.8	2.6	2.9	2.9	2.8	2.7
48	The teaching is too teacher-centred	2.9	2.6	2.8	3.0	2.6	2.7
49	I feel able to ask the questions I want	2.4	2.6	3.2	2.7	2.9	3.1
50	The students irritate the teachers	3.2	2.6	2.9	3.0	2.6	3.0
	Age	22.6	25.6	28.0	24.3	25.5	27.7

Notes: Negative items, where scores have been reversed, are in italics. Item scores that indicate real problems (score 2 or below) are bolded.

*Statistically significant difference between the semesters in the two different years.

**Statistically significant difference between semesters 2 and 6 in 2003 and semesters 6 and 10 in 2005, respectively.

Discussion

The results of this inventory indicate that the overall educational climate in this programme is good even during a period of ongoing intense curriculum debate. The total score of 145 is the highest score of those that have been published, although several are in the same range (Al-Hazimi et al. 2004; Varma et al. 2005; Miles & Leinster 2007).

The reasons for the good result could be that the reform in the beginning of the 1990s was in a direction of increased student-centeredness. This was an explanation for the good scores (139) by Dundee University (Al-Hazimi et al. 2004). University of East Anglia, which has a problem-based curriculum, also reported a good score (143; Miles & Leinster 2007), and similar results were obtained in Birmingham where the curriculum has also been reformed (139; Varma et al. 2005). However, Bouhaimed et al. (2009) did not see an improvement in the score after a reform to a PBL curriculum. Other possible explanations could be that students in Sweden have legal rights to be members of deciding boards, and that student evaluations of all courses are compulsory. The students thus have good opportunities to have an influence on their education. We could conclude that our initial worry that the educational climate was bad and thus a cause of the complaints from our students was not substantiated. Our conclusion is rather that we have, in an international perspective, a good climate. Maybe our students make use of this good climate to bring up relevant suggestions for further improvement, and these suggestions could be interpreted as complaints by the teachers.

The semester scores from 2003 were compared. Semester 6 seems to have a better climate than the others (Table 1). However, when the results from 2005 were compared, it seems that a cohort effect could have contributed. The positive results on semester 6 in 2003 were now in semester 10, and actually semester 10 seems to have the best climate in 2005 (Table 1). The course in semester 10 had actually undergone a substantial development between 2003 and 2005, in particular concerning clinical practice, and these improvements could have contributed to the very good scores in semester 10 in 2005. However, in the subscale, ‘perceptions of teachers’, there was a statistically significant increase in this cohort from semester 2 in 2003 to semester 6 in 2005, which probably reflects a genuinely better perception of teachers in semester 6. Two of the items in that subscale concern the relationship between teachers and patients. Students in semester 2 have not seen many patients, and many did not answer that question. Those who did gave rather low ratings, but it is difficult to know the reason.

A difference between semesters in the subscales that was consistent between the years was on ‘academic self-perception’, with a lower result on semester 2. This result is not surprising since students in semester 2 are probably still not confident about studying in university, and the pre-clinical part may seem distant from their future profession, despite early training in patient communication and clinical cases in PBL. Their confidence is likely to improve as they successfully move on in the programme, and the content becomes more clinically relevant. The mean results on the different subscales were all good, being around 70% of maximum results. Gender differences in the results on the different subscales in this study were small, and only on one subscale in one of the years there was a significant difference. This was in contrast to several other studies where there have been differences in the perception of the education climate between the genders (e.g. Roff et al. 2001; Bassaw et al. 2003; Bouhaimed et al. 2009). These good results could actually mask problem areas, and we decided to put more emphasis on the analysis of individual items.

The analyses of individual items revealed certain problem areas in the education programme. Our students did not feel that there was a good support system for students who get stressed (item 3). In all published results of the DREEM inventory, this item score is low so it seems to be a common problem in medical education. At the time of the first inventory (2003), there was a debate about stressed students at the university. The Student Health Service had reported that medical (and law) students were more stressed than others, and another study reported that medical students were more stressed than nursing students (Jonsson & Ojehagen 2006). The introductory course in the medical programme was changed as a result of all those reports in order to relieve the stress in the beginning of the programme. We hope that the significant increase in the score on this item between 2003 and 2005 is a result of this new course introduced in the spring of 2003, but more studies are needed to confirm this result.

Students also did not perceive that they received enough feedback and constructive criticism. Local university evaluations have shown that all students at Lund University felt a lack of feedback from their teachers. The scores on these items in DREEM were actually very low, especially compared to the high overall mean. They were as low, or lower, than results from recent studies (e.g. Roff et al. 2001; Al-Hazimi et al. 2004; Till 2004; Abraham et al. 2008). Since our students definitely felt there was lack of feedback and constructive criticism, we identified this as another area where educational development was needed. During the 2 years between the inventories, a project concerning development of PBL took place. The aims of the project were mainly to develop a more active role for the tutor. We took advantage of this project and included feedback as an important task for the tutor in all workshops and seminars during the project. Also, all workshops on supervision for clinical teachers emphasized feedback. Significant improvements for the scores on feedback and constructive criticism were found in semesters 6 and 10 of 2005 (Table 2). However, no improvement was detected in semester 2, despite the efforts to develop the role of the tutor in PBL. The faculty introduced an extended staff development programme of 10 weeks at the end of 2003. All these efforts in educational development could have contributed to the improved result in 2005. The developments that took part in semester 10 during the 2 years also included more structured feedback to students in their clinical practice.

We have not been completely successful in the reform to a more student-centred, problem-based programme with early introduction of training of communication skills. A substantial group of students still perceived that there was too much factual knowledge to memorize in the short term, they were tired and did not develop their confidence and they were not learning enough about empathy. It is known that such perceptions are correlated to a risk for surface learning strategies, where a risk in turn is less long-term knowledge (Kreber 2003). It has been shown that the students who have negative perceptions of the climate may be the less successful students (Pimparyon et al. 2000; Mayya & Roff 2004). This study shows that there is a room for improvement, in the support to students who have difficulties and in the way we communicate the essentials of the programme to the students. The curriculum introduced in 2005 was intended to counteract the problem of too much factual knowledge. At the time of our second investigation, the reform had just been introduced. The effects of this reform need to be addressed.

This study has helped us to identify problem areas in a medical programme where the climate was otherwise perceived as good. The results were used to direct educational development to weak areas. The results from the second year (2005) indicated that there had been an improvement, although we cannot completely rule out cohort effects as an explanation of the better results in the second year. The interventions and other factors that we believe have contributed to the positive results and to the improvements over the years are summarized in Table 3. We believe that the DREEM instrument can be used to pinpoint areas in need of development. Furthermore, we have shown that the educational climate can be maintained during an intense ongoing educational debate on a reform or alternatively as a consequence of this active debate.

Table 3.  Interventions and other factors that could have contributed to the good scores and to the improvements from 2003 to 2005

Intervention between 2003 and 2005	Contributed to
Improved clinical practice with structured feedback on semester 10	Improvement between years
An introductory course in the first semester to relieve stress	Improvement between years
Faculty development focussing on constructive feedback	Improvement between years
Previous intervention	Contributed to
PBL and the Harvard case method	Overall good results
Other factors
Student representation on deciding boards	Overall good results
A society with equal opportunities for women and men	Small difference between genders

Acknowledgement

We thank Gunnar Hjert for the first translation of the DREEM instrument.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Gudrun Edgren

Gudrun Edgren, PhD, MMedEd, is a senior lecturer and head of the Centre for Teaching and Learning at the Faculty of Medicine, Lund University. At the centre, she teaches on courses and workshops in medical education and takes part in evaluations and research projects in education.

Ann-Christin Haffling

Ann-Christin Haffling, MD, a general practitioner, working halftime as a university teacher, at the time of the study responsible for the organization of the programme in Community Medicine and of the 10th semester in medical programme, Faculty of Medicine, Lund University.

Ulf Jakobsson

Ulf Jakobsson, PhD, is a senior lecturer, the Faculty of Medicine, Lund University. He is a teacher in research methodology and statistics at undergraduate and postgraduate levels in nursing at the Faculty of Medicine, Lund University.

Sean Mcaleer

Sean Mcaleer, BSc, DPhil, is a senior lecturer at the Centre for Medical Education, University of Dundee.

Nils Danielsen

Nils Danielsen, MD, PhD, is a senior lecturer and is the chairman for the Committee for Biomedical, Medical and Public Health Education, Faculty of Medicine, Lund University. This committee makes all strategic decisions related to teaching including financing. ND teaches anatomy and is enrolled as a tutor in PBL.