The Dundee Ready Education Environment Measure (DREEM): A review of its adoption and use

Abstract

Background: The Dundee Ready Education Environment Measure (DREEM) was published in 1997 as a tool to evaluate educational environments of medical schools and other health training settings and a recent review concluded that it was the most suitable such instrument.

Aims: This study aimed to review the settings and purposes to which the DREEM has been applied and the approaches used to analyse and report it, with a view to guiding future users towards appropriate methodology.

Method: A systematic literature review was conducted using the Web of Knowledge databases of all articles reporting DREEM data between 1997 and 4 January 2011.

Results: The review found 40 publications, using data from 20 countries. DREEM is used in evaluation for diagnostic purposes, comparison between different groups and comparison with ideal/expected scores. A variety of non-parametric and parametric statistical methods have been applied, but their use is inconsistent.

Conclusions: DREEM has been used internationally for different purposes and is regarded as a useful tool by users. However, reporting and analysis differs between publications. This lack of uniformity makes comparison between institutions difficult. Most users of DREEM are not statisticians and there is a need for informed guidelines on its reporting and statistical analysis.

Introduction

There is evidence that the educational environment encountered by students has an impact on satisfaction with the course of study, perceived well-being, aspirations and academic achievement (Plucker 1998; Pimparyon et al. 2000; Genn 2001; Lizzio et al. 2002, Audin et al. 2003; Mayya & Roff 2004). In addition to being measurable, the educational environment can also be changed; thus enhancing the quality of the environment and the medical education process itself (Genn & Harden 1986). Innovations in medical curricula and increasing diversity of the student population on medical courses have lead to increasing recognition of both a desire and a need to evaluate the educational environment of medical schools (Miles 2011).

The Dundee Ready Education Environment Measure (DREEM) was designed to measure the educational environment specifically for medical schools and other health professions (Roff et al. 1997). It was developed over 10 years ago using a Delphi panel of faculty members from international medical schools/health professions and then tested on students in several countries for validation purposes. There are other related tools including the pre-cursor to the DREEM, the MEEM (Medical Education Environment Measure) and several subsequent tools that have been designed to measure the educational environment in specific post-graduate medical settings: the PHEEM (Postgraduate Hospital Educational Environment Measure), STEEM (Surgical Theatre Educational Environment Measure) and ATEEM (Anaesthetic Theatre Educational Environment Measure).

Table

Practice points

The DREEM was developed to evaluate the educational environment in undergraduate medical education institutions and has recently been recommended as the most suitable tool for this purpose. The DREEM has been translated into eight languages and used in at least 20 countries. It is used as an evaluation measure to diagnose deficiencies in the current educational environment, to compare different groups’ experiences with the educational environment, and to compare actual experiences of the educational environment with an ideal/expected in the same group. It has also been used to examine the relationship between the educational environment and other measures. Methods of reporting and analysis of DREEM data appear inconsistent. Greater clarity and uniformity of approach in data reporting and analysis would enable meaningful comparisons across institutions and guide good practice.

The DREEM is a 50-statement closed question questionnaire. These 50 items fall into one of the following five subscales: Students’ perception of learning (12 items); Students’ perceptions of teachers (11 items); Students’ academic self-perceptions (8 items); Students’ perceptions of atmosphere (12 items) and Students’ social self-perceptions (7 items). Each of the 50 statements is scored on a five-point scale, with the following labels: “Strongly agree” (4), “Agree” (3), “Unsure” (2) (some publications use the midpoint “Uncertain”), “Disagree” (1) and “Strongly disagree” (0). Reverse coding is required for items 4, 8, 9, 17, 25, 35, 39, 48 and 50. Thus, higher scores indicate a more positive evaluation. The DREEM has a maximum score of 200, representing an ideal educational environment.

The results of the DREEM can be considered at three levels: (i) individual items, (ii) subscales and (iii) overall DREEM. The raw scores obtained for the items making up each of the five subscales are summed for each participant, and then the mean of this summed score is taken to give subscale summary scores. To obtain the overall DREEM score, the subscale summary scores are summed. Examination of the individual items by looking at the mean score obtained across all participants for each item enables the identification of specific strengths and weakness within the educational environment.

In the original article describing the DREEM, the developers reported the achieved total and subscale scores as a percentage of the maximum score possible, but made no recommendations regarding interpretation of the DREEM. Subsequently, two of the developers (McAleer & Roff 2001) provided guidance as to how to interpret scores at each of the three levels. They present a verbal description of a range of scores for the overall DREEM and each of the five subscales. These score descriptors can be used as a guide when interpreting the subscales. Regarding individual items, those with a mean score of ≥3.5 are regarded as especially strong areas, items with a mean score of ≤2.0 need particular attention and items with mean scores between 2 and 3 are areas of the educational environment that could be improved. The developers, however, did not provide any recommendations as to how to analyse the DREEM at any of the three levels should medical educators wish to make comparisons, for example, between different groups of students or over time with the same group of students.

The DREEM is used at the Norwich Medical School, University of East Anglia (UEA), as part of the annual evaluation process so that we can compare our own data over time and across cohorts, and to enable comparisons to be drawn with other medical schools using the DREEM. However, whilst using the DREEM we have become aware that there is considerable inconsistency in the way the DREEM is reported and analysed, which makes comparison between institutions difficult.

The aim of this study was to review, systematically, how the DREEM has been used in the 10 or so years since its development. In particular, to see how evaluators and researchers have reported data collected from DREEM and compared and interpreted the results. In order to meet this aim, we identified articles in the published literature that have used the DREEM and then we reviewed the purposes for which the DREEM has been used, the settings in which it has been applied and the approaches used to report and analyse DREEM data.

Method

The literature review was conducted using Web of Knowledge, with all databases (includes Web of Science and MEDLINE) on 29 March 2010; updated 15 November 2010 and 4 January 2011. The following search terms were used: “DREEM” “Dundee Ready Education Environment Measure” and “Dundee Ready Educational Environment Measure”. Further DREEM articles were identified from references in the Web of Knowledge articles. Only research articles written in English were included; discussion articles, letters and Masters/PhD theses were not included. Only articles reporting the use of the DREEM were included in the review. Studies using the MEEM, PHEEM, STEEM and ATEEM were not included. Articles reporting the translation and/or validation of the DREEM for a different language or for a different group of health professionals were only included if actual DREEM data were reported. Thus, articles were not included if only the results of factor analysis, measures of internal consistency or other developmental issues were reported or if the paper specifically said that the data reported was not true evaluation data and only for the purposes of validation (e.g. Dimoliatis et al. 2010). This resulted in 40 published papers that detailed studies that had used the DREEM. A systematic review was conducted using a custom designed data extraction sheet. Two of the authors read all the articles to check their suitability for inclusion in the review, and to determine what information should be reported in the review table.

Results

Use of the DREEM

Table 1 outlines the DREEM articles identified in the review. Information has been provided about the overall aim of each paper and the specific DREEM-related objective/s, the sample size and type, the country in which the data collection took place and the language of the DREEM (where specified as not being English), and what DREEM data has been reported and how it has been analysed.

Table 1.  Summary data from studies that have used the DREEM

Author/s (year of publication)	Stated Aims	Sample & Setting	Country, language	Reporting, analysis, interpretation
Abraham et al (2007)	Aim to compare the perceptions of academic achievers and under-achievers. And to look at gender differences.	100 of 108 medical students in clinical phase at Melaka Manipal Medical College India 72 academic achievers, 28 under-achievers (have failed something in the past 2 years) 58 male & 42 female. Date of data collection unknown	India	Mean item (all 50) (and SD), subscale and total scores reported for academic achievers and under-achievers. Mean (and SD) item scores reported for males and females, separately by achievement status where a significant difference. T test on subscale and item means to compare achievers and under-achievers, male and female. Interpretation of item scores as per McAleer and Roff's (2001) guidelines.
Abraham et al (2008)	Aim to compare first year and clinical phase students. And to look at gender differences.	108 clinical phase (58 male & 42 female) students and 118 first year (56 male & 55 female) students at Melaka Manipal Medical College, India Data collected July 2006	India	Mean item (all 50) (and SD), subscale and total DREEM scores reported separately for the two groups of students. Mean (and SD) item scores reported for males and females where a significant difference Mann-Whitney used on total, subscale and item means to compare year group, and male and female. Interpretation of item scores as per McAleer and Roff's (2001) guidelines.
Aghamolaei and Fazel (2010)	Aim to examine perceptions of the educational environment	182/210 (86.6%) general medicine students at Hormozgan University of Medical Science, Iran at the time of the study 38.5% male & 61.5% female. 57.7% basic sciences & pathophysiology, 42.3% clinical Data collected 2009	Iran Farsi, Persian (translated for this study)	Mean (and SD) item (all 50), subscale and total scores reported. Mann-Whitney used to compare subscales scores between genders and course (basic science & pathophysiology versus clinical) Interpretation of item scores as per McAleer and Roff's (2001) guidelines. Reliability analysis using Cronbach's alpha conducted following translation of DREEM into Farsi
Al-Ayed and Sheik (2008)	Aim to collect baseline data pre-change in curriculum. To compare students in all five years, and males and females	College of Medicine at King Saud University 223 (out of 500 distributed, 44.6%) responses. 222 analysed. 43 first year, 55 second year, 27 third year, 55 fourth year, 42 fifth year (pre-med year not included). 155 male & 67 female 98 from basic sciences years (years 1 & 2) & 124 from clinical years (years 3 to 5). Data collected 2005	Saudi Arabia Arabic & English available for participants. Arabic version refined for local needs	Total scores reported overall and by pre-clinical vs clinical. Total and subscale scores reported by year separately. Independent samples t test to compare genders for each item and subscale. Total and subscale scores for five year groups compared using ANOVA and Duncan's multiple range test. Total and subscale scores compared with the maximum possible score using single sample t-test.
Al-Hazimi et al. (2004a)	To use DREEM diagnose the environment and climate of a traditional undergraduate medical curriculum. The findings will be used as evidence to support plans to implement a new undergraduate medical curriculum.	450 undergraduate medical students (year 2 = 117, year 3 = 115, year 5 = 120, year 6 = 98) at King Abdul Aziz University, Saudi Arabia. Date of data collection unknown	Saudi Arabia Arabic (unclear if translated for this study, but an Arabic version refined following local piloting)	Mean item (all 50), subscale and total scores reported. No analysis performed – Data commented on descriptively.
Al-Hazimi et al. (2004b)	To compare three traditional medical schools in the Middle East with a modernised medical school in the UK. Also compared year of study and gender.	452 (95%) students at Medical School at King Abdul Aziz University, Saudi Arabia; 278 (75%) at Medical School, Umm Al-Qura University, Saudi Arabia; 179 (81%) at Faculty of Medicine and Health Sciences, University of Sana’a, Republic of Yemen; 145 (78%) Dundee University, UK Refer to source paper for participant breakdown by year of study and gender Date of data collection unknown	Saudi Arabia & UK English for Dundee, Arabic for other three medical schools	Item (all 50), subscale and total scores reported for each university separately Non parametric test used to compare each of 3 middle eastern universities with a UK university at item, subscale and total level. Non parametric test used to compare item scores between genders at each university; items with significant differences reported.
Avalos et al. (2007)	Aim to examine the educational environment prior to implementation of proposed major reforms	389 of 476 (82%) students in Years 1 to 6 at an Irish Medical School Year 1 = 71, Year 2 = 96, Year 3 = 78, Year 4 = 94, Year 5 = 72, Year 6 = 65. 149 male & 237 female 222 preclinical & 167 clinical students 247 Irish & 142 non-Irish students Data collected during 2004/5 academic year	Ireland	Mean (and SD) scores reported for total and subscale scores for whole sample, each year group, and Irish versus non-Irish separately. Percentage of maximum score reported for total and subscale scores for whole sample. T test on subscale and item means between gender, Irish vs non, preclinical vs clinical years. Mean item scores reported for significant comparisons. ANOVA between subscales for each year. Interpretation of total and item scores as per McAleer and Roff's (2001) guidelines.
Bassaw et al. (2003)	To identify the students’ perceptions of elements of the educational environment, with a view to modifying those elements perceived to be in need of remediation.	70 of 82 final year medical students (44 male & 26 female) from the Medical Faculty, the University of the West Indies, St Augustine Campus 36 of 40 medical interns (18 male & 18 female) employed by the Ministry of Health of Trinidad and Tobago who had graduated with the MBBS degree. Date of data collection unknown	West Indies	Total and subscale means reported for all respondents and by the four male/female and student/intern groups. Item (all 50) scores reported for males and females. T tests used to compare genders and intern vs student at subscale and item level.
Bennett et al. (2010)	Aim to compare student experience at 6 different hospitals (small versus large for comparison purposes) providing clinical teaching during the academic year under study	All Year 3 students (n = 108) on the undergraduate medical programme in the School of Medicine, University College Cork asked to complete the DREEM after each of 2 rotations (199/216 returned) 49 DREEM returned for large hospitals & 150 for small hospitals 51:57 male:female ratio 81:18 (9 unknown) undergraduate:postgraduate ratio Data collected during 2008/9 academic year	Ireland	Item (all 50), subscale and total means reported for all hospitals, and small and large hospitals separately. Small and large hospitals compared at subscale and item level using Mann-Whitney-U test. Similar testing at subscale level of English vs non-English mother-tongue, gender and graduate vs undergraduate. Interpretation of total score as per McAleer and Roff's (2001) guidelines.
Bourhaimed et al. (2009)	To compare perception of the educational environment between students undertaking traditional teaching in Year 1 to PBL in Year 2 in a medical school revising its curriculum	202 students at the Faculty of Medicine, Kuwait University. 109 of 127 (86%) Year 1, 93 of 105 (89%) Year 2 students 51 male & 144 female Data collected during 2006/7 academic year	Kuwait	Mean, associated 95% CI and percentage of maximum score reported for total DREEM score and each subscale, for all respondents and by year and gender. T test between male and female, Year 1 versus Year 2 used for total and subscale scores. Logistic regression used to see which subscale scores predict membership of Year 1 or Year 2.
Carmody et al. (2009)	To examine if perception of learning environment of obstetrics & gynaecology unit vary according to delivery site. Also look at differences by gender, and relationship between DREEM and examination results.	Obstetrics & gynaecology unit taught in year 5 of 6 year undergraduate course in Western Australia. 172 students (131 metropolitan, 41 rural) invited to participate. Completed by 94% (81% metropolitan, 20% rural) 55% female, 45% male Date of data collection unknown	Australia	Mean (and SD) of total and subscale score reported for all types of site, gender and exam scores. Total and subscale scores compared between groups using ANOVA and t test for site, gender, high versus low exam scores, and presence or absence of resident specialist obstetrician/gynaecologist at the rural sites. Linear regression used to predict academic scores from DREEM subscales and other independent variables.
Demirören et al. (2008)	Aim to examine perceptions of students in a recently restructured curriculum	Year 1 = 255/285, Year 3 (end of pre-clinical phase) = 192/245, Year 5 (end of clinical phase) = 106/138. Data collected at beginning of second term, January 2007	Turkey Turkish (translated for this study)	Mean item (all 50) scores reported for each year group. Mean (and SD) of total and subscale scores reported for whole sample and for each year group. ANOVA and t test used to compare year groups and pairs of year groups respectively at subscale and item level. Interpretation of item and subscale scores as per McAleer and Roff's (2001) guidelines.
Denz-Penhey and Murdoch (2009)	Aim to compare the findings of DREEM with qualitative interviews asking about curriculum, assessment, site issues	Rural Clinical School of Western Australia 62 students across 10 sites Interviewed students and staff. DREEM completed by students at the same time Data collected 2007	Australia	Mean subscale scores and percentage of maximum reported, and total DREEM score reported for the school overall and separately for each of the 10 sites. Validity study – aim to see whether the descriptive results from the DREEM correspond to qualitative results obtained from the same sample.
Denz-Penhey and Murdoch (2010)	Aim to compare the experiences of medical students undertaking a year long clinical clerkship at different sites to see if those at smaller sites were disadvantaged	Rural Clinical School of Western Austalia 10 sites (4 sites large, 6 sites small) N = 342 Data collected 2007 and 2008	Australia	DREEM used with qualitative and other quantitative data to compare large versus small teaching sites. Mean total and subscale scores for 2008 DREEM reported for large and small sites. P values reported for comparison of small versus large sites but type of test unclear.
de Oliveira Filho et al. (2005a)	To investigate if program violations of the Common Program Requirements (CPR) regarding duty hours, time for educational activity, and faculty supervision adversely affected resident's perceptions of their educational environment and the quality of their lives.	62 of 88 residents (years 1 to 4) from 4 teaching hospitals in Florianopolis, Brazil. Data collected October to December 2003	Brazil Portuguese version adapted (replaced ‘students’ with ‘residents’ and ‘school’ with ‘institution’)	Mean (and SD) only reported when T test was significant. Questionnaire on compliance with CPRs used to define several pairs of groups (e.g. more than 36 consecutive hours on duty yes/no). T test used to compare DREEM subscales between these pairs of groups.
de Oliveira Filho et al. (2005b)	To investigate if the DREEM can be used for medical residents (as opposed to medical students). Comparison with the 50 item Qualify of School Life Scale (for concurrent validity).	Data collected from 12 training programmes (at 6 institutions, in 3 Brazilian cities). Convenience sample of 97 (73%) of the 132 year 1 to 4 residents in anaesthesiology, surgery, internal medicine and paediatrics. Data collected February to December 2003. 33 completed it again January 2004 for test-retest reliability purposes	Brazil Portuguese version adapted (replacing ‘students’ with ‘residents’ and ‘course’ or ‘school’ with ‘residency’)	Mean (and SD), minimum and maximum scores reported for total DREEM and the subscales. Validation of the DREEM for medical residents as opposed to medical students. Psychometric properties of the DREEM as applied to medical residents reported.
de Oliveira Filho and Vieira (2007)	To relate academic performance to other factors including perception of the educational environment at three Brazilian training centres in anaesthesiology	60 of 63 anaesthesia residents at three Brazilian training centres. N = 33 for February 2003 entry residents & n = 30 for February 2004 entry Data collected July 2003 to July 2005	Brazil Portuguese version adapted (replaced ‘student’ with ‘resident’ and ‘school’ with ‘institution’	DREEM data used to predict academic performance. But no DREEM data reported in publication
de Oliveira Filho and Schonhorst (2005).	To compare a new PBL-based 8 day introductory course (2004) to anaesthesiology to the old course (2003). The DREEM was used with other measures at the end of the course.	2003 = 7 first year residents, 2004 = 5 first year residents at the Hospital Governador Celso Ramos. Data collected during 2003 and 2004 courses	Brazil	Subscale and total percentage of maximum achievable (and SD) reported separately for both courses Subscale and total scores for both courses compared using t test.
Dunne et al. (2006)	Aim to examine the educational environment at a UK medical school	DREEM given to all 1370 medical students, 749 (55%) responded. 271 male, 477 female 692 pre-clinical (years 1 & 2), 678 clinical (years 3 to 5) Data collecting during 2002/3 academic year	UK	Mean subscale and total scores reported for each year group, by gender and clinical vs pre-clinical. Mean item scores reported where there was a significant difference (male vs female, clinical vs pre-clinical). Mann Whitney Test, Kruskal-Wallis and analysis of variance used to compare year group, gender and clinical vs pre-clinical at item, subscale and total levels. Interpretation of item, subscale and total scores as per McAleer and Roff's (2001) guidelines.
Edgren et al. (2010)	Aim to identify student perceptions of educational climate and compare perceptions at two time points for a curriculum undergoing reform. And to identify areas in need of more improvement	Lund University, Sweden 503 students. 82% (201/246) in 2003. 75% (194/257) in 2005 58.9% female, 41.1% male To note, cohort in semester 2 2003 was the same as semester 6 2005. Cohort of semester 6 in 2003 same as semester 10 2005 Data collected in semesters 2, 6 and 10 in 2003 and 2005	Sweden Swedish (translated for this study)	Mean item (all 50), subscale and total scores reported for each time point. 2003 compared with 2005 and semesters within a year. Tested using Mann-Whitney U test (years) and Kruskal-Wallis (semesters within a year). Significance level adjusted for the latter as there are three possible comparisons between semesters. Some analysis by gender but no details provided. Interpretation of item, subscale and total scores at each time point as per McAleer and Roff's (2001) guidelines. Cronbach's alpha to assess the internal consistency of the subscales of the instrument but data not reported for each subscale
Jiffry et al. (2005).	To compare the pre-, para- and clinical phases of the curriculum, and gender. Also, to get a baseline as the curriculum is being reviewed.	Medical students in the Faculty of Medical Sciences of the University of Sri Jayewardenepura, Sri Lanka. N = 339. 147 students in pre-clinical (64 male, 83 female), 116 students in para-clinical (58 male, 58 female), 76 students in the clinical (39 male, 37 female) phase of the curriculum. Data collected 2002	Sri Lanka	Mean item (all 50) scores reported (for all three phases combined?). Low means discussed overall and by phase, and items scores significantly differently between males and females reported. Mean (and SD) subscale and total scores, and percentage of maximum score reported separately for the three phases and combined. T test used to compare total, subscales and items between phases, and to compare item scores between genders. Interpretation of item, subscale and total scores as per McAleer and Roff's (2001) guidelines.
Lai et al. (2009)	Aimed to correlate perceptions of educational environment with self-perceived clinical, practical and personal skills competence and readiness to work using abbreviated IMU Student Competency Survey (29 items – developed internally to profile self-perceived competence of students).	Clinical School campus of International Medical University, Malaysia 59 of 71 final year medical students (Year 5) Data collected May 2006 (midway through clerkship)	Malaysia Five questions modified for the authors’ particular setting	Subscale and total DREEM scores reported. Reports correlation (Spearman's correlation) of the DREEM subscales with four domains of IMU Student Competency Survey. Interpretation of subscale and total scores as per McAleer and Roff's (2001) guidelines. Reliability analysis for overall DREEM using Cronbach's alpha reported
Mayya and Roff (2004).	To compare academic achievers and under-acheivers in their perceptions of the educational environment, and to compare genders	Medical students in the clinical years at Kasturba Medical College, India. 508 of 669 students completed DREEM Academic under- achievers = those who had failed at least once in the university exam (n = 105/147. 64 males, 41 females) Academic achievers = had not failed (n = 403/522. 231 males, 172 females) Data collected between February and March 2003	India	Mean (and SD) item (all 50), subscale and total scores reported separately for academic achievers and under-achievers. Mean (and SD) subscale and total scores also reported for whole sample. Item, subscale and total scores compared for academic achievers and under-achievers using T test. Male and Female item scores compared using T test for under-achievers and achievers separately – only significant differences reported. Interpretation of item and subscale scores as per McAleer and Roff's (2001) guidelines. Internal consistency measures for subscales and total DREEM using Cronbach's alpha reported for both groups separately and overall
McKendree (2009)	Aim to investigate perceptions of educational environment for students experiencing the same curriculum, by the same faculty at 2 different campuses, Hull York Medical School, UK. Assessment data also compared for the 2 campuses.	126/139 (61 Hull, 65 York) Year 1 and 90/129 (43 Hull, 47 York) Year 2 students Data collected April of the 2004/5 academic year	UK	Total DREEM scores reported for the two years separately. Subscale and total DREEM scores reported for the 2 campuses separately and combined. The 5 subscale scores compared simultaneously between the two campuses using multivariate analysis of variance. Post hoc tests using Scheffe.
Miles and Leinster (2007)	To compare first year medical students’ expectations of the medical environment they are about to enter at the beginning of the year with the actual environment experienced at the end of the year. Also compared gender and educational background	87 /130 (66.92% of the starting cohort) first year students at the University of East Anglia, UK completed the DREEM on both occasions and gave permission for their data to be published. 30 male, 57 female 46 school leavers, 17 access to medicine, 24 graduates Data collected beginning and end of 2004/5 academic year from same students	UK	Mean (and SD) actual and expected scores, and the difference (dissonance) scores reported for each item, subscale and total. Wilcoxon signed rank test for related samples used to compare actual and expected DREEM scores for each subscale. Mann-Whitney-U used to compare subscale dissonances between genders. Kruskall-Wallis used to compare subscale dissonances between the three educational backgrounds. Significance level adjusted for multiple comparisons (Bonferroni).
Miles and Leinster (2009)	Aim to compare the views of students regarding their educational environment with staff perceptions of how the students see the educational environment	403 undergraduate medical students across Years 1 to 4 at the University of East Anglia 73 staff involved in teaching for this academic year Data collected at end of 2005/6 academic year	UK	Mean (and SD) scores for items (all 50), subscales and total DREEM reported for staff and students. Total and percentage Agreement, Disagreement and Unsure reported for staff and students for all 50 items. Staff and students subscale and items scores compared with Mann-Whitney U test, with corrected significance level (Bonferroni). Chi squared test to compare categorised item scores between staff and students with corrected significance level (Bonferroni).
O’Brien et al. (2008)	Aim to collect data about student experience for the new Bachelor of Science Nursing programme at the National University of Singapore	National University of Singapore 49 Year 1 nursing students asked to complete the DREEM in all modules of two semesters 6 modules in semester 1: 273 (out of 294 distributed) DREEM questionnaires completed. 5 modules in semester 2: 182 (out of 245) completed. Data collected during the 2006/7 academic year	Singapore Revised for Singapore cultural and educational background.	For each semester separate cluster analysis performed; two clusters identified per semester. Subscale means compared using t test and subscale category of score (using developers cut-offs) using chi squared test, between pairs of clusters. Factor analysis and Cronbach's alpha calculated to validate the revised questionnaire in Singaporean context.
O’Rouke et al. (2010)	Aim to evaluate the new 100 item Medical Student Stress Profile instrument (MSSP). Inclusion of the DREEM to explore relationship between stress and perceptions of the environment (NB DREEM described as 30 item, but this is likely to be an error).	Three cohorts of students at the School of Medicine, University College Cork, Ireland. 2007 and 2008 final year students and 2008 first year graduate entry medical students. 239 medical students Data collected in 2007 and 2008	Ireland	Correlations of the DREEM subscales and total DREEM with the 12 subscales of the MSSP calculated as part of the validation of the new MSSP instrument. But, no actual DREEM data reported in publication.
Riquelme et al. (2009)	Aim to find out student perception of the educational environment following major curriculum reform, and to compare year of training. Also to check the internal consistency of a Spanish version of DREEM	Pontificia Universidad Católica de Chile Medical School. 90.5% of undergraduate medical students in Years 3, 4 and 5 completed DREEM (Year 3 = 108 of 110; Year 4 =106 of 108; Year 5 =83 of 110). Data collected December 2005	Chile Spanish (translated for this study)	Mean (and SD) item (all 50), subscale and total scores reported for each year group and overall, and percentage of maximum score reported for subscale and total scores. ANOVA used to compare the three year groups at all three levels. Interpretation of item scores as per McAleer and Roff's (2001) guidelines. Cronbach's alpha calculated to check the internal consistency of the Spanish version
Roff et al. (2001)	To explore the value of the DREEM in ‘diagnosing’ the environment of educational institutions and making comparisons of two culturally diverse situations.	127 students from a Nigerian undergraduate medical school (year 4 = 22, year 5 = 36, year 6 = 69); 110 males, 27 females. 86 students from the BP Koirala Institute of Health Sciences in Dharan, Nepal (year 1 = 29, year 2 = 29, year 3= 28); 70 males, 16 females. Date of data collection unknown	Nigeria and Nepal	Mean item (all 50), subscale and total scores reported for each country. Items compared between genders, between year groups and between country using t tests. Only significant results reported for gender and year group. Subscale and total means compared between countries. Subscale means compared between gender and year group for Nepal. Descriptive comparison of the results for the two countries.
Sobral (2004)	To measure medical students’ perception of their first year medical studies experiences using the 36 item Course Valuing Inventory (CVI). This was compared with the DREEM to test the relevance of the CVI as a measure of educational ambience.	41 first year medical students at the University of Brasilia completed the DREEM. DREEM data collected 2000 (other data collected 1997 to 2000)	Brazil	Mean (and SD) total DREEM score presented. Correlations between DREEM total score and CVI factors used to validate the CVI. Correlations between subscale scores and CVI factors discussed but data not reported.
Thomas et al. (2009)	Aim to collect information about student perceptions of the educational environment because the data hasn’t been collected before; and to compare first and final year students.	63 first year, 63 final year students at Manipal College of Dental Sciences, India Date of data collection unknown	India	Mean (and SD) item (all 50), subscale and total scores reported separately for both first and final year students. Compare first and final year students using students t-test. Interpretation of item scores as per McAleer and Roff's (2001) guidelines.
Thomé et al. (2006)	Investigating student and teacher views about a portfolio method of assessment for a biomedical laboratory science programme. DREEM used to assess educational climate.	Lund University, Sweden 21 students completed the DREEM Data from 18 Year 4 medical students at the same institution also reported for comparison purposes Data collected 2002/3 academic year	Sweden	Mean item score within subscale given for study sample, and compared descriptively with those of a sample from the same medical school and another published sample.
Till (2004)	The Canadian Memorial Chiropractic College is in the fourth year of implementing a new four-year Integrative Curriculum. This study aimed to determine how students are experiencing the new educational environment using the DREEM.	The Canadian Memorial Chiropractic College, Toronto, Canada 148 (96%) first year, 131 (87%) second year and 128 (84%) third year students (NB the fourth year students were the last of the old curriculum, so were not included). 54% female, 46% male Data collected February 2002	Canada	Mean item, subscale and total scores reported for each year group. Gender calculated, but not reported. T test between pairs of year groups and genders. The percentage of those with positive, uncertain and negative responses calculated for each item, but not all reported. Attention drawn to items where responses are divided between positive and negative (bimodal), and those with a large number of uncertain responses. Items where majority of respondents indicated problems listed and discussed.
Till (2005)	To compare the ideal educational environment with actual environment experienced and to see if this information can be used to identify areas needing attention by looking at areas of dissonance. Also comparing gender and year of study.	N = 342 first, second and third year students at the Canadian Memorial Chiropractic College, Toronto, Canada. Completed Actual DREEM followed immediately by Ideal DREEM Actual: 148 (96%) first, 131 (87%) second, and 128 (84%) third year students. Ideal: 146 (95%) first, 123 (82%) second, and 73 (48%) third year students. Data collected February 2002	Canada	Mean actual and ideal item scores presented for all 50 items, with their difference (dissonance), for the three year groups. Total and subscale actual and ideal scores given or reported elsewhere. Items ranked according to dissonance, and described for year groups and for each gender. Unspecified statistical test of dissonance compared between pairs of years, and for each year between gender for item scores.
Tokuda et al. (2010)	Aim to investigate relationship between graduates’ perceived preparedness for postgraduate training with educational environment and pass rate on the National Licensure Examination	DREEM and preparedness in 6 clinical areas questionnaire data from 2429 first year residents (36% response rate). This provided DREEM data for 80 medical schools in Japan Data collected November 2008 (middle of 2008/9 academic year)	Japan Japanese (translated for this study but noted as being similar to other published Japanese version/s)	Reports correlations (Spearman) of mean DREEM scores (total and subscale) and measures of preparedness and pass rate percentages for 80 medical schools (NB unclear if these correlations between DREEM and pass rate were for both total and subscale DREEM as no detail provided).
Varma et al. (2005)	Aim to compare student experience at 8 different teaching hospital sites.	206 final year Birmingham University medical students (100% response rate) asked to rate final year module in Obstetrics and Gynaecology. 42% male, 58% female Data collected in 2000	UK	DREEM subscale means and total reported for each site and overall. Percentage of maximum score available also reported overall for subscales and total DREEM. Subscale scores compared between 8 hospital sites using ANOVA.
Vieira et al. (2003)	To evaluate a questionnaire designed to examine emerging attitudes during the development of a doctor – patient relationship. The DREEM was also completed to assess the programme's educational environment and climate.	41 first year medical students, enrolled on a teaching programme concerned with the attitudes in the doctor-patient relationship taking place at the outpatient clinic of the General Medicine Service, University of São Paulo Date of data collection unknown	Brazil	Total and subscale scores reported as means and percentage of total possible score. Items with means below 2 reported. No analysis performed – The DREEM used as a descriptive outcome. Interpretation of item scores as per McAleer and Roff's (2001) guidelines
Wang et al. (2009)	Aim to see if the Chinese version of DREEM works with nursing students	214 of 221 nursing students (Year 2 = 54, Year 3 = 58, Year 4 = 53, Year 5 = 49) at the School of Nursing of China Medical University. Year 1 students had attended a programme with a different curriculum so were excluded. Data collected between December 2004 and January 2005	China Chinese	Validation of the DREEM for Chinese nursing students. Factor analysis and Cronbach's alpha reported Included comparison of their data with other studies in the Discussion.
Whittle et al. (2007)	Aim to evaluate all five years of the medical school and identify areas where improvements needed using the DREEM, an open ended question and subsequent further investigation.	968 students completed DREEM, 86% response rate (Year 1 = 212, Year 2 = 225, Year 3 = 194, Year 4 = 167, Year 5 = 170) Date of data collection unknown	UK	Mean item (all 50) scores reported separately for each year and all years combined. No analysis performed – Weak areas discussed in light of open ended questions. Interpretation of item scores as per McAleer and Roff's (2001) guidelines.

Notes: Stated aims: Aims specifically stated as being under investigation in the study (additional analyses are reported in the “Reporting, Analysis, Interpretation” column).

Sample and setting: Setting of study, total sample size and group sizes (where groups are compared) as given by authors, date of DREEM data collection (where available).

Country, Language: The country in which the study took place, the language where stated as not English, and details of any revisions to the ‘standard’ DREEM.

Reporting, Analysis, Interpretation: First paragraph describes which DREEM data has been reported and for what. Second paragraph details the analysis that has been performed, including types of statistical test and the groups being tested. The third paragraph, if present, indicates whether McAleer and Roff's (2001) guidelines regarding interpretation have been referred to. Italics at the end of entries in this column indicate where the DREEM has been used for ‘non-standard’ purposes or where the psychometric properties of the DREEM have been examined.

Through the 40 papers identified in the published, peer-reviewed literature, it is found that the DREEM has been used in 20 countries, including: Australia, Brazil, Canada, Chile, China, India, Iran, Ireland, Japan, Kuwait, Malaysia, Nepal, Nigeria, Saudi Arabia, Singapore, Sri Lanka, Sweden, Turkey, the UK and the West Indies. It has been translated into at least eight languages (Arabic, Chinese, Japanese, Persian, Portuguese, Spanish, Swedish, Turkish) for evaluation purposes.

The DREEM has most commonly been used with medical students in undergraduate medical schools (n = 31 papers); but it has also been used with other groups including graduates (interns/residents) (Bassaw et al. 2003; de Oliveira Filho et al. 2005a; 2005b; de Oliveira Filho & Schonhorst 2005; de Oliveira Filho & Vieira 2007; Tokuda et al. 2010), and nursing (Wang et al. 2009), dental (Thomas et al. 2009) and chiropractic (Till 2004, 2005) students, and teaching staff (Miles & Leinster 2009).

It has been used to explore a number of issues, which can be broadly classified into four groups:

1. Diagnostic

   • Investigating perception of the current educational environment and identifying areas of strength and weakness (e.g. Roff et al. 2001; Bassaw et al. 2003; Vieira et al. 2003; Dunne et al. 2006; Whittle et al. 2007; Thomas et al. 2009; Aghamolaei & Fazel 2010).

   • Measuring the existing educational environment in the established medical schools as a pre-cursor to curriculum change, both to identify priority areas for change and to act as a baseline for comparison after curriculum change (e.g. Al-Hazimi et al. 2004a; Avalos et al. 2007; Al-Ayed & Sheik 2008).

   • Investigating the impact of a new/revised curriculum on perception of educational environment (e.g. Till 2004; Demirören et al. 2008; O’Brien et al. 2008; Riquelme et al. 2009; Edgren et al. 2010).

2. Comparison – Different groups

   • Comparing different medical education institutions (e.g. Roff et al. 2001; Al-Hazimi et al. 2004b).

   • Comparing students at different stages of their medical training.

     • Clinical versus non/pre-clinical (e.g. Jiffry et al. 2005; Dunne et al. 2006; Avalos et al. 2007; Abraham et al. 2008).

     • Students in different year groups (e.g. Al-Hazimi et al. 2004b; Dunne et al. 2006; Al-Ayed & Sheik 2008; Demirören et al. 2008).

     • Students versus graduates (e.g. Bassaw et al. 2003).

   • Comparing gender (n = 16 papers, e.g. Mayya & Roff 2004; Dunne et al. 2006; Abraham et al. 2007; Al-Ayed & Sheik 2008; Bourhaimed et al. 2009).

   • Comparing different sites of learning (e.g. clinical placement location or university site) (e.g. Varma et al. 2005; Carmody et al. 2009; McKendree 2009; Bennet et al. 2010; Denz-Penhey & Murdoch 2010).

   • Comparing academic achievers with those students are who under-achieving (e.g. Mayya & Roff 2004; Abraham et al. 2007).

   • Comparing staff and students (e.g. Miles & Leinster 2009).

   • Comparing perception of an old course with a new course following curriculum change (including traditional versus problem-based learning) (e.g. Bourhaimed et al. 2009).

3. Comparison – Same groups

   • Comparing the actual educational environment experienced by students with what they expected on starting medical school (e.g. Miles & Leinster 2007).

   • Comparing the actual educational environment experienced by students with their ideal/preferred environment (e.g. Till 2005).

4. Relationship with other measures

Examining the relationship between perceptions of the educational environment, as measured by DREEM, with other measures, such as:

• Academic performance (e.g. de Oliveira Filho & Schonhorst 2005).

• Self perceived competency/preparedness to work (e.g. Lai et al. 2009; Tokuda et al. 2010).

• Stress (e.g. O’Rouke et al. 2010).

• Qualitative evaluation data (e.g. Denz-Penhey & Murdoch 2009).

• Aspects of working environment (e.g. de Oliveira Filho et al. 2005a).

• Another student experience instrument (e.g. Sobral 2004).

• An assessment measure (e.g. Thomé et al. 2006).

The published literature on DREEM also includes studies investigating if DREEM can be used with another group of participants (e.g. de Oliveira Filho et al. 2005b = residents; Wang et al, 2009 = nursing students) or in another language (e.g. Riquelme et al. 2009).

Reporting and analysis of DREEM data

The majority of authors report the means of individual items, subscales and the total score. Some authors report the standard deviation as well as the mean (n = 15 papers), but most do not. In a few cases, the score as a percentage of the maximum possible scores is presented (Jiffry et al. 2005; Varma et al. 2005; Avalos et al. 2007; Bourhaimed et al. 2009; Riquelme et al. 2009). Some report scores for all 50 DREEM items; others only for a subset (e.g. those which are particularly low and need attention). A couple report the percentages of respondents who agree, disagree or are unsure for each item (Till 2004; Miles & Leinster 2009). Where the aim is to diagnose problem areas, several authors have adopted the threshold for the mean approach recommended by the DREEM's developers as reported above (i.e. items scoring less than or equal to 2.0 need attention, with some authors adopting a stricter less than or equal to 2.5 threshold) (n = 14 papers). Some authors compare their results with other published results, usually in the “Discussion” section of their article.

Where scores from two separate groups of students are compared, reporting is sometimes purely descriptive. However, usually either an independent samples t-test (parametric) or Mann–Whitney U test (non-parametric, equivalent to Wilcoxon Mann Whitney) is used. This choice does not appear to be related to sample size. Some authors report test results for all 50 items, perhaps with significant items highlighted, but some report significant items only (e.g. Dunne et al. 2006). Where items are reported by the percentage who agree, disagree, etc., a chi-squared test is used between groups.

When three or more groups require comparison, both the one way ANOVA (parametric) and the Kruskal–Wallis test (non-parametric) have been used on subscale scores and the total, and sometimes on item scores. Sometimes adjustment is made for multiple tests (e.g. Bonferroni in Miles and Leinster 2007), but sometimes not. Sometimes tests between particular pairs of groups are performed.

Where scores are to be compared with an expected or maximum possible score, the Wilcoxon signed rank test is used or the single sample t-test. One paper (Till 2005) ranks the items in order of the mean difference between the actual and expected scores to indicate which areas score furthest from the ideal.

The DREEM has also been used to investigate the association between perception of the educational environment and other measures (e.g. Lai et al. 2009; O’Rourke et al. 2010), for instance, using correlation or linear regression. Here the selection of a method will depend on the nature of the other measures.

Some papers whose primary aim is to validate the use of the DREEM in a new type of environment also report their data as evaluation data, as described above (e.g. de Oliveira Filho et al. 2005b; Wang et al. 2009), in addition to reporting the results of statistical analysis specific to the validation goal (e.g. factor analysis, calculation of Cronbach's alpha co-efficients).

Discussion

The DREEM is a validated measure designed specifically to evaluate the educational environment of trainee health professionals. It has been used internationally in different settings, predominantly with medical students but also with other healthcare groups. The literature reporting the use of the DREEM clearly indicates that those who have used it have found it to be a helpful tool for a number of evaluation-related purposes, including: diagnostic, comparison of different groups, comparison of the same group under different conditions and examining the relationship of the educational environment with other measures.

However, the review also shows that there is little consensus on how DREEM data has been both analysed and reported. Both parametric (e.g. t-test, analysis of variance) and non-parametric (e.g. Mann–Whitney–U, Kruskal–Wallis) tests have been used. Whereas the choice of differing approaches could be due to the purpose of the test or the size of the sample, this does not seem to be the case. Those involved in routine evaluation are not likely to have a statistical background, so such inconsistencies could lead to confusion and possible misinterpretation of the areas for change. Furthermore, they make it difficult to interpret the obtained DREEM data in one institution in comparison to that collected elsewhere.

As detailed above, often papers only report means for those individual items where there was a significant difference between subgroups of interest. Often authors are constrained in what they can report due to journal imposed limits on the number of words or tables allowed. Evaluators using the DREEM for internal course evaluation purposes would have no such constraints. To enable useful comparisons of DREEM data between institutions, ideally the means need to be reported for each item, broken down by any key subgroups such as gender or year of study. Where this is not possible for practical reasons, authors should make their data available to any interested party.

One could argue that most evaluation is conducted with the goal of improving the educational environment at the institution performing that evaluation, thus there is no need for comparisons. It is possible to diagnose the educational environment of your institution completely independently, identifying your own strengths and weaknesses using DREEM. However, it can also be argued that it is useful to know whether issues and concerns raised by students at your institution are common to other institutions. Furthermore, lessons can be learnt about how other institutions are dealing with areas of weakness that may assist in remediation of these same areas at your own institution. For example, scores on the item “The teachers are good at providing feedback to students” are commonly low, and whilst medical educators receiving a low score on this item will likely want to act to improve their performance in this area so that student satisfaction is increased, it may be of interest and possibly reassuring to know that students at other institutions also rate this area as poor. Moreover, other institutions may have introduced innovative strategies to combat this problem that your institution could take on board too. But for this to happen, researchers need to report DREEM data to the same level of detail, and conduct a comparable analysis.

Examination of the published literature suggests that the DREEM has not been used, as yet, for profiling students. In the future when investigating, for example, predictors of success at medical school medical educators may find it useful to include the perception of educational environment as a possible predictor, utilising DREEM scores

Recently, following a review of the available instruments, Soemantri et al. (2010) concluded that DREEM was likely to be the most suitable instrument for measuring the educational environment in undergraduate medical institutions. Such a review may lead to a further increase in the frequency with which the DREEM is used and its data are reported. This reinforces the need for consistent analysis and reporting, where possible, to enable the DREEM to be used more effectively and consistently across institutions.

The lack of uniformity and clarity about the best way to analyse and report DREEM, and the fact that those who undertake course evaluation are not usually statisticians identifies the need for some clear, evidence-based recommendations on how the DREEM should be processed by evaluators, both for routine evaluation and for publication.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.