The National Student Survey: Is it just a bad DREEM?

Abstract

Background: In this article, we consider the need for medical schools to improve the overall experience given to students by gaining appropriate feedback and ask whether the UK National Student Survey (NSS) is an appropriate tool.

Aims: We compare the currently used NSS data against data collected via an alternative, well validated, questionnaire – the Dundee Ready Education Environment Measure (DREEM).

Methods: The DREEM data was collected in January to April 2011, from the same cohort of students who were completing the UK online NSS. The NSS results were released into the public domain as frequency tables from which we calculated the standard deviations of each item. The DREEM questionnaire data were rescaled to match the NSS questionnaire data.

Results: The results were similar from each questionnaire, with a wide range of responses. Both DREEM and NSS data showed Assessment and Feedback to be the greatest problem, but the DREEM questions were specific, contextualised and could be used for curriculum development.

Conclusions: This comparison shows the benefits of using a medical school-specific questionnaire to gain quality feedback in order to precisely alter elements of the course rather than relying on a generic questionnaire to gauge students’ opinions.

Introduction

There is an understandable desire to improve the student experience. In the United Kingdom, the drive to do so has increased markedly as the emphasis has shifted from funding through taxation to (partial) funding by the student themselves. As part of the package devised to ensure acceptance of the introduction of fees, all UK University students complete the National Student Survey (NSS; Ipsos MORI 2011), giving their opinions on their student experience. This is a 25 element Likert-style (1 good, 5 poor) questionnaire, and universities are ranked according to their score for ‘overall satisfaction’. Methodological problems with the NSS questionnaire (Yorke 2009) compounded by the importance of identifying real, as opposed to postulated, problems mean that medical schools have to approach the data with care (Cooper 2007). To establish concurrent validity, we compare the results obtained from the NSS with a Dundee Ready Education Environment Measure (DREEM; Roff 2005) completed by the same students.

Background

The goal of any medical education curriculum is to produce graduates who possess the necessary knowledge and problem solving skills as well as the professional attitudes required to function as a doctor (General Medical Council 2009). The educational environment attempts to ensure task orientation on specific scientific goals and social-emotional orientation where the goal is to develop a caring and nurturing attitude to sick people (Dunne et al. 2006).

Table

Practice points

The need for medical schools to improve students’ overall experience has been hampered by limitations in acquiring appropriate feedback via the UK National Student Survey. Methodological difficulties with the NSS questionnaire compounded by the importance of identifying real, as opposed to postulated, problems mean that medical schools have to approach NSS data with care. The DREEM offers a more reliable and precise feedback tool for evaluating study programmes in medical schools than the NSS. The DREEM questionnaire has a more diverse question base than the NSS and is designed for assessing healthcare environments.

The role and responsibilities of the medical school in preparing medical students to join their profession has been fiercely debated for over a century (Flexner 2002). To identify those factors that are of central importance, Roff et al. developed the 50-question DREEM (Roff et al. 1997).

The aim of this tool was to ‘develop and validate a universal diagnostic inventory for assessing the whole or parts of the educational environment and climate of health professions/medical schools’ (Roff et al. 1997), which would allow them to evaluate and develop their programmes of study. The research questionnaire design comprises a combination of qualitative and quantitative techniques meaning that is can be used to develop a non-culturally specific instrument, which is both standardised and validated, enabling the DREEM study to be used globally (Roff 2005).

The DREEM gives a global score out of 200 for the 50 items it contains. It has five sub-scales relating to (i) Students’ Perceptions of Learning, (ii) Students’ Perceptions of Teachers, (iii) Students’ Academic Self-Perceptions, (iv) Students’ Perceptions of Atmosphere and (v) Students’ Social Self-Perceptions. The data taken from the DREEM has a consistently high reliability and allows for data to be collected and analysed alongside variables such as gender, year of study, ethnicity and age (Roff 2005).

The DREEM can be used to generate a ‘profile’ of an institution's strengths or weaknesses as perceived by a cohort of students at any one time (McAleer & Roff 2002) allowing for a cross-section of student's opinions to be sampled and analysed alongside a range of variables.

Methodology

The publically available data (frequency tables) from the NSS has been used in this study. All final year students across the institution were requested to complete the national on-line survey between February and April in their final year. Liverpool Medical School had a published response rate of 57% (180 of 314 students). There are some issues about the sample used, since students who were intercalating a degree in another programme (or even University) in their penultimate year were included in the survey, and those who had returned to the programme after intercalating or suspension were not included. Each of these exceptions account for around 10% of the cohort.

With the permission of the authors, the DREEM questionnaire was adapted to suit the programme at Liverpool. ‘Problem-Based Learning’ (PBL) was included alongside other learning environments, such as ‘lectures and seminars’ (see Appendix).

Questionnaires from 101 participants were analysed, and these were a convenience sample obtained from fifth year medical students through various methods: In Liverpool, all students spend their final year on clinical assistantship rotations, which can present challenges for obtaining a representative sample. Participants were drawn from students attending compulsory University sessions, for example, their group sessions on the primary care placement, at a hospital induction day or when attending their Advanced Life Support and Medical Simulation courses.

This broad spectrum of fifth years allowed for a large enough sample size (approximately 30% of the final year students) to draw impressions of the course from each student's unique experience. Importantly, the participants included students excluded from the NSS (who had re-sat previous years, or intercalated degrees in other subjects).

Students were asked to read the DREEM questionnaire's 50 statements and use the Likert-type scale to respond, which ranged from strongly agree to strongly disagree with the mid-point option of being uncertain. Each statement was then scored between 4 and 0 depending on the statement and how strongly the student agreed or disagreed (McAleer & Roff 2002).

The forms were analysed, giving each student a score out of 200 for course satisfaction. Because the scale used by the NSS ranges from 1 (Definitely disagree) to 5 (Definitely agree), we rescaled our DREEM data to match the NSS scale to allow for comparison. Since the NSS data are only made available as frequency tables, we calculated the standard deviations for each of the items. We recognise that there are difficulties in using Likert item data in this way (Jamieson 2004), but we have followed Norman's argument (Norman 2010) and calculated the standard deviation for the Likert scales in each domain. The level of abstraction applied to the NSS data before release means that analyses based on individual responses to given questions are not possible. The DREEM data was aggregated according to the domains covered by the NSS, the only domain in the NSS not covered in DREEM is ‘Learning Resources’.

Results

The data extracted from the NSS is shown in Table 1, alongside the scores obtained from the equivalent questions from the DREEM questionnaire.

Table 1.  A table showing the mean score for each NSS sub-section for Liverpool Medical School in 2011 compared with the mean score of the DREEM questionnaire data applicable to that sub-section

NSS			DREEM
Item	Average score	Standard deviation	Item	Average score	Standard deviation
The teaching on my course	3.9	0.9	DREEM Q – 1,2,7,14,20,35,37,39,40,48	3.6	0.8
Assessment and feedback	2.8	1.2	DREEM Q – 29,32,38	2.9	1.1
Academic support	3.5	1.1	DREEM Q – 3, 49	3.4	0.3
Organisation and management	3.6	1.0	DREEM Q – 12,24	3.2	0.9
Learning resources	4.3	1.1	No equivalent
Personal development	4.3	0.7	DREEM Q – 10, 16,22,30,31,36,41,44	3.4	0.7
Overall satisfaction	3.7	1.0	DREEM Q – 21,42,45	3.9	0.7

The NSS standard deviation was calculated from the frequency tables, and the raw DREEM data was rescaled from 0–4 to 1–5.

Two things are clear from a cursory inspection of the data. The first is that the scores are similar between the two instruments; the second is that the standard deviations are high. The poorest score is obtained under ‘assessment and feedback’. From the NSS survey, the issues relate to the level of detail and timeliness of the feedback the students receive; and from the DREEM questionnaire, it is additionally clear that there is a lack of clarity about what the students feel they need to know. Although the scores in both instruments are low, the standard deviation of the data is very high, which indicates that opinions within the student body vary greatly.

In the NSS, the students feel that learning resources are good, but there is no direct equivalent to this element in the DREEM questionnaire.

The average overall score for the DREEM questionnaires was 133, with a range of 64–170 (Figure 1). This indicates that, although there are a small number of very dissatisfied students, there were more positive than negative perceptions regarding the course as a whole, but falls short of excellence (151–200) as defined by the original authors of the report (McAleer & Roff 2002).

Figure 1. A graph showing the distribution of scores from fifth Year Students at Liverpool Medical School from the DREEM questionnaire data collected.

With regards to the subscales with the DREEM questionnaire, the results differed, all the average scores for each individual statement were out of four; but in this discussion, we have rescaled them from 0–4 to 1–5. Table 1 shows the responses of the students to the DREEM questionnaire.

The Student's Perception of Learning had a higher overall score 3.6 ± 0.9 (mean ± s.d, n = 12 items, 101 respondents), which meant that according to the DREEM questionnaire, the students learning had ‘a more positive perception’ (Table 2). This included an average score of four in two of the statements – those being ‘I am encouraged to participate in teaching sessions’ and ‘the teaching helps to develop my confidence’. One statement that was scored particularly low in this section was ‘I am clear about the learning objectives of the course’, which scored 2.25 on an average – a score 3 or under is deemed to be a problem area. There is no equivalent section in the NSS. The learning objectives for the programme are given in the programme handbook, and the programme complies with Tomorrow's Doctors (General Medical Council 2009), but students have now been given much more detailed learning objectives on-line.

Table 2.  The results of the DREEM questionnaire, also showing (in italics) the elements not included in the NSS

DREEM domain scores (Items in italics not included in NSS)	Average score	Standard deviation
Students’ perception of learning
DREEM Q – 1,7,13,16,20,21,24,25,38,44,47,48	3.6	0.9
13,25,47	3.6	0.9
Students’ perceptions of course organisers
DREEM Q – 2,6,8,9,18,29,32,37,39,40,49	3.6	0.9
6,8,9,18	3.7	0.9
Students’ academic self perceptions
DREEM Q - 5, 10,22,26,27,31,41,45	3.8	0.9
5,26,27	3.4	1.0
Students’ perceptions of atmosphere
DREEM Q – 11,12,17,23,30,33,34,35,  36,42,43,50	3.6	0.9
11,17,23,33,34,43,50	3.7	0.9
Students’ social self perceptions
DREEM Q – 3,4,14,15,19,28,46	3.8	0.9
4,15,19,28,42	4.0	0.8

The raw DREEM data was rescaled from 0–4 to 1–5.

The Student's Perception of Course Organisers statements had an overall average score of 3.6 ± 0.9 (mean ± s.d, n = 11 items, 101 respondents). The highest scoring items do not figure on the NSS questionnaire. These were ‘the course organisers espouse a patient centred approach to consulting’ (average score 4.1) and ‘the course organisers get angry in teaching sessions’ (average score 4 – this statement was recoded to ensure that disagreement is deemed a positive remark). The only statement in this section to score three or under was ‘the course organisers are good at providing feedback to students’ (2.7 average score). The assessment and feedback section on the NSS also scored poorly (2.8 ± 1.2), which provides some concurrent validation for the data.

The Student's Self Perception scored highly on average with 3.8 ± 0.9 (mean ± s.d, n = 8 items, 101 respondents.) Statements that achieved four or above were:

• ‘I am confident about passing the year’ (4.4)

• ‘Last year's work has been good preparation for this year's work’ (4)

• ‘I have learned a lot about empathy in my profession’ (4.2)

• Much of what I have to learn seems relevant to a career in healthcare (4.1)

One statement in this section that scored three or under was ‘I am able to memorise all I need’, which had an average score of 2.7. There is no direct equivalent for any of these items in the NSS, but they reflect the vocational nature of our programme.

The Student's Perception of Atmosphere scored 3.6 ± 0.9 (mean ± s.d, n = 12 items, 101 respondents.), and this was thought to be ‘a positive attitude’ overall. The highest scoring item ‘There are opportunities for me to develop interpersonal skills’ scored 4 on an average. Which is reflected in the NSS the personal development domain, which also rated very highly 4.3 ± 0.7 (mean ± s.d, n = 3 items, 180 respondents).

Finally, the Student's Social Self Perceptions scored 3.6 ± 0.9 (mean ± s.d, n = 12 items, 101 respondents), which meant that the students thought their social life was ‘not too bad’. The poorest scoring item was ‘there is a good support system for students who get stressed’. This does not figure in the NSS, since the three questions in this domain all relate to purely academic support. Above an average of four were the following statements:

• ‘I have good friends on this course’ (4.4)

• ‘My social life is good’ (4.2)

• ‘My accommodation is pleasant’ (4.2)

Despite these scores, the support system figures in the NSS and the scores were similar (3.5 ± 1.1 (mean ± s.d, n = 3 items, 180 respondents), and reflect a the fact that the organisation of students support systems (both academic and personal) within the University was, at that time, in a state of flux.

Discussion

The DREEM results shown from a reasonable sample size and cross-sectional spread of the final year cohort give a more precise and specific idea to the institution of which areas require further attention in order to enhance and improve future learning and student satisfaction. Overall, there are seemingly only a few statements where the majority of student's found the services of the course lacking.

The mean score of 133 compares favourably to other UK medical schools that have used this instrument (Dunne et al. 2006). The use of the DREEM questionnaire gives clear indications where reform should occur and allows for these reforms to be prioritised. Areas that are in obvious need of reform, in the sample of students taking the DREEM, for example, feedback from educators (a low score of 2.7) can be identified from the data collected, and this one important aspect can become a high priority in order to improve the course overall.

The NSS results were similar to the DREEM results, but suffer from three major drawbacks (Figure 2). Although the DREEM questionnaire was designed and validated for use in many different healthcare environments, the NSS questionnaire was designed for use in a general University environment. NSS assumes a teacher-centred curriculum, whereas the DREEM questionnaire focuses on issues that are important in medical education.

Figure 2. A graph showing the similarities between the NSS and DREEM questionnaire scores (both on a scale of 1–5 (strongly disagree to strongly agree)).

The second, major, issue relates to the way that the NSS data are aggregated before presentation. The individual data are subsumed into frequency tables, so even if one were so minded, it is impossible (say) to determine whether students who felt poorly supported also had issues with the level of feedback they received.

There is the additional problem that the NSS data are further aggregated to give a single measure of percent of students who are ‘satisfied’ with their programme. This brings into sharp relief the greatest problem faced by both the DREEM and the NSS instruments, where students rank items along a five-point scale with a neutral central value. It is difficult to be sure what threshold an individual student has for shifting from a neutral position, and yet in the NSS, neutral is regarded as equivalent to disagreeing.

Problems arose with the DREEM questionnaire when students wanted to be more specific in their feedback. Phrases in the questionnaire such as ‘teaching’ and ‘course organisers’ seemed to be too broad to garner students’ real thoughts about precise aspects. For example ‘teaching’ could be seen as their current consultant or as far back as their first year lectures. In addition, ‘course organisers’ could be viewed as the current head of year, head of PBL, Director of Clinical Studies or the Head of the Medical School. This meant that many students questioned specific aspects of the questionnaire and the objectivity required to view the course overall may not have been present in some students for whom a subjective opinion in one area of the course was a real issue that that particular student wanted to address. These limitations applied to both instruments, but the respondents to the NSS had no way of voicing or highlighting their concerns.

Conclusion

The DREEM questionnaire has a more diverse question base than the NSS; its 50 questions cover a broader scope and give Medical Schools a precise opportunity for reform using a tool that is designed specifically for medical education. The importance of external ‘motivators’ in improving the perceived performance of medical schools is unarguable (Ryan & Deci 2000), but the detailed measures required to improve are best derived for an instrument designed to give managerially useful information. The DREEM tool has greater diagnostic value than the NSS and gives the Medical School the precision for reform to better develop the medical professionals of the future.

Declaration of interest: The authors report no declarations of interest.

Appendix

Dundee Ready Education Environment Measure (DREEM)

Please indicate whether you Strongly Agree, Agree, are Unsure, Disagree or Strongly Disagree with the statements below. It is about how YOU perceive the course.

Please tick the appropriate box.

Table

Question	Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
1. I am encouraged to participate during teaching sessions
2. The course organisers are knowledgeable
3. There is a good support system for students who get stressed
4. I am too tired to enjoy the course
5. Learning strategies which worked for me before continue to work for me now
6. The course organisers espouse a patient centred approach to consulting
7. The teaching is often stimulating
8. The course organisers ridicule the students
9. The course organisers are authoritarian
10. I am confident about my passing this year
11. The atmosphere is relaxed during consultation teaching
12. This course is well timetabled
13. The teaching is student centred
14. I am rarely bored on this course
15. I have good friends on this course
16. The teaching helps to develop my competence
17. Cheating is a problem on this course
18. The course organisers have good communication skills with patients
19. My social life is good
20. The teaching is well focused
21. I feel I am being well prepared for my profession
22. The teaching helps to develop my confidence
23. The atmosphere is relaxed during teaching
24. The teaching time is put to good use
25. The teaching over emphasises factual learning
26. Last years work has been a good preparation for this years work
27. I am able to memorise all I need
28. I seldom feel lonely
29. The course organisers are good at providing feedback to students
30. There are opportunities for me to develop interpersonal skills
31. I have learnt a lot about empathy in my profession
32. The course organisers provide constructive criticism here
33. I feel comfortable in teaching sessions socially
34. The atmosphere is relaxed during seminars/tutorials/PBL
35. I find the teaching experience disappointing
36. I am able to concentrate well
37. The course organisers give clear examples
38. I am clear about the learning objectives of the course
39. The course organisers get angry in teaching sessions
40. The course organisers are well prepared for their teaching sessions
41. My problem solving skills are being well developed here
42. The enjoyment outweighs the stress of the course
43. The atmosphere motivates me as a learner
44. The teaching encourages me to be an active learner
45. Much of what I have to learn seems relevant to a career in healthcare
46. My accommodation is pleasant
47. Long term learning is emphasised over short term learning
48. The teaching is too teacher centred
49. I feel able to ask the questions I want
50. The students irritate the course organisers