‘Student experience’ surveys: some methodological considerations and an empirical investigation

Abstract

The outcomes of surveys of ‘the student experience’ that are sector‐wide, such as the National Student Survey in the UK and the Course Experience Questionnaire in Australia, are of considerable significance for intending students and institutions. Whilst they may satisfy psychometric criteria, some aspects of their design have been submitted to less attention than they warrant. The use – or non‐use – of negatively stated items in, and aspects of response style following the administration of, such instruments are reviewed. An empirical study follows up some of the issues that are discussed. The article as a whole evidences grounds for both comfort and discomfort on the part of those who construct and administer such surveys.

Keywords:

• acquiescence bias
• questionnaire design
• response style
• surveys
• student experience

Acknowledgements

I am very grateful to the following colleagues who facilitated engagement with the various student groups: Alex Douglas, Martin Hudson, Bernard Longden, Bob McClelland, Mark Scott and David Smith. I am also grateful to the anonymous referees who commented on an earlier draft.

Notes

1. In this report, it appears in a number of places that the set of responses runs from ‘definitely disagree’ to ‘definitely agree’. J.T.E. Richardson (personal communication, 23 October 2007) confirms that this order was used in the tabulations in the report in order to align with other aspects of the reporting, and does not reflect the ordering used in the survey instrument (a copy of which appears in Surridge 2006).

2. In practice, there may be relatively little to choose between the two methods of representation. An analysis of the data from the first‐year experience study conducted by Yorke and Longden (2007) showed that the Pearson r correlation between the two methods was +0.98.

3. These items were in any event discarded from the final instrument since they were all concerned, in one way or another, with preparation for assessment: this was not the focus of the survey.

4. However, one difference in respect of Item 8 for the pairing of Variants A and C, and one in respect of Item 2 for the pairing of Variants B and D, came close to being statistically significant.

5. However, two differences (those for Item 2 in respect of Variants A and B, and for Item 9 in respect of Variants C and D) came close to statistical significance.