Designing your sample efficiently: clustering effects in education surveys

Abstract

Background: Education, and information about education, is highly structured: individuals are grouped into classes, which are grouped into schools, which are grouped into local authorities, which are grouped into countries. The degree of similarity among members of a group, such as a school or classroom, is a very important factor in the design and analysis of studies in education.

Purpose: The aim of this article is to provide information on this degree of similarity within schools to enable those involved in carrying out surveys of schools to do so most efficiently in terms of resources and minimum disturbance of schools.

Sources of data: This paper uses data from 13 studies at primary and secondary level conducted by the National Foundation for Educational Research in England and Wales.

Main argument: The degree of similarity among members of a group is measured by two statistics, the intra-cluster correlation and the design effect. The study described here classifies outcomes into a number of categories and estimates intra-cluster correlation and design effect. The relevance of the results to survey design and analysis is discussed, and examples of how to use these are given.

Conclusions: The main findings of this study, rather than conclusions as such, are the intra-cluster correlations for each topic category. However, the paper reaches some tentative conclusions about the degree of clustering by topic. Using Hox's convention (Multilevel analysis: Techniques and applications, Lawrence Erlbaum, London, 2002) for the size of intra-cluster correlations, it was found that the degree of clustering of achievement was high, while ethnic and language variables were highly clustered in secondary but not primary. By contrast, attitudes towards school, educationally relevant home characteristics, and perception of school policies have quite low values of ρ (mean < 0.05), defined as small.

Keywords:

• design effect
• intra-cluster correlation
• survey design
• education research
• youth culture

Acknowledgements

I am grateful to my NFER colleagues, Liz Cleaver, Lesley Kendall, Ian Schagen and Chris Whetton, for taking the time to read this and earlier versions and making suggestions that have improved the structure of this paper.

Notes

1. Widely but not universally: for a contradictory point of view and discussion see Jesson (2002), Gorard (2002), Schagen and Goldstein (2002).

2. None of the schools samples here was designated single sex, so the degree of variation represents solely natural fluctuation.