Summary statistics, educational achievement gaps and the ecological fallacy

Abstract

Summary statistics continue to play an important role in identifying and monitoring patterns and trends in educational inequalities between differing groups of pupils over time. However, this article argues that their uncritical use can also encourage the labelling of whole groups of pupils as ‘underachievers’ or ‘overachievers’ as the findings of group‐level data are simply applied to individual group members, a practice commonly termed the ‘ecological fallacy’. Some of the adverse consequences of this will be outlined in relation to current debates concerning gender and ethnic differences in educational attainment. It will be argued that one way of countering this uncritical use of summary statistics, and the ecological fallacy that it tends to encourage, is to make much more use of the principles and methods of what has been termed ‘exploratory data analysis’. Such an approach is illustrated through a secondary analysis of data from the Youth Cohort Study of England and Wales, focusing on gender and ethnic differences in educational attainment. It will be shown that, by placing an emphasis on the graphical display of data and on encouraging researchers to describe those data more qualitatively, such an approach represents an essential addition to the use of simple summary statistics and helps to avoid the limitations associated with them.

Notes

1. See 〈http://www.data‐archive.ac.uk〉 for further details.

2. One issue to clarify is whether weighting the sample partly in relation to gender will then impact upon the consequent analysis of gender differences. In essence, if a dataset is weighted simply in terms of gender then this will effect the overall estimates produced in relation to the proportions of males and females in the sample. The main reason for doing this is simply to produce estimates for the population that are less biased and thus more representative. However, more specific estimates in relation to the proportion of boys (or girls) achieving five or more GCSE Grades A∗–C or the mean GCSE Scores (and their corresponding standard deviations) for boys (or girls) will remain unaffected.

3. For those who may not be familiar with the notion of the standard deviation, it is basically a measure of the spread of scores. It is estimated that just over two thirds (68%) of scores will be within the range of one standard deviation plus or minus the mean. Thus, in relation to girls in the present study, the standard deviation of 19.0 indicates that two thirds of girls have achieved GCSE Scores within the range 41.6 ± 19.0 i.e. between 22.6 (i.e. 41.6 − 19.0) and 60.6 (i.e. 41.6 + 19.0). From the same calculations it can also be seen that approximately two thirds of boys have also obtained GCSE Scores in the range 18.0 to 56.2. This therefore should indicate to the reader that considerable overlap exists in the respective scores of the two groups.

4. It is sometimes argued that with such a low response rate of just 55% then there is little point using inferential statistics to analyse the data, given that any error due to random sampling that such an analysis would help identify is likely to be completely overshadowed by the errors resulting from non‐response. However, a more pragmatic approach is to make use of as much information as possible. Clearly, a key concern should be to assess what potential biases have been introduced by non‐response and to correct for these as best one can. However, and beyond this, inferential statistics can still play a role in aiding the interpretation of any findings derived from the data. If it is shown with the use of inferential statistics, for example, that a finding could have occurred simply due to the random nature of the sample selected rather than being likely to reflect something real within the population as a whole then this remains useful information. Regardless of the additional biases resulting from non‐response, this would be enough in itself to make us wary of reading much into the finding. However, and conversely, the low response rate would mean that we could not simply accept uncritically any findings that were sufficiently confirmed through inferential statistics. While in such circumstances we would be able to conclude that our finding is likely not to have simply been caused by the vagaries of random sampling we would still need to be mindful of the possible errors introduced by non response.

5. Clearly, it could be argued that this is a simplistic and misleading way to interpret the data given that this is such a crude measure of attainment. More specifically, it could be argued that simply because two pupils are categorised as ‘achieving five or more GCSE Grades A∗–C’ this does not mean they can be ‘matched up’ as suggested here. For example, one may just have five GCSE C grades while the other may have ten GCSE A∗ grades. However, to argue this is rather missing the point. Presumably by this logic, if both students gained ten GCSE A∗ grades then they could be ‘matched up’. However we could then go a stage further. It may be that the first student only just gained the required marks in each GCSE to achieve an A∗ grade while the other student achieved almost perfect scores each time. The key point therefore is that any measure of educational attainment is arbitrary. When examining gender or ethnic differences as we are doing in this article, all we can do is to use the measures that are provided. In relation to the GCSE Benchmark therefore educational attainment is simply measured in terms of whether a pupil gains five or more GCSE Grades A∗–C. By this measure we can ‘match up’ 91.1% of boys and girls. Whether this is an appropriate and valid measure of educational attainment is another matter.

6. In other words, the ‘extra’ 9.9% of girls achieving the GCSE Benchmark compared to boys (i.e. from the figures in Table 3 this is calculated as: 53.8 − 43.9). It will be noted that this is what is commonly referred to as the ‘gender gap’ as discussed earlier. However, it should be apparent from this discussion that it makes no sense to see it in terms of a gap as such as this implies a real distance (of 9.9 percentage points) between all boys and all girls.

7. As with the gender differences discussed earlier, this is the ‘extra’ proportion of Chinese pupils achieving the GCSE Benchmark compared to Bangladeshi pupils (i.e. from 73.4 − 28.4).

8. There are no universal criteria used for deciding where the ‘cut‐off’ point is in terms of designating individual scores as ‘outliers’ or not (Hartwig & Dearing, 1979). SPSS calculates outliers as being beyond two inter‐quartile ranges from the median. Thus, for girls, their median score is 43 and the inter‐quartile range is calculated as 27 (this, it will be remembered, is also the distance between the upper and lower edges of the box). Outliers are therefore classified as those ±54 from this (i.e. 2×27). Thus, in terms of upper scores this would mean anything above 97.