Assessing the impact of Pupil Premium funding on primary school segregation and attainment

ABSTRACT

Pupil Premium funding has been provided to schools in England since 2011, to reduce socio-economic segregation, and the attainment gap between disadvantaged pupils and their peers. There is little evidence it works. Some stakeholders are now considering whether funding should cease, or have a new objective. Evaluating the impact of such a funding policy is not easy because the relevant outcomes are sensitive to demographic, economic and other concurrent policy changes. These issues are addressed using the National Pupil Database, comparing the poverty gap in primary schools from 2006 to 2019, focusing on pupils who would have attracted Pupil Premium funding, if it existed, in any year and under any economic conditions. After 2010, the segregation of long-term disadvantaged pupils and their peers reduced in Years 1 and 6, and their attainment improved relative to their peers at a ges 7 and 11. However, assessments changed after 2014 and this complicates the pattern. A regression model suggests that relative attainment for poor pupils improved markedly in the Pupil Premium era, even in regions like the North of England which have faced criticism for apparently “failing“ their poor pupils. We argue for retaining the Pupil Premium policy.

KEYWORDS:

• Disadvantage
• Pupil Premium funding
• social segregation
• primary school attainment
• poverty attainment gap

Introduction

In 2011, Pupil Premium funding was introduced in England. The funding was given to all state-funded schools in proportion to their intake of disadvantaged pupils, with disadvantage largely defined in terms of eligibility for free school meals (FSM). The money is intended to provide assistance and interventions to improve the attainment of poorer pupils, and so reduce the poverty attainment gap. Nine years later, this paper provides strong evidence of the likely impact of this funding on the socio-economic segregation of poorer pupils between schools, and on the attainment gap. Previous attempts to assess these impacts have been problematic in various ways (Gorard, Siddiqui, and See 2019). Here these problems are addressed by incorporating changes in economic indicators over time, using data from well before 2011 up to 2019, and most notably by focusing on long-term disadvantaged pupils who would presumably have attracted Pupil Premium in any era (were it available), and under any economic or policy circumstances.

The paper looks briefly at some of the international evidence on targeted funding intended to improve educational outcomes, and how this is best delivered. It then summarises the Pupil Premium policy in England, the prior attempts to assess its impact, and the various limitations of these. The paper describes the methods used, presents the results, and considers some implications for policy from our new large-scale study.

What is the prior evidence on this approach?

Although the focus of this paper is on poverty, financial poverty is not the only kind of educational disadvantage (Craske 2018). And, as this paper shows, poverty is often linked to other indicators of possible disadvantage such as having a special educational need. However, using money to try and improve attainment for poorer pupils is a widespread policy that has been used by governments and other agencies for decades (Children’s World Report 2020), where poverty is assumed to be a barrier to the attainment of academic objectives (Rolle, Houck, and McColl 2008). There are many variations, with money given to the school system to deploy, or to areas with high levels of disadvantage, to families either for them to deploy or as an incentive (so that parents encourage school attendance, (for example), to teachers, or to the disadvantaged students themselves as an incentive (Toutkoushian and Michael 2007).

There is little robust evidence on which, if any, of these approaches is more effective. Large experimental studies suggest that simply paying students to improve does not improve their intrinsic motivation, and does not lead to improved attainment, largely because students say that they do not know how to improve (Fryer 2010). Paying students to improve the inputs into education production, such as for attendance or classroom behaviour where students know what to do, might be more effective. In a large experimental study where students were paid for academic achievement in five core subjects, there was only a benefit in maths, not for reading or other subjects (Bettinger 2010).

So far, the evidence on paying students directly is not strong, but it might be a better approach than providing the money to teachers. In a large experiment looking at attainment in maths, there was improvement in a treatment group wherein students were paid if they increased their scores, and in a group wherein students, teachers, and school leaders received the incentive, compared to a control group. There was little or no improvement for a group wherein only teachers were paid extra if students improved (Behrman et al. 2015). Channelling extra funding to attract and retain teachers in disadvantaged areas and schools has also not been found to be very effective (Clotfelter, Ladd, and Vigdor 2005; Feng 2014). It generally does not make any difference once the funding ceases (See et al. 2020).

What about paying families/parents? Lack of parental engagement in children’s learning and school life is not a major causal factor in equalising the academic outcomes of disadvantaged children. Funding programmes to support parental involvement of disadvantaged children have not shown very promising effects for attainment outcomes (Gorard and See 2013). Some evidence comes from Riccio et al. (2010), where low-income families were offered cash rewards for engaging in activities related to children’s education (among a range of other things such as health and employment). There were more than 50 outcomes measures, making the claims for the success for any one outcome very weak. There appeared to be a small benefit for maths scores for one year, and some improvements in attendance.

A large quasi-experiment, with standardised outcome measures, suggested that providing extra resources for the schools themselves led to greater gains than incentives for teachers did (Lavy 2002), even though other trials suggest that simply giving extra money to schools has no direct impact on test scores (Mbiti et al. 2019). The Pupil Premium is perhaps more like the additional funding that schools in England already received for individual pupils with statements of special educational needs or disability. The evidence on the impact of such funding for pupils facing challenges is mixed but reasonably promising overall (Henry, Fortner, and Thompson 2010; Holmlund, McNally, and Viarengo 2010). Both PP and SEN funds are intended to assist those pupils facing long-term challenges in accessing the curriculum. However, while SEN funding is largely tied to individual and their identified needs, Pupil Premium funding does not have to be spent on the individual who attracted it.

What is the Pupil Premium intended to achieve?

From April 2011, ‘additional’ Pupil Premium (PP) funding was allocated to publicly-funded schools in England in proportion to their intake of pupils from low-income families known to be eligible for free school meals (FSM), and children who had been looked after by the state continuously for more than six months (Education and Skills Funding Agency 2018). There is a smaller premium for pupils having parents in the armed forces, or whose parent(s) had died while in the armed forces (Foster and Long 2018). By 2018, PP was funded at £2,300 per looked-after pupil, and £1,320 per pupil ever FSM-eligible at primary school, with schools receiving an average of around £150k per annum.

On average, socio-economically less advantaged students have lower attainment outcomes at school, and poorer opportunities once they have left school (Lessof et al. 2018; Gorard and Siddiqui 2019; Hanushek et al. 2019). Schools are required to use PP funding to support low-income and other target pupils, but they can choose how to do this. Most is spent on interventions to help improve the attainment of low attaining disadvantaged students (Fellows and Barton 2018). The reduction of the achievement gap between children coming from richer and poorer families was a key purpose of PP funding (Gov.uk 2010), and schools are increasingly judged in terms of their results for Pupil Premium pupils (DfE 2019).

Having poorer children clustered in particular schools is potentially damaging in a variety of ways – such as lowering aspiration and participation for individuals, and reducing social and ethnic cohesion (Danhier 2018; Hewstone et al. 2018; Bhattacharya 2021). It is often more damaging for lower-attaining and more disadvantaged pupils (Dickerson, Maragkou, and McIntosh 2018). So, another objective of the funding from the outset was to reduce social and economic segregation between school intakes. The funding was devised to ‘act as an incentive for schools to stop cream-skimming children from wealthier areas as they will lose out financially’ (Freedman and Horner 2008, 41), and was introduced to help reduce ‘any disincentive that schools might have to recruit such pupils’ (Gov.uk 2010).

Reducing the poverty attainment gap and lowering social segregation between schools are the two objectives assessed in this paper.

The prior evidence on Pupil Premium impact

There have been no robust prior studies of the impact of Pupil Premium funding on SES segregation between schools, although Gorard, Siddiqui, and See (2019) present a time series analysis from 1989 to 2018, which suggested an unexpected decline in between school segregation during the Pupil Premium era. But that analysis used only school-level data, and disadvantage was defined as eligibility for FSM in one year, which is unstable and so not really comparable over time. As shown in that study, the proportion of FSM-eligible pupils clearly changed annually with the economic conditions, and this created a confound for the analysis. The proportion of pupils registered as FSM-eligible changed historically from nearly 20% of the school population to just above 11%. When there are more FSM-eligible pupils both segregation and the attainment gap appear to decline in a way that has nothing to do with improved attainment or reduced clustering of the poorest pupils.

Work on the raw-score attainment gap suggests that it has been declining to some extent since 2011, but perhaps no more than it declined before 2011 in some accounts. According to the Education Policy Institute (2017), the gap has been reducing very slowly and erratically since at least 2007, with no clear difference before and after 2011 The Social Mobility Commission (2019) also report that the gap has been falling slowly, but again not necessarily more so since 2011. As with analyses conducted by the DfE, Ofsted, EEF, Demos, and the National Audit Office among others, prior studies have several problems. Some only look at data since 2011 which cannot portray a change since the policy was introduced. For example, DfE (2018c) used a version of the attainment gap that had not been used before 2011, and so any changes they portray cannot be attributed to a policy change in 2011 (there is no counterfactual). Prior studies generally take no account of changes over time in the economy, the legal definitions of indicators of disadvantage, the impact of the incentive to register pupils as FSM-eligible because of the Pupil Premium, or the amount of missing data, and so the prevalence of disadvantage. Analyses are complicated by changes in the metrics used, the curriculum, and in the way attainment has been assessed and scored (see below).

Worse, the length and depth of any indicator of disadvantage is usually ignored and, because such factors are clearly related to attainment and progress (Gorard 2018), previous estimates of the attainment gap at any level of aggregation are problematic (e.g. Exley 2015; National Audit Office 2015; Public Accounts Committee 2015; Education Endowment Foundation 2017,; DfE 2018b). Average attainment is lower for long-term FSM-eligible pupils, and this means that simply computing an attainment gap between Ever- and never-FSM pupils will seriously misrepresent the challenges faced by poorer areas and schools with poorer intakes, and the chances of their pupils making good progress. The problems cannot be avoided by using value-added or Progress 8 scores. Despite value-added being intended to be independent of the underlying raw-scores, it is not (nor is it stable, Gorard 2018). The same flaw creates a situation where the only group that has average positive value-added progress in England consists of those pupils who are never eligible for FSM. All groups of pupils ever eligible for FSM for any amount of time have negative progress scores, and these scores are substantially lower for every year that a pupil is known to be eligible. There is no obvious reason why progress scores (as opposed to absolute attainment) should behave like this. The longest term disadvantaged pupils have very low progress scores indeed, and therefore years, areas or schools with proportionately more of this group will appear to have lower progress and higher attainment gaps as well.

Another factor that is ignored in previous presentations of the attainment gap at any level of aggregation over time is the proportion of pupils in any area or year not in state-maintained schools. If the pupils attending private provision are not included in the attainment gap calculation, because it is not clear whether they are FSM-eligible or not, this may distort the result.

A substantial part of the apparent change in the official attainment gap in any year, area or school is therefore linked to factors such as changes in private school attendance, the economy, and the duration of poverty (none of which are strictly anything to do with attainment levels, but which become confounding variables unless accounted for). For a fuller discussion and evidence on these difficult issues see Gorard, Siddiqui, and See (2019).

What we need is a definition of disadvantage that can be used for analysis of cohorts long before and long after 2011, and a dataset that contains common variables and measures available over all of that time. Under current conditions it is not possible to use actual measures of household income (HMRC have repeatedly refused access). We need to identify pupils who would have been eligible for Pupil Premium funding in any era, even when there was no funding, and under any legal, economic, or other conditions (see methods section).

A recent study by Hutchinson, Reader, and Akhal (2020) took our advice, and looked at outcomes for what they called the ‘persistently disadvantaged’ pupils. They compared results for these long-term FSM-eligible pupils with the results of pupils who had never been eligible (excluding from analysis those who had ever been eligible in the past six years, but not persistently so). This is an advance on prior work, but it still has the problem of sensitivity to legal, economic and other changes over time that are nothing to do with schools, teachers, or attainment. As we have shown previously (and again below), the temporarily disadvantaged students are more like the never disadvantaged than like the long-term disadvantaged. Changes in the economy may be more likely to affect them, for example, and so some years will include pupils as never disadvantaged who might have been temporarily so during a recession. It is safer always to compare the stable disadvantaged group (however defined) with the equally stable group of all of the remaining pupils in any cohort (and not have an excluded middle group).

In the same way, Hutchinson, Reader, and Akhal (2020) excluded non-standard schools like Pupil Referral Units (PRUs) from their figures, even though PRUs are part of the Pupil Premium scheme, and numbers in PRUs can also change over time according to policy, practice and prevalence. This affects the apparent attainment gap because potential PRU students tend to have lower average attainment, and are more likely to be disadvantaged. In years when more are in mainstream settings their scores will increase the estimated gap. This issue is of less relevance for primary schools perhaps, but the same kind of thing happens over time with intakes to specials schools, hospitals, and private schools.

National assessments in primary schools

Key Stage 1 (KS1, attainment at age 7) and KS2 (attainment at age 11) outcomes at primary school in England were reported using a standard scale until 2016. This was based on levels 1 to 8, with 2 being the expected level at age 7 KS1, and 4 the expected level at age 11 KS2. Each level was sub-divided into categories a, b and c and each category was given a notional point score, with level 1 c being 7, level 1b is 9, 1 c 11, 2 c 13 and so on.

Since 2005, KS1 levels have been based more on teacher assessment and KS2 levels on statutory tests. In 2016, levels were replaced at KS1 with descriptor categories assessed by teachers for reading, writing and maths (which can be mapped onto the old point scores) and scaled scores from 85 to 115 from tests (with 100 being the expected standard). Pupils who do not achieve the lowest scaled score on the test would be assessed by the teacher using one of the new categories for pre-KS1 standards. Teacher assessment in English was revised for 2018, with new criteria for tasks that pupils have shown that they can do. Teacher assessments for all subjects were revised for 2019. For these reasons, the analysis below presents KS1 scores (not levels) as standardised z-scores, to assist comparisons over time.

There was a widespread boycott of KS2 tests in 2010, and many results are missing nationally. For these reasons, the attainment scores for 2010 should be largely disregarded in what follows, otherwise they may make results after 2010 look misleadingly better. The KS2 curriculum changed in 2014, and the relevant cohort faced new tests in 2016 (DfE 2016). In 2016, KS2 students were assessed using scaled scores matched against a new curriculum. These scores range from 80, the lowest possible scaled score, to 120, the highest possible scaled score. A scaled score of 100 or more means that the child has met the expected standard in each KS2 SATs test; a scaled score of 99 or less means they have not reached the expected standard. As with KS1 we present the key results below in terms of z-scores in an attempt to even out these changes in metrics, to some extent.

Methods

The problems outlined above with the existing evidence base means that we need a new and stronger analysis focusing on whether Pupil Premium has made a difference. Here the analyses are based on the idea in Gorard, Siddiqui, and See (2019) of using the most stably disadvantaged group who presumably would have been labelled disadvantaged in any year and under any conditions – the long term FSM-eligible. This group is compared to all of the other pupils in the cohort, including those who had been only temporarily FSM-eligible. In the absence of an RCT or similar, this is perhaps the strongest way to test impact. For comparison we have also run all analyses with disadvantage defined as FSM-eligible in the year of the outcome measure (e.g. Year 6 for KS2), and as ever FSM-eligible (in the six years at primary school). But these analyses are not generally presented in this paper.

The datasets

The data comes from the combined National Pupil Database (NPD) and Census datasets, held by DfE and accessed via ONS, for the Key Stage 2 (KS2) cohorts for school years 2005/6 to 2018/19, with indicators from the previous KS1. All pupils in state-maintained schools at KS2 are included (regardless of school type, including PRUs and special schools), but only those in the correct age band for that school year. And only variables that appear for all year cohorts can be used. For example, in later NPD versions there are variables for pupils living in state care, and those whose parents are in the armed forces. Such information could be used to help define Pupil Premium status more accurately, even though the vast majority of eligibility is based on FSM. However, this cannot be done for earlier cohorts, and to use it for only some cohorts would appear to change the outcomes of the disadvantaged group in a way that was unrelated to Pupil Premium Policy or schools or teaching.

The 2006 primary cohort is the earliest we can use with reasonably complete information about their prior KS1 characteristics and outcomes. Even so, this cohort has incomplete information about their FSM status in the first year. We could ignore this year, and start with 2007. However, given that the 2010 attainment data is affected by the boycott, this would give us only three clear cohorts before 2011 (2007–2009). So, the analysis is from 2006 to 2019, and the headline figures are based on those pupils known to be eligible for at least five years (out of six) while at primary school. All analysis have been run with six years (and not 2006) as well. The results are substantively the same.

New indicators

A new indicator was created registering how many years each pupil had been FSM-eligible for, at each Key Stage, as an indication of that stable subset of pupils living in long-term poverty. Other new indicators were the percentage of all pupils known to be FSM-eligible, and the number missing data on FSM, for each year and KS.

All KS1 and KS2 (average English and maths) points scores were converted to z-scores. Results are presented for both the raw- and standardised z-scores. A simple progress measure was created by regressing individual prior attainment score (KS1 points) on later attainment score (KS2 points). All models predicted later attainment with R of around 0.7 to around 0.8. The residual score (difference between predicted and observed KS2 scores) was saved as a new indicator for each pupil.

A measure of segregation was created for each individual, showing how clustered (or otherwise) their school was in terms of pupil intake known to be eligible for FSM. This segregation residual was computed as the number of FSM-eligible pupils in an individual’s school divided by the number of FSM pupils in the dataset (all maintained schools in England). From this we subtracted the number of all pupils in the schools divided by the number of pupils in all schools (Gorard 2015).

From outside the NPD, we added data for each year on GDP and GDP annual growth, as an indicator of economic change over time, and on annual private school attendance from DfE (2020). We also grouped cases into nine official Economic Areas.

Analysis

All 14 datasets for KS2 cohorts were aggregated into one, retaining only those attainment, school, area and background variables common to all. There were 7,956,233 cases. The first step in the analysis was to produce descriptive figures. Pupils missing data on one or more key variables had a new category added for ‘missing’ or, where the variable is a real number, a new flag variable created to mark whether that number is missing or not (and mean score used to replace the missing value). This retains all cases successfully so that all analyses use the same N, while respecting missing data and permitting it to be assessed accurately in the context of each finding (Gorard 2020).

Background characteristics such as sex, age-in-year, language, ethnicity, and special educational need status were cross-tabulated with disadvantaged and non-disadvantaged groups, defined in terms of long term FSM-eligibility or not, current FSM-eligibility or not, and ever FSM-eligible or not.

The same disadvantaged categories were used to compare means for attainment, value-added residuals and segregation residuals, and so create ‘effect’ sizes computed as the difference between the means for any disadvantaged group and the rest, divided by their overall standard deviation. To aid comparison, all effect sizes are presented in line graphs as positive (absolute) values. This becomes clearer when explained in the text. Strictly, line graphs are not the most appropriate for 14 years of data points, but they are more readable. Readers just have to remember that the graphs are based on 14 points and are not continuously measured. The underlying exact figures, with their signs, appear in the Appendix. Times series graphs are presented for GDP change over time, the percentage of long-term FSM-eligible pupils, and the percentage in private schools and so excluded from NPD and the official attainment gaps. These are used as context for changes in segregation and the attainment gap.

A simple multiple regression model is also presented, predicting/explaining the attainment gap for each year at KS2. It has two steps. The first step has three predictors to represent economic and related changes – GDP change, percentage of long-term FSM-eligible pupils, and percentage in private schools. The second step uses a flag variable set to 0 if the year is before 2011, and 1 otherwise.

Findings

All KS2 cohorts, 2006 to 2019 combined

Using the overall dataset for years 2006 to 2019, with around 8 million cases, it is clear that the few pupils missing FSM data have attainment more like the majority not-disadvantaged pupils at KS1, and more like the disadvantaged pupils at KS2 (Table 1). In fact, their KS2 scores are worse than the disadvantaged groups, forming a kind of super-disadvantaged group, as shown previously at KS4 (Gorard 2012). Given that the dataset is based on KS2 cohorts, the missing values at KS1 are most likely for those pupils coming to primary schools in England later, after Year 1, from other home countries, overseas, private schools, or home tutoring. Those missing FSM data at KS2 (but having KS2 scores) may be more mobile pupils such as recent evacuees or from Traveller families.

Table 1. Comparing KS1 and KS2 scores for pupils missing FSM data with the rest.

	Not FSM-eligible	FSM-eligible Year 6	Missing FSM data
KS1 Points	15.92	13.55	15.09
KS2 Points	52.31	47.52	37.40

Note: 12,340 cases are missing KS2 FSM data

Table 2 shows that for each outcome measure there is a worsening trend linked to longer-term disadvantage. The gap between never FSM-eligible pupils and the rest is large, with only the never-eligible group having a positive value-added score for KS2 (i.e. all of the other pupil groups make differentially negative average progress at primary school). Similarly, and not surprisingly, only the never-eligible pupil group generally attend schools with fewer than their fair share of disadvantaged pupils (segregation). However, what has not been so clear until recently is that the picture worsens in all aspects for longer-term disadvantaged pupils. In fact, the gap between pupils only FSM-eligible for one year and those always eligible is usually larger than that between pupils only FSM-eligible for one year and those never eligible. Again this shows why using current FSM status or EverFSM6, as happens in official analyses and even much academic research, does not provide a fair picture of the situation.

Table 2. KS scores, value-added, and segregation linked to length of known disadvantage.

	Never FSM-eligible	1 year FSM-eligible	2 year FSM-eligible	3 year FSM-eligible	4 year FSM-eligible	5 year FSM-eligible
KS1 Points	16.16	14.74	14.38	14.24	13.89	13.48
KS2 Points	51.89	55.27	47.80	51.94	49.94	42.90
KS2 VA Residual	0.04	−0.06	−0.07	−0.07	−0.10	−0.13
FSM segregation	−0.04	0.05	0.06	0.07	0.09	0.12

There is a clear difference in the extent to which long-term FSM-eligible pupils (eligible for every year while at primary school) have historically been segregated from their peers (effect size 0.97), and in their attainment at KS1 (effect size of 0.59). There is also a small difference at KS2 both in raw-score and value-added terms (Table 3). These attainment differences are smaller than some readers might expect because those pupils FSM-eligible for only five years are included here in the (technically) not long-term disadvantaged group, as are those only eligible for four years and so on. These issues are examined further below.

Table 3. Comparing KS scores, value-added, and segregation between long-term disadvantaged pupils and the rest.

	<6 years FSM-eligible	6 years FSM-eligible	Overall Standard Deviation	‘Effect’ size
KS1 Points	15.72	13.31	4.09	−0.59
KS2 Points	51.53	49.73	35.03	−0.05
KS2 VA Residual	0.01	−0.16	1.15	−0.15
FSM segregation	−0.01	0.15	0.16	0.97

Long-term FSM-eligible pupils are slightly more likely to be non-white (especially black or mixed) in ethnic origin, and/or to have a first language other than English. Long-term FSM-eligible pupils are over twice as likely to be recorded as having a special educational need (SEN) or disability of some kind (Table 4). This factor alone could account for a large part of the difference in attainment at KS1 and KS2, and in the attainment gap otherwise apparently related to poverty.

Table 4. Percentage of pupils listed as having SEN, for disadvantaged groups and the rest.

	Never FSM-eligible	FSM-eligible Year 6	6 years FSM eligible
No SEN listed	81	63	60
SEN no statement	2	6	7
SEN statement	16	31	33
Unknown	1	-	-

Note: ‘Statement’ includes those with an Education, Health and Care Plan. ‘No statement’ includes School Action, School Action Plus and SEN support.

Identifying the comparison group over time

Figure 1 illustrates why, for the purposes of assessing the impact of Pupil Premium funding, the basis for comparison should be the long-term or permanently disadvantaged (the lower line in the graph). They are much less affected by changes in legal definitions of FSM-eligibility and variations in economic conditions, than the temporarily disadvantaged group. The proportion is relatively stable, and we can assume that the vast majority would have been labelled FSM-eligible in any era.

Figure 1. Percentage of long-term and temporary FSM-eligible pupils, by KS2, 2006–2019.

Precise figures appear in the Appendix, Table A1

The group representing pupils who had been FSM-eligible for one or more years but not permanently (the top line in Figure 1) is much more volatile over time, ranging from under 15% to over 21% of the total. Figure 2 repeats Figure 1 at a different scale, and includes the majority of pupils each year who are never FSM-eligible. Because the long-term FSM-eligible group is so stable, the never eligible group is largely the inverse in shape of the temporarily eligible group. Both are volatile. Comparing the long-term and stable eligible group with either of the others would still lead to results influenced by the prevailing economic and legal conditions. It is not a solution, as Hutchinson et al. (2020) did, to use the never eligible group. However, combining all except the long-term eligible creates a second ‘flat’ group (the changes in the top two lines largely cancel out). This is what is done in this paper.

Figure 2. Percentage of long-term, temporary and never FSM-eligible pupils, KS2, 2006–2019.

Annual changes for KS1 cohorts 2006 to 2019

Looking at pupils known to be eligible for FSM in their first year at primary school, it is clear that the clustering of FSM-eligible pupils within specific schools (socio-economic segregation) was flat from 2006 to 2011, but has declined since 2011 (Figure 3). There is a strong possibility that this welcome decline in socio-economic segregation between schools is linked to change in the ‘attractiveness’ of disadvantaged pupils when allocating schools places, as planned when Pupil Premium funding was introduced, and as already identified based on a school-level analysis at KS4 (Gorard, Siddiqui, and See 2019). The imapct of the funding incentive on Year 1 school places could be immediate. It has been noted before that when a change of policy is linked to changes in segregation, the biggest changes occur while each school is filling up with pupils arriving since the change in policy (Gorard 2015). Perhaps something similar is happening here. By 2017, a full six cohorts had arrived in their schools during the Pupil Premium era, and graph shows signs of flattening out.

Figure 3. Change in ‘effect’ size for the gap between FSM-eligible pupils and the rest, FSM Segregation Year 1.

Note: in order to make changes over time clearer these graphs do not have zero as their origin. The exact figures appear in Appendix Table A2.

Figure 4. suggests that the difference in attainment between long-term FSM-eligible pupils and the rest has also declined in the Pupil Premium era. Importantly, this improvement in equity occurred at the same time as an improvement in raw KS1 scores for both groups (see Table A3 in the Appendix). The figures show a period of volatility from 2006, with no overall pattern until 2011, and then a decrease in the poverty attainment gap for the years following. There was a widespread boycott of KS assessments in 2010, and so the results (a sudden apparent growth in the gap) for this year may not be directly comparable. The changes in assessment from 2015 (see above) may have led to a temporary increase in the gap. By 2019, the gap is at a historic low.

Figure 4. Change in ‘effect’ size for the gap between long-term disadvantaged pupils and the rest, KS1 Points z-scores, 2006–2019.

Note: in order to make changes over time clearer these graphs do not have zero as their origin. The exact figures appear in Appendix Table A4.

As would be expected, the gap is lower at every stage between pupils who are EverFSM6 and the rest, then between those FSM-eligible in one year and the rest. The gaps are largest for the long-term disadvantaged, who would presumably have attracted Pupil Premium funding in any era, whatever the conditions in place. Looking at this most disadvantaged group, it is clear not only that their attainment has improved, but also that they are slowly catching up with those pupils who have only been temporarily FSM-eligible and with the pupils never eligible. For KS1 cohorts, it looks as though the Pupil Premium funding era is linked to success in meeting two of its objectives – reduced segregation of school intakes, and a lower poverty attainment gap.

In order to help illustrate these changes, Figure 5 shows the ‘effect’ sizes for the attainment gap between long-term FSM-eligible pupils and the rest, for five Economic Regions of England over time. There is a decline in all areas since 2011. The gap has declined particularly strongly in the North West and West Midlands (and London; the figures for all regions are in Appendix Table A5). These areas are important to note because they have been criticised for overly large poverty gaps, and for, in effect, failing their poor pupils. In fact, it is areas like the South East and South West that have the highest gaps and lowest average attainment for the pupils facing long-term poverty.

Figure 5. Change in ‘effect’ size for the gap between disadvantaged pupils and the rest, KS1 Points z-scores, by Economic Region, 2006–2019.

Note: in order to make changes over time clearer these graphs do not have zero as their origin. The exact figures appear in Appendix Table A5.

Annual changes for KS2 cohorts 2006 to 2019

As with pupil intakes in Year 1, socio-economic segregation between primary schools in Year 6 (end of KS2) has declined since 2011, having been flattish since 2006 (Figure 6). By 2019, the extent to which long-term FSM-eligible pupils are clustered in schools with others like them is at a historic low. This is good news, uniquely associated with the PP era.

Figure 6. Change in ‘effect’ size for the gap between long-term disadvantaged pupils and the rest, FSM Segregation KS2.

The exact figures appear in Appendix Table A6.

The picture for the attainment gap at KS2 is not as clear as for segregation, or for the attainment gap at KS1 (Figure 7). The attainment gap in 2019 is substantially lower than in 2006, and indeed much lower than in 2010. However, the gap grew in 2014, and then again from 2016 with the changes in KS2 assessment and scoring, and there is no clear ongoing trend of improvement since the introduction of Pupil Premium funding. More research is needed to try and take account of these abrupt changes in assessment, which have not been picked up by prior analysis using the official definition of disadvantage (see above). It is noteworthy that the former permanent secretary at the DfE said that ‘after 2015, the focus on the attainment gap … diminished’. The Conservative government was more focussed on academies, and felt that narrowing the attainment gap was ‘more the contribution of the Lib Dems’ who had formed part of the coalition government in 2010 (Whittaker 2021).

Figure 7. Change in ‘effect’ size for the gap between disadvantaged pupils and the rest, KS2 Points.

The exact raw and z-scores figures appear in Appendix Tables A7 and A8.

Figure 8 is based on the approach used by DfE (2014), and subsequently, to overcome abrupt changes (see above) by comparing the average ranking of attainment for disadvantaged pupils and the rest. Their argument is that giving each pupil a national ranked score on any assessment, rather than a points score, makes different assessments and grading systems more comparable over time. Figure 8 shows that this approach does not overcome the change in difficulty in 2014. So it is still not clear what is happening here in relation to Pupil Premium funding.

Figure 8. Change in ranking gap, KS2 Points.

The figures appear in Appendix Table A9.

At KS2 it is also possible to consider the progress made by disadvantaged groups between KS1 and KS2. Figure 9 shows the gap in simple value-added residuals between long-term disadvantaged pupils and the rest. The gap was declining until 2013. The changes to assessment in 2014 and again in 2016 are associated with a reversal of this historic trend. Now, the most disadvantaged groups face an increasing attainment gap, with long-term FSM-eligible pupils making less and less relative progress at primary schools. This can explain some of the pattern in Figure 8. Insofar as value-added scores are meaningful, they suggest that the reforms to scoring for KS2 have disproportionately affected long-term disadvantaged pupils, and undone much of the progress in reducing the attainment gap since 2006. Two policies appear to be working against each other here. By their nature, measured as progress between one KS and the next, these KS2 VA scores may be worsened by the very improvement noted for KS1 in Figure 4.

Figure 9. Change in ‘effect’ size for the gap between disadvantaged pupils and the rest, KS2 VA Residual.

The attainment gap at KS2 is again substantially lower in the North and the West Midlands, and has decreased further since 2010, despite the change in national scoring and value-added since 2014 (Figure 10). The decrease has been most marked in the North East. These are all notable findings, because these are the areas that policy-makers and other commentators has criticised for having high attainment gaps. As with KS1 it is areas like the South East and South West that have among the highest gaps for long-term disadvantaged pupils, and have shown the least improvement over time.

Figure 10. Change in ‘effect’ size for the gap between disadvantaged pupils and the rest, KS2 Points z-scores, Economic Regions 2006–2019.

The figures for all areas appear in Appendix Table A10.

The economy

There are reasons, other than Pupil Premium funding and changes in test severity, why the poverty attainment gap may change over time. One of these is changes in the economy. When the economy is weaker there are more FSM-eligible pupils. Length of poverty is linked to lower average attainment, and so these newly eligible pupils will have higher overall attainment. And so the apparent FSM versus non-FSM gap may reduce. Also, in better economic times more students may attend private schools (largely excluded from NPD and official measures of the gap), which could have the same effect by taking somewhat higher attaining scores out of the calculation (Gorard, Siddiqui, and See 2019). This is a key reason why the emphasis in this paper is on the long-term FSM-eligible who would have been disadvantaged and not attending private schools in any era under almost any economic circumstances.

However, there is no clear link between either the attainment and segregation gaps, and changes in GDP 2006–2019 (Figure 11). All of the economic indicators used here have been converted to z-scores for comparability. For segregation (above), there are indications from 2006 to 2013 that it is partly the inverse of GDP, rising when GPD declines and vice versa. However, from 2013 onwards segregation continues to fall while GDP improves slightly. This suggests that another, non-economic, factor like PP is the cause. There is also no clear link between GDP change and the percentage of long-term FSM-eligible pupils.

Figure 11. Annual changes in GDP, 2006–2019.

Insofar as we can tell using PISA data, the segregation of the (very few) poorest pupils in private schools has remained relatively static since 2004 (Martínez-Garrido, Siddiqui, and Gorard 2020). However, even in PISA the data on private school intakes for England is weak. DfE (2020) figures show that the proportion of pupils in private schools has declined slightly since 2011 (Figure 12), and the possible impact of this on increasing the attainment gap is investigated in the next section via a regression model.

Figure 12. Percentage of pupils attending private schools, 2006–2019.

Predicting the annual attainment gap

The final stage in the analysis for this paper is a simple regression model, used to predict/explain the national KS2 attainment gaps 2006 to 2019, using economic indicators and a flag representing the Pupil Premium era or not. This makes it possible to consider the potential impact of Pupil Premium funding net of economic changes by looking at the latter first.

Table 5 shows that it is possible to explain almost 75% of the variation in the attainment gap using only three economic indicators and the Pupil Premium flag, and that the vast majority of the explanation comes from Pupil Premium. Put another way, the attainment gap for the relatively stable group of long-term disadvantaged pupils is slightly affected by economic conditions, but the best single predictor (by far) of the gap in any year is whether it is during or before the Pupil Premium era.

Table 5. Strength of model predicting KS2 attainment over time.

Step	R	R-squared
1. Economic indicators	0.34	0.12
2. Whether in Pupil Premium era	0.86	0.74

Table 6 shows the coefficients for all four predictors. All other things being equal, having more pupils in the private sector is associated with an apparent decrease in the attainment gap, and vice versa, as predicted by Gorard, Siddiqui, and See (2019), because those on the margins of private schools will tend to have higher than average attainment. In any year where most of these are in state-funded schools they are included in the NPD gap calculation and so the apparent gap is higher, whereas if they are removed from the calculation the gap is lower. When the economy is better, the gap grows slightly, and also when there are slightly fewer long-term disadvantaged pupils, as there would be when the economy was better. The gap changes with the economy because the temporarily disadvantaged (or advantaged) have somewhat higher average attainment than the permanently disadvantaged (see above).

Table 6. Coefficients for model predicting KS2 attainment over time.

Coefficients	Standardised
GPD change	+0.34
Percentage of pupils FSM-eligible for 5+ years	−0.25
Percentage of pupils in private schools	−0.82
Pupil Premium era flag	−1.40

The Pupil Premium era flag has the largest predictive coefficient, and more importantly produces by far the largest increase in R-squared. In the Pupil Premium era, net of other factors included here, the attainment gap is clearly lower. This is not a test of causal influence, and there are many other variables that we might wish to include in the model, but it is clear that something quite remarkable has happened since the advent of Pupil Premium, even for the KS2 attainment gap.

Implications

Some commentators are saying that the Pupil Premium funding is not always used properly by schools, or is otherwise not being effective. They are calling for the money to be used for school general funds (Allen 2018; Morris and Dobson 2020), or used to recruit, develop and retain teachers (Staufenberg 2019). With this kind of uncertainty about the impact and future of the policy, it is important to have good evidence of its effectiveness, and whether it is in need of modification or cancellation.

Unfortunately, from a research perspective, Pupil Premium funding was introduced suddenly and without experimental trial. It was introduced nationally across England, meaning that there is no ‘clean’ comparator. Without robust evaluation it is not possible to say definitively whether the funding is the cause of the improvements noted in this paper, or merely a strong correlate. However, by using the most stable disadvantaged pupil group, the most years of data possible, and factoring in economic and other changes, this paper presents the strongest evidence so far that the Pupil Premium has worked, and is working (2020 data are not available, due to the lockdown).

The evidence presented in this paper is generally positive. Since the extra funding was introduced in 2011, socio-economic segregation between primary schools has declined in a way not predicted by longer term historical figures. This decline is evident both for FSM-eligible pupils when they arrive in Year 1, and for pupils who had been FSM-eligible for five or more years by the end of primary school in Year 6. There is an equivalent drop in the attainment gap at KS1. There was the beginning of a reduction in the attainment gap at KS2, and the gap is still lower than historically. But is has grown again from 2014 onwards, and this is linked to changes in difficulty, scoring, and value-added scores. It seems that the Pupil Premium policy may be being contradicted by other governmental interventions, but has not been picked up in official accounts wherein the attainment gap is strongly affected by changes in the economy and other confounds. Overall though, we can say that the Pupil Premium seems to have worked from the perspective of long-term disadvantaged pupils. They are less clustered in schools, have better KS1 scores, and somewhat better KS2 scores than before 2011.

It would make sense to assume for the present that Pupil Premium should be retained in something like its current form, perhaps with a greater focus on KS1, and calibrated to increase funding for the long-term disadvantaged. It is not like just giving extra money to schools, which prior studies have suggested is less effective, because the funding must be used for the purposes intended, and the government created a parallel programme, including the EEF, to provide guidance to schools on how best to spend it. The policy also differs from one based on giving money to poorer families on condition that their children attend school (Baird, McIntosh, and Özler 2011; De Janvry et al. 2006; Morley and Coady 2003), or to regions, as with the Opportunity Areas (DfE 2018a) or Excellence in Cities policies in England, and it differs from a policy based on extra funding for schools themselves that is not tied to their intake, such as the original Specialist Schools and then the Academies programme (Gorard 2005).

The paper also shows that areas of England like the North and the Midlands (along with London), with the greatest proportion of long-term disadvantaged pupils, tend to have the lowest attainment gaps and show the greatest improvement over time. If there are lessons to be learned about how to deploy Pupil Premium funding, and so improve life chances for poorer children, these are the areas that policy-makers and other commentators should look at for advice (rather than denigrating).

Acknowledgments

This work contains statistical data from ONS, which is Crown Copyright. The use of the ONS statistical data in this work does not imply the endorsement of the ONS in relation to the interpretation or analysis of the statistical data. This work uses research datasets, which may not exactly reproduce National Statistics aggregates.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Stephen Gorard

Professor Stephen Gorard is Director of the Durham University Evidence Centre for Education (DECE) https://www.dur.ac.uk/dece/, Professor of Education and Public Policy, and a Fellow of the Academy of Social Sciences. He is passionate about exposing and reducing inequality in education, and improving the quality of education research.

Nadia Siddiqui

Dr Nadia Siddiqui is an Associate Professor at Durham University, leading a British Academy funded study of schooling in Pakistan and India.  She is a  member of the Cabinet Office What Works Trial Advice panel.

Beng Huat See

Professor Beng Huat See leads an ESRC-funded study of teacher supply. She is a a Fellow of the Academy of Social Sciences, The Higher Education Academy, and the Wolfson Research Institute.

Appendix

Table A1. Percentage of pupils long-term and temporarily FSM-eligible.

	N	% 5+ years FSM	% 6 years FSM	% Temporary FSM (less than 5 years)	% Temporary FSM (less than 6 years)
2006	574,399	9.6	-	12.3	24.7
2007	552,088	11.8	8.4	12.0	14.8
2008	567,810	11.9	8.3	11.8	18.1
2009	564,734	12.5	8.9	11.9	18.1
2010	571,338	10.9	8.1	13.4	19.4
2011	532,160	11.9	8.8	13.8	19.7
2012	525,131	11.6	8.5	14.0	19.9
2013	502,014	11.9	8.6	14.2	20.3
2014	553,154	11.5	8.2	13.7	19.9
2015	571,444	12.0	8.6	14.4	21.0
2016	583,714	11.7	8.5	14.9	21.3
2017	597,491	11.3	8.4	15.1	21.1
2018	617,669	10.7	8.0	15.2	20.7
2019	643,087	10.3	8.3	15.3	20.0

Table A2. Year 1 segregation of FSM-eligible pupils.

Segregation Year 1	Majority	FSM Year 1	Overall SD	ES
2006	-	-	-	-
2007	−0.043	0.131	0.156	1.115
2008	−0.043	0.130	0.155	1.116
2009	−0.053	0.140	0.173	1.116
2010	−0.053	0.140	0.177	1.090
2011	−0.052	0.140	0.171	1.123
2012	−0.049	0.138	0.167	1.120
2013	−0.047	0.134	0.164	1.104
2014	−0.051	0.116	0.155	1.077
2015	−0.048	0.115	0.161	1.012
2016	−0.051	0.106	0.161	0.975
2017	−0.051	0.101	0.159	0.956
2018	−0.049	0.098	0.157	0.936
2019	−0.046	0.094	0.151	0.927

Table A3. Attainment gap, KS1 points (raw scores).

KS1 Points	Majority	FSM 5+ years	Overall SD	ES
2006	15.78	13.18	3.78	−0.69
2007	15.80	13.08	3.99	−0.68
2008	15.81	13.17	3.96	−0.67
2009	15.75	13.08	3.90	−0.68
2010	15.57	12.80	3.97	−0.70
2011	15.56	12.97	3.88	−0.67
2012	15.53	13.00	3.85	−0.66
2013	15.54	13.16	3.79	−0.63
2014	15.53	13.19	3.78	−0.62
2015	15.61	13.21	3.75	−0.64
2016	15.90	13.65	3.66	−0.61
2017	16.10	13.90	3.60	−0.61
2018	16.27	14.12	3.56	−0.60
2019	16.44	14.37	3.50	−0.59

Table A4. Attainment gap, KS1 points (z-scores).

KS1 Points z-scores	Majority	FSM 5+ Years KS2	Overall SD	ES
2006	0.069	−0.618	1	−0.687
2007	0.083	−0.600	1	−0.683
2008	0.082	−0.585	1	−0.667
2009	0.089	−0.595	1	−0.684
2010	0.077	−0.621	1	−0.698
2011	0.082	−0.585	1	−0.667
2012	0.080	−0.578	1	−0.658
2013	0.077	−0.553	1	−0.630
2014	0.074	−0.545	1	−0.619
2015	0.080	−0.560	1	−0.640
2016	0.075	−0.540	1	−0.615
2017	0.072	−0.539	1	−0.612
2018	0.068	−0.537	1	−0.604
2019	0.064	−0.528	1	−0.592

Table A5. Attainment gap, KS1 points (z-scores), by Economic Region.

	East England	East Midlands	London	North East	North West	South East	South West	West Midlands	Yorkshire and Humber
2006	0.69	0.77	0.60	0.73	0.65	0.81	0.73	0.68	0.68
2007	0.66	0.72	0.59	0.71	0.68	0.80	0.70	0.68	0.68
2008	0.69	0.69	0.57	0.67	0.66	0.75	0.73	0.68	0.67
2009	0.66	0.69	0.57	0.74	0.67	0.77	0.72	0.69	0.72
2010	0.66	0.71	0.56	0.79	0.70	0.75	0.73	0.74	0.77
2011	0.68	0.70	0.51	0.71	0.70	0.78	0.70	0.65	0.68
2012	0.67	0.66	0.51	0.73	0.67	0.76	0.74	0.65	0.66
2013	0.67	0.69	0.47	0.69	0.66	0.75	0.70	0.64	0.64
2014	0.64	0.63	0.46	0.68	0.63	0.73	0.70	0.59	0.60
2015	0.69	0.67	0.48	0.70	0.66	0.76	0.72	0.59	0.64
2016	0.63	0.65	0.46	0.66	0.63	0.74	0.63	0.59	0.62
2017	0.64	0.64	0.46	0.65	0.62	0.73	0.67	0.59	0.61
2018	0.63	0.60	0.49	0.62	0.60	0.72	0.65	0.57	0.60
2019	0.63	0.65	0.46	0.62	0.57	0.70	0.66	0.54	0.59

Table A6. KS2 segregation of pupils FSM-eligible for five or more years.

FSM Segregation	Majority	FSM 5 Years KS2	Overall SD	ES
2006	−0.0183	0.1708	0.15700	1.204
2007	−0.0221	0.1590	0.15603	1.161
2008	−0.0208	0.1541	0.15223	1.149
2009	−0.0224	0.1572	0.15254	1.177
2010	−0.0221	0.1611	0.15266	1.200
2011	−0.0208	0.1531	0.15557	1.118
2012	−0.0198	0.1492	0.15240	1.109
2013	−0.0182	0.1438	0.15190	1.066
2014	−0.0166	0.1258	0.13719	1.038
2015	−0.0163	0.1189	0.13411	1.008
2016	−0.0150	0.1118	0.12798	0.991
2017	−0.0140	0.1068	0.12391	0.975
2018	−0.0128	0.1046	0.12056	0.974
2019	−0.0129	0.1103	0.12898	0.955

Table A7. Attainment gap, KS2 points (raw scores).

KS2 Points	Majority	FSM 5+ years	Overall SD	ES
2006	27.870	24.832	4.369	−0.700
2007	27.990	25.147	4.244	−0.670
2008	27.892	25.262	4.458	−0.590
2009	27.958	25.319	4.397	−0.600
2010	27.667	24.724	4.529	−0.650
2011	27.614	24.898	4.764	−0.570
2012	28.570	25.832	4.965	−0.551
2013	28.537	26.004	4.966	−0.510
2014	28.829	26.241	4.962	−0.522
2015	28.911	26.295	4.882	−0.536
2016	102.189	97.700	7.944	−0.565
2017	103.290	98.406	8.484	−0.576
2018	103.731	98.925	8.344	−0.576
2019	103.860	98.905	8.547	−0.580

Table A8. Attainment gap, KS2 points (z-scores).

KS2 Points z-scores	Majority	FSM 5+ Years KS2	Overall SD	ES
2006	0.067	−0.628	1	−0.695
2007	0.079	−0.591	1	−0.670
2008	0.070	−0.520	1	−0.590
2009	0.075	−0.525	1	−0.600
2010	0.071	−0.579	1	−0.650
2011	0.068	−0.502	1	−0.570
2012	0.064	−0.487	1	−0.551
2013	0.061	−0.449	1	−0.510
2014	0.060	−0.461	1	−0.522
2015	0.064	−0.472	1	−0.536
2016	0.066	−0.499	1	−0.565
2017	0.065	−0.510	1	−0.576
2018	0.062	−0.514	1	−0.576
2019	0.060	−0.520	1	−0.580

Table A9. Rank gap, KS2 Average Points.

	FSM (5+ years) eligible and the rest	FSM (6 years) eligible and the rest
2006	0.674	-
2007	0.646	0.63
2008	0.615	0.604
2009	0.623	0.611
2010	0.621	0.615
2011	0.581	0.581
2012	0.567	0.562
2013	0.529	0.532
2014	0.543	0.552
2015	0.559	0.564
2016	0.552	0.565
2017	0.56	0.57
2018	0.558	0.571
2019	0.562	0.565

Table A10. Attainment gap, KS2 points (z-scores), by Economic Region.

	East England	East Midlands	London	North East	North West	South East	South West	West Midlands	Yorkshire and Humber
2006	0.70	0.72	0.46	0.70	0.61	0.81	0.72	0.61	0.70
2007	0.69	0.70	0.47	0.66	0.63	0.84	0.72	0.63	0.67
2008	0.67	0.62	0.41	0.61	0.57	0.76	0.69	0.56	0.60
2009	0.66	0.61	0.41	0.61	0.58	0.76	0.65	0.57	0.63
2010	0.66	0.69	0.48	0.73	0.61	0.74	0.65	0.61	0.70
2011	0.62	0.62	0.36	0.59	0.58	0.73	0.64	0.51	0.57
2012	0.64	0.60	0.36	0.59	0.53	0.70	0.66	0.52	0.56
2013	0.64	0.63	0.38	0.57	0.55	0.73	0.66	0.52	0.57
2014	0.62	0.60	0.38	0.58	0.54	0.67	0.68	0.49	0.52
2015	0.64	0.59	0.40	0.55	0.53	0.71	0.67	0.51	0.55
2016	0.66	0.60	0.42	0.61	0.55	0.74	0.64	0.55	0.58
2017	0.67	0.62	0.43	0.58	0.56	0.73	0.66	0.57	0.59
2018	0.67	0.59	0.47	0.57	0.56	0.75	0.67	0.54	0.54
2019	0.67	0.64	0.46	0.57	0.54	0.75	0.69	0.54	0.55