Challenges facing interventions to promote equity in the early years: exploring the ‘impact’, legacy and lessons learned from a national evaluation of Children’s Centres in England

ABSTRACT

This paper discusses the challenges facing a national evaluation of an early years intervention programme, Sure Start Children’s Centres (SSCCs), that was implemented across England in the first decade of the 21st century. The paper describes the rationale for the evaluation’s mixed methods research design and the ecological theoretical approach adopted. It investigates the SSCC policy aim of combatting the ‘impact’ of multiple disadvantage on outcomes for families, parents and children. Based on a clustered sample (2,600 families) it provides evidence of statistical effects for different user groups, including non-users. It points to the complexities in evaluation in non-experimental interventions where there was an emphasis on services to meet local needs and where families could choose which services to access and change patterns of service use over time. The paper synthesises findings and considers how complex, volatile and uncertain environments affected SSCC provision, particularly linked to a change of government and austerity policies after 2010. The paper identifies lessons learned, explores implications for future early years interventions in uncertain times, and proposes alternative approaches to evaluation (a realist approach based on mixed methods and theoretically driven models) where randomised experimental designs are inappropriate for the evaluation of certain complex policies.

KEYWORDS:

• Children’s centre
• disadvantage
• effectiveness
• family services
• impact
• parenting

Introduction

The early years are a crucial period in children’s lives of particular relevance to policymakers seeking to reduce inequities in life chances and combat the effects of disadvantage (Institute for Fiscal Studies, 2019). The development of services and interventions that support families to promote better outcomes for children became a key policy goal in many countries in the first decade of the 21st century (OECD, 2001, 2006). Parenting has received particular attention and the role of the early years home learning environment is an example of such concerns (Attig & Weinert, 2020; Institute for Fiscal Studies, 2019; Lehrl et al., 2020) with a number of large-scale, longitudinal studies demonstrating associations between various child and family background characteristics and the quality of the early years home learning environment (e.g. Attig & Weinert, 2020; Hall et al., 2021; Sammons, Toth et al., 2015; Toth et al., 2020). Various small-scale experimental/quasi-experimental interventions have been developed to support parenting activities to foster better child development (e.g. Institute for Fiscal Studies, 2019; Jelley & Sylva, 2017).

One of the largest and most ambitious policy interventions to support families in England was the Sure Start initiative implemented soon after the turn of the millennium. It represented a radical departure from previous 20th century education policy in England that had paid scant attention to early years education and care in contrast to many other EU countries. Sure Start represented a comprehensive, inclusive, geographically focussed attempt to combat disadvantage through supporting families and parenting with a particular focus on disadvantaged communities. The election of a Labour Government in 1997 provided major new investment, first in universal pre-school education but from 1998 plans were made for new forms of services with a broader focus on families with very young children. The original Sure Start programme quickly evolved into a delivery model through Sure Start Children’s Centres (SSCCs) linking multiagency services via neighbourhood centre-based provision. It demonstrated a shift in policy priorities in England to support families and combat disadvantage (Cheater, 2019; DCSF, 2009; Eisenstadt, 2011; Evangelou et al., 2017).

This paper synthesises evidence from a wide ranging six-year national evaluation of SSCCs, the Evaluation of Children’s Centres in England (EvCCE) study that explored both provision of services and their use. EvCCE was guided by an internal policy review by the evaluation commissioners, the Department for Children, Schools and Families (DCSF, 2009). This explicitly noted a National Audit Office review (Melhuish, 2004) on the ‘impact of early years provision’ that illuminated the broad way ‘impact’ was conceived in relation to early childhood experiences and disadvantage. It argued that a substantial amount of evidence suggests that an individual’s early childhood experiences have a lasting effect on their life chances, including health, educational and occupational outcomes and noted that the impact on those from the most disadvantaged backgrounds is particularly acute.

The evaluation of the SSCCs was not commissioned according to the economic perspectives outlined in HM Treasury’s Green Book (2003) of the era. This noted that, where possible, the comparative assessment should include a ‘control group’, to whom the activity was not applied, although it recognized that in complex ongoing policy interventions a counterfactual might not be feasible. In developing the expanded SSCCs programme the DCSF was informed by an earlier area-based evaluation (Melhuish et al., 2008) and also drew particular attention to findings by the ongoing longitudinal Effective Provision of Pre-School and Primary School (EPPE) research that used statistical models to explore the effects of pre-school experiences on later child outcomes (Sylva et al., 2010). EvCCE built on the EPPE approach and developed methods to measure the effects of SSCCs on relevant outcomes for families and children using statistical models (Sammons, Hall et al., 2015; Sammons, Smees et al., 2015).

This paper also draws on a follow-up study of how austerity cuts to public services affected SSCC provision from 2009 to 2017 (Smith et al., 2018). It discusses the legacy of the SSCC initiative in England, and more recent developments and lessons learned from EvCCE and the SSCC policy for supporting families with young children. A major theme in the paper is the methodological complexity of evaluating the ‘impact’ of large and complex policies such as SSCCs. The implications for future Early Years interventions are explored.

Background

Following the original Sure Start initiative of 1998, SSCCs were launched in 2002 with the aim of giving disadvantaged children the ‘best possible start in life’ (DCSF, 2009; Eisenstadt, 2011). Children’s Centres were intended to be community-based facilities delivering a range of services to better support the needs of all families with young children in disadvantaged neighbourhoods. They were part of a much wider policy emphasis on promoting equality in the early years that included reducing the number of children living in poverty and making pre-school available to all children from age 3 (for disadvantaged families from age 2) to combat the multifaceted 'impacts' of disadvantage on children’s educational, health and later life chances. Similar developments occurred in other parts of the UK which have their own devolved educational services, and internationally the OECD initiated studies of early years policies to promote a ‘Strong Start’ (OECD, 2001, 2006).

SSCCs provided integrated multi-agency services at a single local point of access. The emphasis on disadvantaged neighbourhoods and multi-agency work echoed other policies in education evident at the turn of the millennium in schools including the New Community Schools initiative in Scotland (Sammons et al., 2003) that linked nurseries, primary and secondary schools, bringing together health, social services and education in the most disadvantaged areas and the subsequent Extended Schools policy in England. However, England’s SSCCs were unique in their focus on families with very young children (0–3) seeking to support better outcomes for children, parents and communities by recognising the value of very early intervention to improve parenting aspirations and skills, providing access to early education, and addressing family and child health and life chances through multiagency strategies to promote health, parenting and economic wellbeing.

An ecological perspective

The theoretical underpinnings of SSCCs’ provision aligns with an ecological approach (Bronfenbrenner & Morris, 2006; Bronfenbrenner, 1994) which distinguishes nested influences on children’s development from more proximal (e.g. family) to more distal (e.g. neighbourhood) (see Eisenstadt, 2011; Evangelou et al., 2014; Sammons, Hall et al., 2015; Sammons et al., 2017). Figure 1 illustrates this general model showing how intertwined parental and family needs were to be addressed by SSCCs.

Figure 1. Idealised ‘model of parental needs’ which may be targeted by children’s centres (Evangelou et al., 2014).

EAL = English as an additional language

Evaluating SSCC: EvCCE design and data sources

EvCCE was government funded by the DCSF (later renamed the Department for Education, reflecting less emphasis on families) in 2009 and conducted jointly by the University of Oxford and NatCen Social Research. It built on but took a different approach from an initial evaluation (National Evaluation of Sure Start – NESS) of the original Sure Start Policy in operation from 1998 to 2002 (Melhuish et al., 2008). NESS sought to establish the effects of Sure Start at the community level, on services, levels of disadvantage and disorder, the general health and well-being of local residents, and the development of their children. After 2003, the policy focus changed to providing a specific Children’s Centre based offer to all families via a local, designated centre. The EvCCE design thus differed from NESS by focussing explicitly on the role of centre-based provision and directly measuring families’ engagement and use of services, data not collected by NESS which adopted a quasi-experimental area-based design. EvCCE investigated in detail reported patterns of service use by registered families, different management, organisational and delivery approaches and explored their statistical effects on subsequent child, mother and family outcomes. EvCCE studied centres and families located in the 30% most deprived areas in England through a mixed methods nested survey design. Figure 2 illustrates the sampling strategy and numbers of centres, families and children in different linked strands of the evaluation. (See detailed reports on the EvCCE project at https://www.gov.uk/government/collections/evaluation-of-childrens-centres-in-england-ecce.)

Figure 2. The EvCCE sampling frame for different strands of the national evaluation (Sammons, Hall et al., 2015).

Strand 1: Survey of Children’s Centre leaders

This strand produced a profile of SSCCs in terms of management, staff, services, users and finance. Data were collected from a stratified sample of over 500 centres using web-based and telephone techniques.

Centre Leaders (managers) were interviewed in 2011 and again in 2013 on key aspects of service provision so that change could be investigated across a two-year period (Tanner et al., 2012). A group of 128 (25.6%) of the original survey of 500 centres was randomly selected for subsequent strands in the evaluation.

Strand 2: Longitudinal survey of families using Children’s Centres

Surveys (over three waves) involved 5,717 families registered at the 128 centres and were specifically linked to a ‘focal child’ in the toddler age group (9–18 months at baseline). Families were interviewed face-to-face at baseline followed up by telephone surveys at child age 24 months and face-to-face at child age 36–42 months. A random sample of approximately 50% of the original 5,717 families were then followed in Wave 3 (2,602 families interviewed) to:

• explore the level of take-up of various children and family services among families with different socio-economic characteristics;

• monitor changes in patterns of service use by families over time; and

• collect data on different aspects of child development and family functioning to enable an analysis of impact on child and family outcomes associated with family engagement with SSCCs and use of different types services.

Information included families’ key socio-demographics, aspects of family functioning, parent physical and mental health, child health, as well as the focal child’s socio-emotional and cognitive development (Maisey et al., 2013, 2015). Interviewers also collected data on families’ reported use of services at their registered SSCC, plus details of other service use including childcare and early education. The longitudinal design provided a baseline assessment of families (Wave 1) alongside early child, parent and family outcome measures (Wave 3) from which to assess ‘impact’ via statistical models of effects (Strand 4).

Strand 3: Investigation of Children’s Centre service delivery and reach

Researchers visited 121 of the 128 Children’s Centres sampled for Strand 2 (a few centres had closed/were unavailable for visits) in 2012 and again in 2013, to assess the range of centre activities and services offered, the nature of multi-agency partnership working, leadership and management, the extent of evidence-based practice (Goff et al., 2013; Sylva et al., 2015) and parenting support provided (Evangelou et al., 2014). Hall et al. (2015) provide a review of family services offered during the evaluation period. Services commonly offered include: health advice, childcare and early education, employment advice, informal drop-in facilities, and specialist support on parenting.

Interviews and surveys with centre staff and parents explored their experiences and perceptions and provide detailed quantitative and qualitative data on service provision and patterns of organisational delivery. Analyses of administrative data enabled a user profile exercise in 117 SSCC areas to assess their geographic and social ‘reach’ (Smith et al., 2014); that is, the extent to which centres served local communities, especially the most disadvantaged families.

Strand 4: Investigating the ‘impact’ of Children’s Centres via statistical models of child, mother and family outcomes

EvCCE faced various challenges in evaluating the ‘impact’ of SSCCs. It recognised that Sure Start had evolved from an area-based initiative seeking to improve services and practice, to prioritise a physical centre offering multiagency services open to all families in the most disadvantaged neighbourhoods. SSCCs were intended to both identify and be responsive to local needs. Moreover, families could choose to use different services in different ways and while many were universal, some services were explicitly targeted for ‘vulnerable’ families via various forms of outreach. Thus, SSCCs were complex and multifaceted and did not represent a single ‘intervention’ based on a common model and so traditional experimental designs to evaluate potential ‘impact’ via comparison of intervention and control groups were inappropriate. Given that SSCCs had been rolled out in all the most disadvantaged areas to maximise the intended benefits to all families in such communities, an alternative theoretically driven evaluation perspective was adopted that aligns with a ‘realist approach’ using ‘a broad repertoire of quantitative and qualitative methods to explore potential context-mechanism-outcome configurations’ (Clarke, 2006, p. 573). In a realist approach ‘[s]pecific hypotheses are derived from the theory and these dictate the appropriate research strategy and tactics such as the choice of where detailed measurements of expected impact need to be undertaken’ (Pawson & Tilley, 1998, pp. 89–90). Such an approach fitted the research aims and links with the ‘ecological’ model that underpinned the family focussed, multi-agency, locally flexible enactment of the SSCCs’ policy and accords with the conceptualisation of ‘impact’ (see DCSF, 2009) that underpinned the policy. Rather than addressing questions of ‘what works’ as in experimental designs (Farrington, 2003), EvCCE developed theoretically driven models to study statistical effects for different groups of SSCCs’ service users based on naturally occurring variation in patterns of use (including statistically derived counterfactuals of ‘non-users’ or ‘minimal user groups’). This design enabled EvCCE’s ‘impact’ team to test research hypotheses about predicted effects on relevant outcomes based on such theories and aligned with the aims of the SSCC policy to produce statistical findings and develop plausible theoretically informed explanations concerning potential context-process-outcome relationships, to inform future policy.

Statistical analyses

Models were based on data for 2,608 families followed up to Wave 3 (child age 36–42 months, mean age 38 months) representing approximately 50% of the original families surveyed in Wave 1. They were registered at 117 SSCCs and had been tracked for around two years of their child’s life. The sample was drawn from registered families (those officially recorded as ‘on the books’ of each centre). Those registered were not necessarily always real-life users of specific services. The family surveys collected details of what services (if any) registered families used enabling study of attendance patterns and intensity and duration of use of different services to be mapped. This means that the evaluation was able to include a number of ‘no user’ groups for particular types of services that enabled comparisons with light, moderate and heavy service users (reflecting different patterns of family SSCC engagement). This was an important and unique feature of the EvCCE study to measure actual service use and different patterns of family engagement (something a previous area-based evaluation had not attempted; Melhuish et al., 2008).

The statistical analyses also explored a range of quantitative data about SSCCs and their characteristics (data collected via surveys conducted in Strands 1 and 3). Data on centre characteristics and provision in the period 2011–13 were combined with family and child outcome data that had been obtained from family interviews in Strand 2 surveys (Sammons, Hall et al., 2015; Sammons, Smees et al., 2015). Cluster analysis was a key technique employed to identify and summarise underlying patterns in families’ reported use of services across the three time points of the family surveys (Waves 1, 2 and 3) and revealed distinct groups. In addition, cluster analysis was also used to examine patterns in services offered across the 121 SSCCs at which families were registered. Again, this technique identified distinct groups of centres showing similarities in their characteristics and services. The naming of these clusters was a form of qualitising the quantitative data, an important feature in the mixed methods evaluation strategy. Cluster analyses proved a valuable tool in summarising large amounts of quantitative data representing ‘real life’ variation in family use of services and in SSCC patterns of local service delivery before further statistical models were developed.

Multilevel statistical regression models explored associations between families’ use of SSCCs and the 13 child, mother and family outcomes measured. It was hypothesised that greater engagement with services would predict better outcomes, although it was also recognised that this assumption might not hold in relation to some specific targeted services aimed at reaching the most vulnerable groups (a point we discuss further in a later section).

The ‘impact’ strand thus addressed two questions:

1. Does family engagement with SSCCs predict better child, mother and family outcomes?

2. Which aspects of SSCCs (management structure, working practices, services offered, and services used) predict better family, parent, and child outcomes?

Outcome measures reflected the wide aims of SSCCs for children and families to promote school readiness, better health and life chances. Six child outcomes were chosen: internalising behaviours, externalising behaviours, pro-social skills, language (naming vocabulary), non-verbal reasoning (picture similarities), and health (a dichotomous variable indicating whether or not a child was in poor health at time of interview). Two mother outcomes were studied: one focusing specifically on mental health, and the other on a more general measure of the mother’s health status (a dichotomy measure of ‘better’ or ‘poorer’). For family functioning, five outcomes were identified: Household Economic Status (HES) (no parent in the household working versus one or more parent working). The Confusion, Hubbub, And Order within the home Scale (CHAOS) provided an indicator of structure of the home environment, while the early years home learning environment scale measured specific features of the home learning environment at child age 3+ years. Two measures of parenting were collected: Parental Distress and Parent-Child Dysfunctional Interaction (Hall et al., 2021; Sammons, Hall et al., 2015).

Complex statistical models are needed to study real-life contexts and variations in outcomes and institutional effects and are of particular relevant to studies, such as EvCCE, ‘ … in order to describe the complex reality that constitutes educational systems we require modelling tools that involve a comparable level of complexity’ (Goldstein, 1998, p. 2). Multilevel approaches are well suited to address the inherent complexity of interventions, such as Sure Start that had evolved to a physical centre-led delivery because they take account of clustering in the data (families nested in the SSCCs at which they were registered). This reflects the ‘real life’ nature of SSCCs’ programme as enacted in different neighbourhoods where participants were not randomly allocated as in traditional RCTs and where quasi-experimental attempts at propensity score matching of centres were inappropriate due to wide local variations in both centre provision and family patterns of usage. Rather the multilevel analyses adopted nested models and controls in line with the underlying ecological theoretical model that informed the evaluation. Models tested how far measures of families’ engagement with their registered SSCC and their use (or lack of use) of services showed effects (statistical associations) in predicting variation in the 13 child, mother and family outcomes, controlling for the net effects of other relevant background characteristics that also predicted such outcomes. The approach reflects those commonly employed in the educational effectiveness research tradition to identify variations in institutional effects and processes via multilevel analyses (Creemers et al., 2010) and is now becoming increasingly used in early childhood research (Attig & Weinert, 2020; Lindorff et al., 2020; Sammons et al., 2013).

The strongest predictors of child, family and mother outcomes were all related to features of family background, including parental qualifications, family socio-economic status (occupational SES) and income. This is in accord with past educational effectiveness research that has consistently demonstrated the adverse influence of social disadvantage in shaping inequalities in education and health (Marmot, 2010; Melhuish et al., 2008; Sammons et al., 2004, 2017; Taggart et al., 2006) and recent international comparisons show that the effects of disadvantage are particularly strong in England (Jerrim et al., 2018).

In addition to models of 13 outcomes that tested different predictors individually and in combination, an overall summary measure of family disadvantage was created and this also showed strong associations with all 13 outcomes.

Thus, the EvCCE ‘impact’ evaluation findings on the continued strong associations between background and outcomes confirmed the powerful effects of background in predicting outcomes that were the intended focus of SSCC policy according to the DCSF (2009) review noted earlier. This review had recognised and framed such effects of disadvantage using the term ‘impact’. It also provided new findings based on theoretical models of the hypothesised ‘impact’ of SSCCs in terms of statistical effects on a wide range of outcomes, while controlling for the effects of relevant background characteristics reflecting the concept of ‘disadvantage’. These results clearly demonstrate the extent of inequality across a wider range of outcomes for toddlers and their families, and identify the important variables that are associated with that inequality from a very young age.

Figure 3 illustrates the modelling strategy adopted in the ‘impact’ analyses.

Figure 3. Identifying children’s centre ‘impacts’ on child, parent & family outcomes with a clustered sample (2600+ children and families from 117 centres) based on multilevel statistical models (Sammons, Hall et al., 2015).

The EvCCE study was able to reduce (but not eliminate) the potential problem of selection bias by comparing data on no and minimal users of SSCC services with families making greater use of services through its longitudinal survey design. It also replicated the multilevel models for the subset of most highly disadvantaged families to check whether the positive effects of family engagement found for the full sample remained evident for the most disadvantaged group (Sammons, Hall et al., 2015). The results again pointed to SSCCs showing positive effects for the most disadvantaged families in the hypothesised direction and in line with the policy intent to promote better outcomes for the most disadvantaged as well as supporting all families in the most disadvantaged neighbourhoods, thus helping to address inequality.

A further important feature of EvCCE that addressed possible selection bias was provided by a separate study of ‘reach’ based on analyses of administrative data collected from Local Authorities (LAs) and of the national Census (2011) to study registrations and family use at centre level (Smith et al., 2014). SSCCs were intended to serve areas, families and children with high social needs. Drawing on LA specifications of their ‘reach areas’, EvCCE mapped reach areas for 117 of 128 centres in the SSCC sample. Analysis of socio-economic indicators of poverty, low income, unemployment, education, health, housing, crime and transport revealed that SSCCs’ reach areas were on average more deprived than both the national average and the LAs in which they were located. Centres typically had very large registration and user numbers. In almost all SSCCs the proportion of registrations in a single year, judged against the average 2011 census population aged 0–4 in a year was over 90% (median 93%) indicating very high registrations of families with young children in reach areas (Smith et al., 2014).

The analyses of administrative data for the ‘reach’ study of SSCCs complemented the main EvCCE longitudinal survey design and provided some reassurance regarding potential selection bias (since the Strand 2 surveys were drawn from all registered families at the sample of centres including non-users).

It must be emphasised that EvCCE does not make any causal claims about ‘impact’ as might be attributed in experimental studies. In line with much other educational effectiveness research and epidemiological studies in general it is only possible to explore patterns of association and identify key variables that are significant predictors of variations in outcomes across time. Strengths of the EvCCE design are that it was based on an appropriate ecological theoretical model, built on well-established methodological modelling approaches from the educational effectiveness research tradition and previous studies of early childhood, and tested plausible hypotheses about the predicted effects of families’ use and engagement with SSCCs using a large sample of families and including groups of non-users of services.

Multilevel models showed that reported use of SSCCs services and certain features of children’s centre organisation were significant predictors of family, mother and child outcomes. In general, identified effects (ES) were relatively small but positive (most ES below 0.30) and in line with the hypotheses. A greater number of statistically significant positive effects were detected for mother and family outcomes (improved mother’s mental health, less chaotic family life, reduced Parent-Child Dysfunctional Interaction, improved Early Home Learning Environment) but fewer effects on child outcomes (e.g. cognitive abilities at age 3+). This might have been anticipated as most SSCCs did not directly offer formal childcare places. This limited the possibility of finding direct effects on child development, since SSCCs were usually not working directly with children who commonly attended childcare elsewhere. However, parents’ engagement with SSCCs predicted better scores for the early home learning environment measure at age 3+, controlling for a baseline measure of earlier toddler home learning environment (Hall et al., 2021; Sammons, Hall et al., 2015). Sure Start policy sought to promote parenting skills and aspirations, the results showing a greater number of positive effects on mother and family outcomes might reflect the emphasis placed on parenting goals in policy documents and also reported by SSCC staff in Strand 3 surveys (Evangelou et al., 2014).

Figure 4 summarises the statistical effects identified for each stakeholder group (child, mother and family) at child age 3+ years. EvCCE distinguished information about family use of a broad range of SSCC services from their use of formal childcare since most of the children in the sample did not attend childcare in their local SSCC. Centres typically signposted families to childcare elsewhere as most did not offer it themselves. Attending formal childcare showed positive effects on child outcomes in our models so it was important to control for this before exploring potential SSCCs’ effects. EvCCE then tested the effects of characteristics and processes of individual SSCCs in models predicting these outcomes.

Figure 4. Overview of positive impacts of children’s centres on outcomes (Sammons, Hall et al., 2015).

CHAOS = Confusion, Hubbub, And Order within the home Scale; HLE = Home Learning Environment scale; DI = Parent Child Disfunctional Interaction scale

In considering the findings of the multilevel analyses, it is important to recognise that ‘small’ effects are typical in much educational research. ‘Targeted interventions and small-scale efficacy trials generally produce larger effect sizes than universal interventions because they target study participants that are most likely to benefit and because there is less variation in outcomes among smaller, non-representative samples‘ (Kraft, 2019, p. 13). Typically educational interventions, particularly those involving larger numbers and longer time scales, produce what are historically deemed small effects (Kraft, 2019). In the light of such recent discussions of the interpretation and meaning of effect sizes, those identified by the EvCCE study are deemed of policy relevance, since they accord with the underpinning theoretical model, are based on large, longitudinal samples (over 2000 families) followed up across two years, and are linked to an ambitious policy initiative funded and enacted at a large-scale across many disadvantaged neighbourhoods. Small effects identified for large numbers of families are likely to have greater potential for ‘impact’ in promoting the policy goal of reducing inequity than larger effects for very small numbers targeted by specific tightly focussed interventions.

Additional secondary analyses of EvCCE data using multilevel structural equation modelling investigated whether use of SSCC services may predict lower scores for subsequent behavioural disorders in young children (Hall et al., 2019) via intermediate improvements by age 3 in the early years home learning environment. These further theoretically driven SEM analyses also pointed to positive direct effects of families’ service use for improvements in the early years home learning environment, leading indirectly to better child behavioural outcomes at age 3+ years. This new finding supports and extends the conclusion that engagement with SSCCs services supported parenting as measured by statistical effects on the early home learning environment measure in the direction hypothesised, and via this predicted better child behavioural outcomes indirectly over the two-year period studied. It accords with other recent longitudinal international studies that explore the ‘impact’ of the early home learning environment on child outcomes via theory driven (Bronfenbrenner & Morris, 2006) statistical models based on educational effectiveness research approaches. ‘According to the educational framework of the home learning environment … . Structural characteristics such as parental education, occupation, and household income … . affect educational processes (e.g. quality of interaction behaviour, joint activities). These educational processes in turn impact child development’ (Attig & Weinert, 2020, p. 2).

The main EvCCE findings indicated that there were two outcome measures (of 13 studied) where no statistically significant impact was detected: Home Economic Status (workless household or not) and child’s health (poor health or not). The absence of detectable statistical effects may reflect the limits of such crude dichotomous measures, as noted by Kraft’s (2019) review of intervention effects that points to the need for relevant, reliable and discriminating outcome and control measures. It may also suggest that SSCCs were not effective at promoting better health outcomes for families and children, compared with effects noted for other outcomes. It is relevant that interviews with centre staff and managers reported barriers to SSCCs working with health services due to different funding and delivery streams inhibiting the intended integrated working. The mixed methods evaluation design supported EvCCE’s ability to identify such barriers to multiagency work and illuminated the statistical findings. We suggest this has important implications for promoting ‘impact’ of policies that rely on multiagency working if budgets and delivery streams are not aligned.

Some analyses showed negative effects for certain outcomes associated with engagement for a small number of very specific children’s centres services (notably greater use of outreach and health visitors services). While not straightforward at first sight, our interpretation of these findings suggests this is positive evidence of SSCCs having another form of ‘impact’ that we conceptualised and defined as impact as reach. It should be noted that policy guidance for SSCCs encouraged them to identify, target and persevere in engaging with such ‘needy’ families. Interviews with SSCC staff supported our interpretations of the meaning of these quantitative findings as many staff emphasised the priority accorded to such vulnerable families. The use of a mixed methods design combining quantitative measures and qualitative interview data thus provided more robust evidence in interpreting these apparently counterintuitive findings on the negative effects of outreach and health visitor contact, and enabled us to draw more plausible explanations to inform policy. Again this fits with the realist approach to the evaluation and shows how mixed methods designs can support better understanding and evidence of policy and practice relevance.

Separate cost-effectiveness analyses were conducted after the main EvCCE impact analyses (Gaheer & Paull, 2016). These employed economic models to identify potential ‘value for money’ (VfM). It is beyond the scope of this paper to discuss these in detail. Gaheer and Paull (2016) report the findings but importantly they note that the original intention to estimate the overall VfM of centres proved infeasible and instead VfM was assessed only for a small number of specific services delivered within centres. The VfM models were limited to simple dichotomies of use of an individual service versus no use. Thus, they could not address the complexity in overall patterns in families’ use of services over time that were revealed via the cluster analyses and then tested in the multilevel models in the main EvCCE ‘impact’ analyses. The overall policy and its enactment via SSCCs thus proved too complex to study and evaluate using traditional economic models based on comparisons of control and intervention groups. Costs were not calculable for the real-life complex and varied patterns of families’ SSCCs service use across a two-year period and so overall VfM could not be investigated.

Lessons learned and implications for early years interventions

Investigating the ‘impact’ of a major new early years policy such as SSCCs in England proved an inherently complex task because such centres had a variety of objectives, and also varied in both their function and organisational set up. The policy strongly encouraged SSCCs to be flexible in developing the most appropriate ways of providing services using consultations and needs assessments tailored to their own local areas and communities. This focus on provision to meet neighbourhood needs was an inherent feature of the Sure Start policy but means SSCCs cannot be viewed as a single ‘intervention’. Individual centres differed markedly in the type and mix of services offered and many were also differently affected by budget cuts and restructuring of services from 2011 due to the ways different LAs sought to deal with significant budget losses (typically 40–60% in the most disadvantaged areas). The EvCCE research reveals that establishing ‘impact’ for complex policy initiatives such as SSCCs is therefore not a matter of identifying a single effect (using a control and intervention group) but rather requires alternative approaches capable of identifying and summarising a range of statistical effects, across a large sample of registered families who showed different patterns of service use (including no or minimal use), paying particular attention to developing robust quantitative measures of both user engagement and different models of SSCCs’ service delivery and organisation across a large sample of centres.

EvCCE shows that ‘impact’ is a complex, multifaceted concept that is difficult to investigate in ‘messy’ real-life contexts compared with tightly focussed single interventions typically studied via experimental designs. The longitudinal, mixed methods evaluation design provided additional qualitative insights and rich evidence on parents’ and staff perspectives and important opportunities to identify and study changes in service provision and organisational models of SSCCs to complement and extend the quantitative findings and this accords with the ‘realist’ approach to evaluation that promotes a theory driven perspective to investigate potential context-process-outcome relationships. EvCCE studied SSCCs as organisations and user families via a longitudinal survey design. This differed from the earlier area-based NESS study that made comparisons of child and family outcomes in Sure Start and non-Sure Start neighbourhoods. The roll out of SSCCs to all 30% most disadvantaged areas prevented any such quasi-experimental comparisons after 2009. EvCCE’s detailed longitudinal surveys of registered families and SSCCs are necessarily more expensive than approaches that rely only on outcome data from individuals living/not living within a neighbourhood close to a SSCC. However, the collection of data on actual centre use by families enabled quantitative linking of different patterns of centre use and subsequent outcomes producing new knowledge and testing ecological theories of potential effects. Moreover, EvCCE also used administrative data sets where appropriate and this was particularly important for the ‘reach’ study (Smith et al., 2014) that enabled firm conclusions to be drawn about how far SSCCs succeeded in reaching local families in more disadvantaged neighbourhoods.

We believe that there is much to learn from the SSCCs’ experience in England for policymakers and evaluators not just in England but also in other contexts. Many early years policy interventions do not follow an experimental design and their evaluation therefore poses different challenges and requires different approaches to knowledge creation. The EvCCE study points to the value of longitudinal, mixed methods designs to study intended policy ‘impacts’ appropriately via analyses of statistical effects on relevant outcomes and is in accord with well-established practice in educational effectiveness research (Hall et al., 2021; Teddlie & Sammons, 2010). We argue that such educational effectiveness research informed and theory-driven approaches to evaluation differ from, and can complement, evidence obtained from more tightly focused RCT or quasi-experimental studies of specific interventions that are often targeted to highly selected groups.

In addition to the complexity of delivering SSCCs in different neighbourhood contexts, another difficulty experienced by the five-year EvCCE study was a nearly 50% reduction in its scope after a change of Government in 2010. This much reduced the scale of the evaluation (only half the original sample of 5000+ families in the first Wave 1 survey were followed up face-to-face in Wave 3). A sub-study planned of older age groups (families with children age 3 to 5 years) was cancelled by the DfE due to austerity cuts in government budgets and the planned follow-up of the main EvCCE child and family sample into primary school (as originally envisaged) was not funded. Thus, EvCCE could not investigate any sustained mid- or long-term effects of SSCCs on children’s or families’ later educational or health outcomes. Yet research through other agencies has continued and provides fresh insight to further understand the fallout of the SSCC experience and its legacy.

Recent research on SSCCs

A recent study of health outcomes based on national data sets in England conducted by the Institute of Fiscal Studies (IFS) (Cattan et al., 2021) investigated the short- and medium-term health impacts on children under five living in a neighbourhood where an SSCC was located providing universal access to early learning, health services, parenting support and parental job assistance. The area-based study of hospitalisation rates when the children were in primary school identified health benefits, especially for boys in the poorest areas. The authors suggest operating mechanisms could include: stronger immune systems, safer parenting practices and home environments, and improved emotional and behavioural development among children (Cattan et al., 2021). These more recent findings are of interest given the EvCCE results reported here that identified no significant effects on child health in the short term. They show why it is important to include longer term follow-ups to evaluate early years policy interventions, and to be aware that some positive effects may take years to emerge. They also demonstrate the way national administrative data sets can be interrogated to provide valuable measures to investigate effects over extended time periods at relatively low cost.

Smith et al. (2018) carried out the Stop Start investigation into subsequent patterns of survival, decline or closure of SSCCs in England, funded by the Sutton Trust and intended to document the role of ‘austerity’ cuts on provision after 2010 and Sure Start legacy. Counting only SSCC ‘registered centres’, the drop in SSCC numbers was substantial at over 30% (2009–2018), with the closure of over 1,000 centres. Number of centres closing is a crucial indicator, but also important are the range and number of services offered by centres that remained open (Figure 5). The term ‘hollowing out’ was coined to reflect the widespread reductions in services and opening hours. Local Authorities (LAs) narrowed their focus on more limited services targeted at families deemed most ‘at risk’ of poor outcomes. Financial pressure was cited (by 84%) as the principal driver of decline in provision and move away from open access services since 2011.

Figure 5. Principal reasons for major changes in children’s centre provision (Smith et al., 2018).

Conclusions

The EvCCE research findings taken together with the earlier NESS study (Melhuish et al., 2008) as well as research by Cattan et al. (2021), all point to modest but positive statistical effects of SSCCs in predicting outcomes especially for the most disadvantaged families. However, while being a government ‘flagship policy’ was an enormous advantage initially, its political origins and costs made SSCCs much less attractive to a new government with an austerity agenda and low commitment to LA provision. England has witnessed a 600,000 increase in the numbers of children living in poverty from 2011/12 to 2018 and major cuts to many public services (Social Mobility Commission, 2020), with further worrying increases in childhood poverty following the COVID-19 pandemic affecting families and now compounded by a cost-of-living crisis especially in the most disadvantaged regions. Some SSCCs struggle on with reduced funding and closure of open access activities in a post-pandemic era when their services might be of greatest benefit.

The experience of the SSCCs programme reveals that sustaining early years interventions at scale is very difficult when there are major political differences in ideological and policy priorities. Large cuts to the DfE research budget prevented longer term follow-up to evaluate the policy as children moved into school. Research Council and other non-government sources of funding have advantages in being less susceptible to cuts if policy priorities alter with a change of government.

The problem of service cuts affecting disadvantaged families in England is increasingly recognised: ‘The biggest victim of the cuts – in both relative and absolute terms – has been Sure Start children’s centres. Local authorities’ spending on these centres fell from £1.5bn in 2009/10 to less than £0.7bn in 2017/18, a real-terms fall of 62%’ (Institute for Government, 2019, para. 7). The government is now funding ‘Family Hubs’ to build on the legacy of SSCCs amidst claims that these will be ‘Sure Start Plus’. Unfortunately, the budget allocation (£14 million originally and £20 million announced in 2021) is very much lower than the annual budget of Sure Start at its peak (£1.8 billion, 2010) with far fewer numbers of Family Hubs, and services much reduced in scope and highly targeted to the most vulnerable rather than open access.

Other new developments include the Innovation programme on Children’s Social Work and What Works Centres intended to encourage innovation and sector led improvement (Lewing et al., 2020). Noting evidence for the benefits of parenting interventions, the Institute for Fiscal Studies (2019) reported a feasibility study for a targeted home visiting programme aimed at improving parenting skills based on an RCT design. While such tightly focussed and targeted interventions may show stronger effects (at least in the short term) than broader social programmes, there are many doubts about scalability and roll-out as Kraft (2019) argues.

There remains a tension in approaches to family support between tightly targeted interventions for relatively small numbers (those deemed most ‘needy’ and representing a potentially stigmatising, deficit model of parenting) and the broader well-funded SSCC model aimed at supporting much larger numbers of families in the 30% most disadvantaged areas to combat disadvantage and enhance equity in the early years.

This synthesis and discussion of the English SSCCs model and the challenges in its evaluation can inform other countries as they develop and evaluate joined-up health, social and education policies. The EvCCE research illustrates complexities in defining and studying the notion of ‘impact’ when interventions are broadly focussed and locally flexible and so not suited to experimental evaluation approaches. We argue that longitudinal, theory driven mixed methods designs such as that reported here can produce rich findings and support more nuanced understandings through the combination of quantitative and qualitative evidence that facilitates plausible explanations of policy and practice relevance; this accords with recent arguments promoting mixed methods theory driven evaluation (Onwuegbuzie & Hitchcock, 2017). Links with administrative data can also provide valuable additional evidence (in EvCCE this enabled investigation of the important topic of ‘reach’ [Smith et al., 2014]), and provide pointers to longer term effects (as studies by Cattan et al., 2021; Melhuish et al., 2008 illustrate). EvCCE suggests that perspectives from educational effectiveness research (Creemers et al., 2010; Hall et al., 2021), including multilevel statistical models based on large, nested samples and informed by an ecological theoretical perspective, can support rigorous evaluation of complex and often ‘messy’ real world early years policy initiatives.

We suggest five key points when planning interventions and evaluations to support families in other contexts:

• Offer a model that avoids stigma because services aimed at ‘high risk’ or vulnerable groups may reduce uptake. We found evidence that universal services were valued and predicted positive effects for all including the most disadvantaged families.

• Seek to ensure agreement between different political parties to maintain support and reduce the risks that an intervention becomes a casualty of change of governments. This may be very difficult because of the ideological positions of political actors, but it is important to avoid ‘start stop’ policies that inevitably waste resources including professional expertise and commitment.

• Ring fence budgets to reduce the likelihood that early intervention monies become used for other purposes such as work with older age groups which spreads funding more thinly.

• Ensure independent and rigorous evaluations are commissioned that include the long-term follow-up of relevant child, parent and family outcomes. Research Council funding to support high-quality studies of different early years and family support programmes has much to commend it by safeguarding academic independence in publishing findings and reducing the likelihood that a change of government would cut short funding for evaluation.

• Recognise that appropriate research designs and methods need to be developed that take account of ‘messy’ real-world enactments in service provision and different patterns of family engagement and choice in locally responsive, multiagency interventions that do not follow a standard model. Longitudinal, mixed methods designs may be particularly appropriate to explore the multi-faceted and complex concept of ‘impact’ in the volatile and changing real-life contexts of the emerging post-pandemic world where early years interventions, multiagency approaches and family support are likely to be of special relevance to young children and families in the most disadvantaged communities.

The changing face of early childhood requires new research approaches (Oppenheim & Rehill, 2020). There are important choices about the best foci for early years services and policies, particularly in the light of changes in family structure and functioning, economic challenges and the impact of the pandemic exacerbating already marked patterns of inequity in life chances. What is the role of the state versus other voluntary or private sector agencies? How far are universal approaches necessary? Are tightly targeted policies aimed at the most vulnerable more or less appropriate during lean economic times? Future researchers will need innovative strategies to establish the reach, costs and effects of culturally and contextually sensitive services and their delivery that are family focussed and aimed at improving the life chances of young children. The tightly controlled methods of the last century are not up to this task – and yet we should not abandon rigour. It is increasingly recognised that there are different types of evaluation that should guide moves towards better evidence informed policy and practice, thus privileging any one type of research evidence may be misguided (Davies et al., 2006). Well constructed theoretically driven mixed methods approaches may help us move beyond a narrow focus on ‘what works’ towards better understanding and explanations to inform future early years policy and practice seeking to promote equity.

Geolocation information

This paper focuses on research carried out in England, one of the four nations in the United Kingdom.

Acknowledgements

The Department for Education in England funded the original EvCCE evaluation led by NATCEN and the University of Oxford; the Sutton Trust funded a follow-up study of Children’s Centres closures. We much appreciate the contribution of all staff and families who were generous with their time during the visits and interviews. This research would not have been possible without the co-operation of children’s centre staff and the many families who contributed to the various EvCCE surveys.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

Data sharing is not applicable to this article as no new data were created or analysed in this paper.

Additional information

Funding

This work was supported by a grant [R13706/CN005] from Department for Education and the Sutton Trust.

Notes on contributors

Pamela Sammons

Pamela Sammons is Professor (emeritus) at the Department of Education, University of Oxford and an Emeritus Fellow at Jesus College, Oxford. Her research over four decades focuses on educational effectiveness and improvement, school leadership, teaching quality, and promoting equity. She has conducted major studies in England, Scotland and Northern Ireland and led the impact analyses for the Evaluation of Children’s Centre’s in England (EvCCE). She has a particular interest in longitudinal and mixed methods research approaches.

Kathy Sylva

Kathy Sylva is Professor of Educational Psychology at Oxford’s Department of Education. She conducts large-scale studies on the effects of early education/care on children’s development and has led RCTs evaluating parent interventions. She has been specialist adviser to Parliamentary Select Committees and currently advises Ofsted, Early Intervention Foundation, and Education Endowment Foundation. She is a Fellow of the British Academy, Academy of Social Sciences, and received British Educational Research Association’s Nisbett Award. She was awarded an OBE for services to children and families in 2008.

James Hall

James Hall is Associate Professor of Education at the University of Southampton. His research programme concentrates on Early Childhood Education and Care (ECEC) by uniting Developmental Psychology with Educational Effectiveness Research. The result is new knowledge regarding: the drivers of academic progress and socioemotional development; educational equities and inequities; and the mechanisms of social disadvantage. His latest book, International Perspectives in Educational Effectiveness Research (edited with Pamela Sammons and Ariel Lindorff), is published by Springer Press.

Maria Evangelou

Maria Evangelou is Professor of Early Childhood Education at the International Hellenic University and Honorary Research Fellow at the Department of Education, Oxford. Her research focuses on the evaluation of early childhood interventions; the development of children at risk of educational underachievement; the language and literacy development in the early years; enhancing parental engagement in children’s learning; parenting education and support; the effects of home learning environment and quality learning environments; early years professional development; and evidence-based practices in education.

Rebecca Smees

Rebecca Smees is a researcher in the Department of Psychology, University of Sussex, investigating multisensory learning in children. She has extensive experience, including senior statistician roles on a number of high profile national evaluations including the national Evaluation of Children’s Centres in England. She has worked at a number of UK institutions including the University of Oxford, University of Brighton and the Institute of Education, London. Her research interests include early years provision, multisensory learning, creativity, play and mental well-being in childhood.