Using Early Childhood Data and Measurement to Leverage Change: Are We Making Progress?

1

ABSTRACT

As countries make continued investments in early childhood care and education (ECCE) systems, data and measurement play a critical role in outlining strengths and areas for improvement. To promote data-driven decision-making across all levels of early childhood systems, measurement must be valid, reliable, and feasible, especially when used at scale. But beyond psychometrics, for maximum impact, the pathway by which data and measurement can lead to change must be articulated, including integration of ECCE into the country's education management systems and increased insight into building demand for ECCE data. This article focuses on the role that measurement can play in ECCE, specifically addressing data on child development outcomes and quality of children’s learning environments. I will outline key issues facing ECCE data and measurement, discuss how data and measurement can facilitate change, and provide next steps toward building country systems and global support for effective use of ECCE data. I will conclude with examples of possible paths forward for ECCE leaders, researchers, and practitioners at the country and global levels.

KEYWORDS:

• Child development
• early care and education
• measurement
• preschool quality

Access to quality early childhood care and education can contribute to later education success. Data on children in low- and middle-income countries (LMIC) demonstrate that in many countries, children who attend preprimary education have stronger skills in early primary grades (Aguilar & Tansini, 2012; Gove et al., 2018; Kim, 2022; Nakajima et al., 2019). At the same time, notable inequities characterize access to quality ECCE (Ashley-Cooper et al., 2019; Baum et al., 2019; Kim et al., 2022; Krafft et al., 2023), necessitating ongoing attention to access to quality early childhood care and education in LMIC (Spier et al., 2019).

Data and measurement play essential roles in building and maintaining effective early childhood systems. Reliable data raise awareness of the importance of early childhood and empower policymakers, stakeholders, early childhood professionals, and parents to make informed decisions on where and how to invest in early childhood care and education (ECCE; Dusabe et al., 2019; Yoshikawa et al., 2018). This is especially the case as ECCE programs are scaled. Research evidence demonstrating the notable returns on investment in early childhood has led to growth in access to early childhood programs all over the world (UNESCO Institute for Statistics, 2022; UNICEF, 2022), but the positive returns are greater when quality is high (e.g., Britto et al., 2011). As countries invest in early childhood, a new set of challenges emerge on how to ensure quality at scale and how to build early childhood systems that promote equity through access to quality ECCE for the children most in need of services (Yoshikawa et al., 2018).

The passage of the Sustainable Development Goals Target 4.2, which addresses the role of early childhood care and education for school readiness, opened a new window for monitoring early childhood development within and across countries (Raikes et al., 2017). The introduction of global monitoring of early child development before the start of school created an opportunity to encourage countries to collect and report equity in children’s language, physical, and cognitive development as part of country efforts to make progress toward the SDGs by 2030. Since the start of the SDGs, 79 out of 193 member states reported on the percent of children developmentally on track using the Early Child Development Index (ECDI) developed and implemented by UNICEF (UNESCO, 2022), with data disaggregated by factors associated with inequity such as family income, geographic location, and gender for many of these countries. Research using the Early Childhood Development Index has identified notable inequities in child development between and within low- and middle-income countries (e.g., McCoy et al., 2016), as well as associations between child development outcomes in the preschool years and country GDP (Bornstein et al., 2022). The existing documentation of disparities in child development and learning in the early childhood years underscores the value of using population-based measures. For tracking progress toward equity, population-based data are critical: it is only through nationally representative or population-level samples, using internationally comparable tools, that we can estimate global progress toward reaching equity in early childhood. Beyond tracking progress toward the SDGs, these data are intended to pave the way for country-level efforts to expand and improve early childhood programs.

However, despite growing calls to invest in early childhood, implementation of effective programming in many low- and middle-income countries is weak, due to lack of capacity, uneven political commitment, and, as one challenge among many, poor use of data on early learning to mobilize and maintain action (Neuman et al., 2021). Although data can play an important role in effective implementation of ECCE, generating and using data to inform early childhood systems has several complex elements. Many countries have a patchwork of data sources on early childhood: responsibility for defining and collecting indicators of health, nutrition, and early learning may sit across ministries, including health, education, and social protection ministries; data representative of entire early childhood populations, which is central for tracking progress toward equity, may be collected on an occasional basis to provide a high-level overview of children’s status; and local researchers and program evaluators may have projects underway that generate data specific to one program model or intervention but may not be actively engaged in dialogs with governments on how best to use the results. Some data are derived from counts of health, well-being, and learning that are collected through routine services (such as the number of low-birth weight babies, collected by hospitals) while other data require direct assessment of children or observation of classrooms using measurement tools that are often based on research tools, thus requiring substantial resources to implement at scale. This assortment of data and measurement activities can be confusing and difficult to leverage for action in early childhood. Beyond academic questions about the psychometrics of valid and reliable tools, a key question lies at the heart of all data and measurement in early childhood: How are the data being used to improve early childhood systems, and by whom?

In this commentary, I raise challenges and opportunities on effectively building and using data within early childhood systems, with emphasis on how data can accelerate positive improvements in ECCE in low- and middle-income countries (LMIC). Beginning with an overview of early childhood measurement, recent steps forward in developing new measures are described. I then outline areas for future investment to ensure that data and measurement lead to the greatest possible impact on early childhood systems.

Measurement tool development and use

The studies described in this special edition relied upon the tools that emerged from the Measuring Early Learning Quality and Outcomes initiative (MELQO; UNESCO, 2017). A collaboration between UNESCO, UNICEF, Brookings Institution, and World Bank, this initiative was designed to promote measurement of quality of children’s learning environments and child learning and development in the preschool years. Building on existing tools used to measure child development and quality of learning environments, this initiative created a consortium of experts to review existing tools and generate guidance and open-source tools that could be adapted for use within low- and middle-income countries (see Raikes et al., 2019 for a description). As outlined in this special edition, the MELQO tools were adapted and implemented at the country level, leading to distinct sets of items and varied psychometric properties depending on the country context. For example, in Tanzania and Zanzibar, the child development module, called Measure of Development and Early Learning (MODEL), showed acceptable psychometric properties (e.g., Raikes et al., 2019), while the Measure of Early Learning Environments (MELE) often required extensive adaptation to fit each setting and produced small associations with child development (e.g., Maldonado et al., 2022; Su et al., 2021). Beyond the psychometric properties of the measures, though, several lessons from country experiences using MELQO have emerged. These lessons include reflections on the resources required to adapt and implement tools across large groups of children, the challenges in training data collectors to reliability when the concepts outlined in measures are not familiar to them, and the importance of aligning tools with local priorities and perspectives on child development and quality of learning environments. Drawing on these insights from MELQO, below I outline recent themes related to measures development for early child development and learning and quality of learning environments.

Investments in new tools, especially to measure child development, are ongoing

Notable investments have been made by multi-lateral organizations, universities, private funders, and non-governmental organizations in developing new tools. There are now several alternatives for measuring child development at the population level (see Fernald et al., 2017, for a thorough discussion of measurement tools). These tools address early social/emotional, cognitive, language, and motor development, generally focused on developmental milestones, such as identifying numbers and letters, building relationships with peers, and regulating emotional states. For example, using items from MELQO, the World Bank recently released a set of items indexing skills for preschool-age children demonstrating some degree of common functioning across countries (Pushparatnam et al., 2021). The IDELA, a direct assessment tool for preschool-age children developed by Save the Children, provides a more in-depth assessment of early skills, including fine motor skills (e.g., drawing a person), letter and number recognition, and emotion identification. IDELA has demonstrated reliability and validity across countries (Halpin et al., 2019). The Early Development Instrument (EDI), a teacher-reported inventory of children’s skills at the start of school, has been validated and used extensively around the world in countries of all income levels (Janus et al., 2007, 2021) and also has been used to document inequities in early development in the United States, among other countries (Halfon et al., 2020). The EDI and the ECDI, as mentioned earlier, are intended for use at the population level to track inequities.

Innovations in tool development may create more robust global measures across ages. While most data available now focus on children in the preschool years, tools to measure children’s holistic development between birth and age 3 years across diverse contexts are important. While children’s development in the first 1,000 days has profound impacts on later development (Shonkoff & Phillips, 2000), limited national or global data have been available to date. The Global Scale for Early Development (GSED), led by the World Health Organization, captures children’s development birth to age 3 years (Cavallera et al., 2023). For the GSED, items were selected based on data from existing longitudinal studies that were constructed into a “D-score” to quantify development over time and across contexts (Weber et al., 2019). The GSED took three steps that can ensure tools are applicable across settings: 1) constructing an underlying scale based on existing data points from a range of countries, 2) using qualitative work to review each item for cultural applicability, and 3) engaging in extensive testing and validation before releasing the scale for use. While these steps do not guarantee that scales will function well across contexts, such steps help ensure that scales, measures, and resulting data are applicable and relevant. Using a different psychometric approach, the Caregiver-Reported Early Development Inventory (CREDI) is available to measure children’s development birth to age 3 years and also shows strong psychometric properties (McCoy et al., 2018; Waldman et al., 2021).

While the ages of children (birth to kindergarten years) and/or the method of data collection (direct assessment, teacher report, or caregiver report) might differ, all tools mentioned here tend to have several items in common, demonstrating both coherence among different measurement tools and also reliance on similar approaches and frameworks for understanding young children’s development in highly diverse contexts. While there is evidence demonstrating the validity and reliability of some of these measures across low- and middle-income countries (e.g., see IDELA evidence by Halpin et al., 2019; see EDI evidence by; Janus et al., 2021), many tools have not yet been fully validated especially across multiple contexts. Important questions have been raised regarding the diversity in early development across contexts (Scheidecker et al., 2022), and a risk of global measurement is that this diversity in development is not captured nor appreciated. As new tools are developed, the nascent evidence base requires additional investment in measurement development to ensure that measures meet acceptable psychometric standards, including cross-cultural relevance. Further, given the reliance on similar items for both new and existing measures, the contribution of each new measure should be articulated clearly so that measurement resources are used wisely. Finally, more attention should be paid to the developmental transition from preschool to early grades. Overall, items used to index early childhood development are generally conceptually aligned with indicators of foundational learning, but the distance between the skills measured for preschool-age children and foundational learning in the early primary grades is notable (see Gochyyev et al., 2019), suggesting that data from both types of tools may be more useful and accurate if there is a more intentional progression and synchronization of tool content. Moreover, measurement of foundational learning is narrow in scope, focused on early academic skills only, while measures of early childhood also include social/emotional skills, which may be useful to continue measuring into the early grades given associations with learning outcomes (Sparapani et al., 2019).

Investments in global tools may be more common than support for building country-specific measures in LMIC

The number of global tools available suggests that there tends to be more investment in global tools to track progress between countries than there is in the development and validation of national-level tools that may have more content specifically aligned to each context, especially in low- and middle-income countries. There are exceptions. South Africa, for example, has developed a new child development measure called the Early Learning Outcomes Measure (ELOM; Dawes et al., 2018; Snelling et al., 2019) that was specifically designed for their context, but many countries do not yet have this type of tool. This raises two related issues: first, the extent to which global tools are aligned well with the culture and context in which they are used; and, second, the extent to which these tools adequately capture learning and development when used in each new context, such as through avoidance of floor and ceiling effects and inclusion of items that are culturally relevant. Using global tools, and particularly population-based measurement, most often requires precise definitions and consistent measurement of key concepts to generate reliable estimates regardless of the context. Yet the definitions of “quality” and “child development” are culturally and contextually bound (e.g., Dahlberg et al., 1999; Serpell & Marfo, 2014), and thus may or may not apply to all groups within countries, let alone regionally or globally (see Raikes, 2022, for a review on cultural impacts on quality, for example).

Use of global measures necessitates careful attention to floor and ceiling effects when used in a new setting. Given the profound impact of environment on children’s development, it is to be expected that children in one setting may have different levels of learning than children in another setting, especially across countries with notable differences in resources available to families with young children. For example, researchers in Liberia report that even tools that have been used across countries, such as MODEL, may fail to have an appropriate range for children in Liberian schools (Oxford Policy Management, (n.d.)). Therefore, tools must be carefully designed to achieve the goal of accurately capturing learning across contexts and in ways that are culturally relevant. Some work suggests that young children living in societies that rely less on printed materials may demonstrate more skills when tested using dimensional objects rather than pictures (Zuilkowski et al., 2016). Capturing the range of competencies that children have, as well as the areas for future development, will yield more accurate insights into effective early childhood systems.

Less global attention is placed on measuring quality of learning environments than child development

Quality of children’s learning environments is a critical element of ensuring that ECCE investments benefit young children and their families. Access to quality ECCE can lead to long-term, positive impacts on children’s development (Raikes et al., 2020; Wolf et al., 2018). Data on early childhood learning environments, including teacher practices, access to toys and other learning materials, safety of ECCE settings, and number of children enrolled in each classroom, can be highly impactful in identifying areas for improvement and shaping investments in early childhood settings (Bassok & Latham, 2017). Tools to measure ECCE quality typically capture multiple elements of classrooms, including teacher/child interactions, daily routines, and the safety of settings and often require trained observers who visit classrooms and stay for 2–3 h on a typical day. Similar to measures of child development, most measures of quality were initially developed for use in high-income countries; many have now been adapted for use in LMIC (see Fernald et al., 2017). However, because there is not yet agreement on what “quality” means across contexts (e.g., Dahlberg et al., 1999; Tobin et al., 2009), it is challenging to think about how to define measure quality across and within countries with diverse populations. As well, training and deploying observers to classrooms is expensive and time-consuming for ongoing monitoring – even though quality measurement may be one of the more important goals of building an effective ECCE data system.

There are almost no relevant data on ECCE quality collected globally that provide insight on the extent to which there is equity in access to quality ECCE (Raikes et al., 2023). The lack of data results from lack of agreement on how quality should be defined cross-culturally, and the lack of standardized, country-level information available on access to quality ECCE in many countries. Yet while data on child development is routinely collected by only a small number of countries, data on access to ECCE are collected through national systems much more frequently (UNESCO, 2022). Further, many countries have some system in place to collect data on ECCE for the purposes of quality assurance and adherence to national quality standards (see Raikes et al., 2021, for a recent look at quality assurance in Africa). As noted above, because these data can be used to instigate specific and actionable feedback to ECCE settings, expanding the amount of data on ECCE quality collected through routine monitoring could be a positive step forward. In sum, data on quality in ECCE may be especially critical in leveraging improvements within country settings, particularly when generated and interpreted as part of an ongoing quality improvement system.

In sum, there are now several choices available for measuring child development at the population level, with fewer focused on measuring quality. Perhaps most importantly, one lesson learned on deep investments in tool development like MELQO is that there has been much less attention placed on defining how the data were used to leverage change within ECCE systems. At present, the emphasis on tool development still often overshadows emphasis on defining the users of data and the local capacity necessary to ensure measures are culturally and contextually valid. The benefits of building new tools, as well as collecting, analyzing, and reporting data, must be large enough to justify the resources required for measurement, thus encouraging careful reflection on how the data are expected to lead to change.

Clarifying purposes for data: How are data used to leverage change?

Noting how many tools there are and the investment of time and funding toward global tool development, this continued focus on tools must be balanced against the data that countries need to make ongoing decisions on how and where to invest within their own countries. How ECCE data are used, and where and how it can lead to changes in early childhood systems is an important area for future research, especially in low- and middle-income countries.

Integrating ECCE into education data systems

The backbone of education data at the country level is the Education Management Information System, or EMIS (Hua & Herstein, 2003). While some country-level efforts to build early childhood data into EMIS have been reported (for example, in Indonesia, see Raflesia et al., 2021), the lack of reliable information in administrative data on ECCE facilities and the ages of children enrolled in them can contribute to skewed or misleading estimates of ECCE access (King et al., 2020). Several types of ECCE, including private and community-based facilities, are at risk of not being registered or counted at all, let alone included in EMIS, leading to notable discrepancies between ECCE access reported through EMIS and parents’ responses on access to ECCE through household surveys (Raikes et al., 2023). As noted below, these discrepancies and lack of reliable data can erode trust in and demand for data. Moreover, there are few if any reliable indicators of ECCE quality in many countries and even less attention to quality at the global level (Raikes et al., 2023), contributing to a global narrative in which quality in ECCE is not adequately emphasized. EMIS serves an essential function in producing data on ECCE, and addressing the present inadequacies should be a focus of efforts to build data-driven ECCE systems.

Building demand for data

To date, much of the effort on early childhood data has been “supply-side” driven, in that new tools are built without documentation of how these tools respond to the specific demands of governments, parents, and ECCE professionals for feasible, culturally relevant data that can be easily aggregated and shared to inform policy and programmatic choices. To date, little work has explicitly examined the process of building demand for data within early childhood systems. However, the broader literature on data use in policymaking outlines key considerations, including emphasis on the importance of locally relevant information in building demand for data. The use of international surveys, for example, produces much valuable information but also can displace investment in government censuses that generate more locally relevant information (Baldwin & Diers, 2009). As well, at least in some settings, there is weak commitment of global partners to invest in EMIS and other local data collection systems (Teller et al., 2011). Other issues that contribute to weak demand for data include lack of transparency on what is being collected and by whom, poor communication of research and program evaluation results to policymakers, and lack of trust arising from contradictory results from different data sources (Teller et al., 2011). These issues are infrequently addressed in the global dialog on ECCE data, which has been, to date, more frequently dominated by discussion of specific tools. Critically, more attention is also needed to demand for data from practitioners, who are essential in using data to improve children’s experiences in ECCE. Although research on practitioners’ engagement and demand for data has been conducted in the United States, where state and federal regulations mandate data collection and use in at least some ECCE settings (Brawley & Stormont, 2014), less work has addressed the demand for data across diverse country contexts as noted below. ECCE professionals could be important stakeholders in increasing demand for data.

Results reporting is perhaps one of the most important purposes of ECCE data, but there is little documentation of how and when countries share information on ECCE, and to what impact. For example, although many African countries recently reported collecting data on quality of ECCE settings through quality assurance mechanisms, few reported preparing annual reports or other ways of sharing data with the larger community, even if schools were either rewarded or punished for not meeting quality standards (Raikes et al., 2021). To promote clear connections between data and decision-making at the national level, Richter et al. (2019) outline an approach to creating country-level dashboards with relevant information on early childhood development across sectors. These types of efforts should be the focus of qualitative and quantitative research, so the field of early childhood can learn more about processes to aggregate and share data in ways that help inform policy choices. Below I have outlined three ways in which ECCE can lead to action, all of which deserve more research attention in LMIC in coming years.

Shaping political prioritization for early child development

Recent work has outlined policy advances in early childhood in several LMIC; there is some evidence on the role that data has played in building equitable early childhood systems. Neuman et al. (2021) identify raising awareness through large-scale measurement of learning outcomes as a key motivator for investing in early childhood education in each of the countries included in this special edition. Neuman et al. (2021) recommend investing in research evidence as one step toward motivating investment in quality ECCE. Underscoring this recommendation, Dusabe et al. (2019) describe how Rwanda implemented preprimary education, including the role of data in building the system. Using a diagnostic, for example, can identify areas of strength and need for improvement – for example, in Tanzania and Zanzibar. What we do not yet know is how these diagnostics created change over time, and whether the lessons learned were addressed in resulting policies or whether the reports served as a one-time discussion point.

Data intended to influence parental decision-making

Some ECCE quality improvement systems are intended to shape parental decision-making through the use of public data on the quality of ECCE programs. The United States has invested heavily in these programs, called Quality Rating and Improvement Systems (QRIS). Some research suggests that making data on ECCE publicly accessible lead parents to select other settings when data indicate that quality is low (Bassok et al., 2019). But these systems can only operate well when parents have more than one option for ECCE, thus limiting their applicability in many places where access to ECCE is constrained (see Mtahabwa, 2011, for an example from Tanzania; see Premani et al., 2021, for an example from Pakistan). However, giving parents insight into ECCE quality is one pathway by which data could lead to increased pressure on ECCE systems to change or articulation of a shared vision of what “good quality” ECCE looks like.

Data intended to inform professional development

Despite the potential power of data on ECCE to improve the quality of ECCE settings, integrating data into routine monitoring and ongoing professional development requires deep investment in ECCE infrastructure at the school and community level, including but not limited to real-time feedback based on specific data points (e.g., Weiland et al., 2018). The Pakistan ELP team (see this issue) mentions the importance of feedback cycles to maximize impacts from data, acknowledging that these cycles take time and intentionality to build. To date, few studies have examined the role of data in influencing changes in teacher practices in early childhood classrooms in LMIC. One potential barrier is the use of complicated ECCE quality measurement tools. Even in countries with long histories of measurement, such as the United States, teachers may have trouble identifying specific changes to make based on complex tools with many variables (Hanno et al., 2021). Recent work has focused on principles of social behavior change in understanding how best to use data. For example, Kalil (2022) describes using insights from behavioral science to design parenting programs that specifically target human cognitive biases to prioritize the present over the future. She concludes that tools to better support young children should be designed using the principles of behavioral science. To date, the ideas of behavioral science have not yet been extensively applied to measurement tool development, suggesting new avenues that may lead to better utilization of data.

Next steps

Looking across tools and the mechanisms by which data may leverage change, three main points emerge. First, impactful use of data rests upon shared definitions of goals and agreement on indicators of progress. Definitions are central to effective systems, yet many measures were developed in high-income countries whose systems and standards may vary substantially from LMIC settings (Fernald et al., 2017). For each country, before beginning the process of measurement, a conversation must take place: What does it mean to have a “high-quality” setting? What does healthy child development look like? How do we balance global definitions with local context and ensure agreement on these core goals of early childhood learning environments across stakeholders? A lack of agreement on core principles then can be translated into a mismatch between what’s being measured and what is actually valued within the early childhood system. Indicators of progress must be defined in ways that are culturally and contextually relevant (e.g., using measures of child development and quality that resonate with policymakers, early childhood professionals, and parents). Once agreement on the definitions of these core indicators is clarified, it is then possible to proceed toward agreement on which data are potentially most critical to inform upcoming policy decisions or programmatic investments.

Second, while country-level decision-making is undoubtedly critical for building data-driven ECCE systems, the only way that equity can be tracked across countries is with global data that are based on shared definitions and precise recommendations for measurement. Taking both of these factors into account, attention by global actors, researchers, and country stakeholders should be jointly placed on data use – who will use the data and for what purpose, and whether the data arrive with the right timing, focus, and clarity of purpose to lead to changes on behalf of young children. To use data well, ongoing investment in local capacity to develop and test contextually relevant measures, identify how and where data will be most effective, and serve as champions for data use is necessary. In addition, attention to how data are shared with and understood by practitioners and other ECCE stakeholders is important, to assess how the data can help shift mind-sets within the ECCE system. As an example of how country capacity can be built to support data and measurement, the Together for Early Childhood Evidence project has convened small country-based teams to discuss how data can be leveraged for impact within their specific country context (see Raikes et al., 2021). These types of efforts can help accelerate the process of generating relevant, actionable data at the country level, which in turn can also help inform global understanding of progress toward building ECCE systems. As we move forward, more investments in country capacity will help ensure that global tools and resulting data are used in the most impactful ways possible.

Third, one area deserves more attention: quality in ECCE. Global actors, including multi-lateral organizations, civil society, and networks of global researchers, can contribute to building data-driven systems in ECCE by increasing global focus on developing and implementing measures of ECCE quality to complement the child outcomes measurement that has largely driven discussion on early childhood in the SDGs (Raikes et al., 2023). Creating cross-country collaborations to examine tool design and functioning across diverse contexts; investing in research agendas addressing demand for ECCE data among policymakers, stakeholders, and ECCE practitioners; and, finally, increasing investments in country-level integration of all types of ECCE into EMIS can greatly improve our understanding of whether investments in ECCE are leading to quality ECCE settings for all children. The ongoing investments in ECCE portend promise for improving the learning outcomes of millions of children, but must be accompanied by large-scale investments in national and global data systems to leverage investments and track progress toward learning goals.

Disclosure statement

No potential conflict of interest was reported by the author(s).