Identifying Progress Toward Ethnoracial Achievement Equity Across U.S. School Districts: A New Approach

Abstract

We draw on novel district-level test score data to describe novel approaches for measuring ethnoracial achievement gaps and assessing trends toward achievement equity from 2009 to 2016. Using SEDA data, we estimate gap trends for each grade over time in each district. We measure trends in both within-district gaps—comparing Black or Hispanic to White students in the same district—and national gaps—comparing a district’s Black or Hispanic students to White students nationally. Withindistrict ethnoracial gaps shrunk in one-third to two-thirds of districts, depending on subject and ethnoracial dyad. Across subjects and ethnoracial dyads, national gaps shrunk in more than half of districts, indicating that non-White students gained on White students nationally, but not in their own districts. Our findings add complexity to the achievement gap literature by (1) estimating gaps at the district level; (2) noting considerable variation in the magnitude of gap shrinkage across districts; (3) pointing to the importance of comparison group and imperfect correspondence of within-district and national gap trends in districts; and (4) identifying variation in gap trends across grades and subjects.

Keywords:

• Educational inequality
• achievement disparities
• elementary and middle school

Notes

1 There are three different forms of NAEP: NAEP Long-Term Trends (LTT) has been administered every 4 years since 1971 to students age 9, 13, and 17 and holds constant the content assessed over time. Main NAEP has been administered every 2 years since 1990 to students in grades 4, 8 and 12. Finally, NAEP Trial Urban District Assessment (TUDA) began in 2002 with six urban school districts (now 27 districts in 2019) to explore the feasibility of using NAEP to report on the performance of public school students at the district level.

2 The EdFacts raw data includes no suppressed cells, nor do they have a minimum cell size for reporting.

3 A complete examination of achievement inequality in the U.S. would include private school students, who comprised about 10% of the elementary and secondary school population during our study period. White students are slightly overrepresented while Black and Hispanic students are underrepresented in private schools. Private schools are not required to administer state tests and therefore comparable achievement data is not available.

4 For more on how SEDA researchers equated test scores into NAEP scores across states, see the section of the SEDA 3.0 technical documentation on Cutscore Estimation and Linking (Fahle et al., 2019).

5 The U.S. Department of Education stipulates that SEDA must drop estimates derived from fewer than 20 students and adds a small amount of random noise, which is described as roughly equivalent to randomly removing one student’s score from each unit- subgroup- subject- grade- year estimate. Raw counts of students therefore cannot be recovered from published estimates. For full details, refer to p. 38 and Table 14 of Fahle et al. (2019).

6 The research team first excluded about 6.7% of cases during an initial round of data cleaning. Common reasons cases were discarded include: Students took incomparable tests within the state-subject-grade-year, or the state (or district or a subgroup) had participation lower than 95% in a subject-grade-year. For details, see Fahle et al. (2019).

7 When collected seasonally—thus allowing researchers to separate school-years from summers—following individual children as they move through school can shed light on whether schooling experiences exacerbate or ameliorate achievement gaps, though much has been written about the difficulties in getting these estimates right (Quinn, 2015; Quinn & McIntyre, 2017; Reardon, 2008; von Hippel & Hamrock, 2019).

8 Reardon (2019) interprets each subgroup’s grade 3–8 mean test score growth as a measure of “middle childhood educational opportunities available to children [in a ] district when they are roughly age 9 to 14” (p. 41).

9 In the SEDA dataset, mean achievement estimates are reported in what is called the cohort-standardized (CS) scale. Fahle et al. (2019) provide the following guidance to interpreting the original CS-scale: “…we standardize the NAEP-linked cutscores relative to a reference cohort of students. This standardization is accomplished by subtracting the national grade-subject-specific mean and dividing by the national grade-subject-specific standard deviation for a reference cohort. We use the average of the three national cohorts that were in 4th grade in 2009, 2011, and 2013. We rescale…such that all means…will be interpretable as an effect size relative to the average of the three national cohorts that were in 4th grade in 2009, 2011, and 2013” (pp. 21–22).

10 In order to interpret the coefficients as described, we actually also require that the White subgroup must be initially outperforming the Black subgroup in 2009 (i.e., ${\beta }_1$ must also be positive). However, as will be shown in Table 2, in practice this is the case in over 99% of district-grade-subject observations.

11 While it is true that inequality can decline as both groups do worse, this phenomenon must be interpreted differently; this form of gap trend shrinking may be interesting in its own right but is not the focus of the current study. In Table 2, we present the number and percent of places that are excluded as based on each requirement. Depending on the subject and racial dyad, this choice (${\beta }_2$ must be positive) excludes between 13% and 35% of district-grades where shrinkage occurs.

12 There are many possible reasons a given mean achievement estimate might be missing, but the primary cause of is the U.S. Department of Education’s requirement that estimates cannot be shared from cells that contain fewer than 20 students. See our introduction of the SEDA data for an overview of missing estimates, and see Fahle et al. (2019) for a complete explanation.

13 The nationwide achievement trend for White students did not decrease during this period. This ensures that a positive ${\beta }_2$ indicates that Black mean achievement is increasing in an absolute sense. See Appendix Table B2.

14 In Online Appendix A, we explore the impact of using six different MDR definitions that vary in terms of restrictiveness/inclusiveness, as well as not using any MDR. The percentage of district-grades that are identified as exhibiting district gap shrinkage is relatively consistent across MDR definitions. However, the standard deviations (or 1st–99th percentile ranges) are quite different due to how the MDRs address outlier trends. For instance, Appendix Table A3 shows that for W-B ELA district gap trends, the 1st–99th range of estimates is from −0.60 to +0.68 with MDR definition #1. However, using the least restrictive MDR definition #6 (column 6), we observe a 1st–99th percentile range of estimates from −1.53 to +1.68. However, when we visually examined district-grades with gap trend estimates larger in magnitude than ±0.70 SDs, we almost always find these are cases with less annual achievement data available with influential data points possibly skewing the trend estimates. We generally do not think these district-grades have sufficient evidence of how their gaps are trending during this period.

15 Recall that the SEDA achievement estimates are rescaled for national gap shrinkage analyses such that a positive ${\beta }_2$ coefficient on the variable ${\mathit{year}}_{ry}^{*}$ captures a shrinking gap between the district’s non-White students and White students nationally over time. However, for the sake of clarity, we reverse scale these coefficients so that—as with district shrinkage estimates—more positive estimates represent widening gaps, and more negative estimates represent narrowing gaps (as noted in Figure 3).

16 These models include district fixed effects. We also find these correlations are quite similar if we use MDR definition 1 instead of our preferred MDR definition 2, or if we limit the analysis to districts in which all 6 grades meet the MDR (see row 3F of Table 1). Results available upon request.

Additional information

Funding

The project was supported by a research grant from the Russell Sage Foundation. We are grateful for their support. All errors are solely attributable to the authors.