Empirical Benchmarks to Interpret Intervention Effects on Student Achievement in Elementary and Secondary School: Meta-Analytic Results from Germany

Figures & data

Table 1. Normative expectations of students’ annual academic growth: total student population.

Domain	Grade transition												Educational stage
	K–1	1–2	2–3	3–4	4–5	5–6	6–7	7–8	8–9	9–10	10–11	11–12	1–4	5–9	5–10	10–12
First Language: German (Reading)
k	--	--	--	--	--	1	1	1	2	3	2	2	--	3	4	2
(mean) ES	--	--	--	--	--	0.27	0.27	0.15	0.13	0.09	0.10	0.10	--	0.16	0.15	0.10
SE	--	--	--	--	--	0.01	0.01	0.01	0.01	0.01	0.01	0.01	--	0.01	0.01	0.01
95CI lb	--	--	--	--	--	0.25	0.25	0.13	0.12	0.07	0.08	0.08	--	0.15	0.14	0.08
95CI ub	--	--	--	--	--	0.29	0.29	0.16	0.14	0.10	0.11	0.11	--	0.18	0.17	0.11
Second Language: English
k	--	--	--	--	--	--	--	--	3	--	--	--	--	3	3	--
(mean) ES	--	--	--	--	--	--	--	--	0.27	--	--	--	--	0.27	0.27	--
SE	--	--	--	--	--	--	--	--	0.02	--	--	--	--	0.02	0.02	--
95CI lb	--	--	--	--	--	--	--	--	0.23	--	--	--	--	0.23	0.23	--
95CI ub	--	--	--	--	--	--	--	--	0.30	--	--	--	--	0.30	0.30	--
Mathematics
k	1	1	1	1	--	1	1	1	1	5	2	2	3	2	5	2
(mean) ES	1.16	0.68	1.04	1.04	--	0.31	0.31	0.30	0.30	0.17	0.16	0.16	0.97	0.30	0.27	0.16
SE	0.05	0.03	0.02	0.02	--	0.01	0.01	0.01	0.01	0.00	0.01	0.01	0.02	0.01	0.01	0.01
95CI lb	1.06	0.63	1.01	1.01	--	0.29	0.29	0.29	0.29	0.16	0.15	0.15	0.93	0.29	0.25	0.15
95CI ub	1.26	0.74	1.07	1.07	--	0.33	0.33	0.32	0.32	0.18	0.17	0.17	1.01	0.32	0.28	0.17
Science
k	1	1	1	--	--	--	1	1	1	2	--	--	2	1	3	--
(mean) ES	0.61	0.84	0.84	--	--	--	0.22	0.22	0.22	0.15	--	--	0.79	0.22	0.21	--
SE	0.03	0.02	0.02	--	--	--	0.01	0.01	0.01	0.02	--	--	0.02	0.01	0.01	--
95CI lb	0.56	0.81	0.81	--	--	--	0.21	0.21	0.21	0.12	--	--	0.75	0.21	0.19	--
95CI ub	0.67	0.88	0.88	--	--	--	0.24	0.24	0.24	0.18	--	--	0.82	0.24	0.22	--
ICT
k	--	--	--	--	--	1	1	1	2	2	2	1	--	1	1	2
(mean) ES	--	--	--	--	--	0.30	0.30	0.30	0.25	0.25	0.25	0.30	--	0.30	0.30	0.25
SE	--	--	--	--	--	0.01	0.01	0.01	0.01	0.01	0.01	0.01	--	0.01	0.01	0.01
95CI lb	--	--	--	--	--	0.29	0.29	0.29	0.24	0.24	0.24	0.29	--	0.29	0.29	0.24
95CI ub	--	--	--	--	--	0.32	0.32	0.32	0.26	0.26	0.26	0.32	--	0.32	0.32	0.26
Average
k	2	2	2	1	--	2	4	4	10	12	6	6	5	12	18	6
(mean) ES	0.74	0.81	0.94	1.04	--	0.29	0.27	0.24	0.22	0.16	0.18	0.18	0.88	0.23	0.21	0.18
SE	0.03	0.02	0.02	0.02	--	0.01	0.01	0.01	0.05	0.03	0.01	0.01	0.02	0.04	0.03	0.01
95CI lb	0.68	0.77	0.91	1.01	--	0.27	0.25	0.22	0.12	0.10	0.17	0.17	0.84	0.15	0.14	0.17
95CI ub	0.80	0.84	0.97	1.07	--	0.31	0.29	0.25	0.32	0.22	0.19	0.19	0.92	0.32	0.27	0.19
σ	--	--	--	--	--	--	--	--	0.16	0.10	--	--	--	0.15	0.14	--
σ: 95CI lb	--	--	--	--	--	--	--	--	0.11	0.07	--	--	--	0.10	0.10	--
σ: 95CI ub	--	--	--	--	--	--	--	--	0.28	0.17	--	--	--	0.24	0.21	--
I^2	--	--	--	--	--	--	--	--	99%	99%	--	--	--	99%	99%	--
95PI lb	--	--	--	--	--	--	--	--	−0.11	−0.04	--	--	--	−0.07	−0.08	--
95PI ub	--	--	--	--	--	--	--	--	0.55	0.37	--	--	--	0.53	0.49	--

Notes. Further results for the meta-analyses, involving results for school types can be found in supplemental Tables B.1 and B.2. The present table shows the standardized effect sizes ES_Growth that were computed by using the average standard deviation obtained for the total student population across two successive points of measurement. Positive values indicate that students’ level of achievement increased relative to their level of achievement observed at the previous measurement point. k: number of effect sizes; (mean) ES: (average) effect size. Average effect sizes were computed by using inverse-variance weighting of effect sizes. SE: standard error of the (mean) effect size. 95CI lb/95CI ub: lower/upper bound of the 95% confidence interval for the (mean) effect size. Evaluation of the heterogeneity of effect sizes with random-effects models (when k ≥ 10): σ = total variability (i.e., standard deviation) of the (true) effect sizes; σ: 95CI lb/σ: 95CI ub: lower/upper bound of the 95% confidence interval for σ; I²: proportion of observed heterogeneity in effect sizes that is real and not due to random noise; 95PI lb/95PI ub: lower/upper bound of the 95% prediction interval for the effect sizes which provides a plausible range of values in which the true effect sizes of about 95% of all relevant populations will fall. Educational Stages: Grade levels 1–4: elementary school; grade levels 5–9/grade levels 5–10: phases to prepare students to obtain the lower secondary school leaving certificate/the intermediate secondary school leaving certificate; grade levels 10–12: upper secondary level to prepare students for the general higher education entrance qualification.

^aThe average for the transition from Grade 8–9 contains two additional growth estimates for the first language German (i.e., writing and grammar).

Table 2. Elementary education (grade 4): normative expectations for performance gaps between student demographic groups.

		Female students vs. male students				Social gradient					Migration background: neither vs. both parents
Domain	Subdomain	k	ESGender	SE	σ	k	ESSES	SE	σ	ESSES.SMD	k	ESMig	SE	σ
German		10	0.18	0.03	0.10	10	0.36	0.02	0.05	0.60	10	0.60	0.06	0.18
	Listening	2	0.07	0.02	--	2	0.40	0.01	--	0.68	2	0.74	0.03	--
	Spelling	2	0.33	0.02	--	2	0.29	0.01	--	0.48	2	0.28	0.03	--
	Reading	6	0.20	0.01	--	6	0.37	0.01	--	0.62	6	0.61	0.02	--
Mathematics		6	−0.16	0.01	--	6	0.38	0.01	--	0.63	6	0.54	0.02	--
Science		4	−0.11	0.02	--	4	0.37	0.01	--	0.62	4	0.82	0.03	--
Average		20	0.03	0.04	0.18	20	0.36	0.01	0.04	0.61	20	0.63	0.04	0.16

Notes. Further meta-analytic results can be found in supplemental Tables C.1 to C.4. All effect sizes were standardized by the standard deviation obtained for the total German student population in each grade level. k: number of observed effect sizes. ES_Gender/ES_SES/ES_Mig: average (weighted) standardized effect sizes. Positive values for gender differences indicate that female students outperform male students. ES_SES for the social gradient can be interpreted like a correlation coefficient that indicates how much domain-specific achievement improves (in standard deviation units) when students’ SES increases by one standard deviation. ES_SES.SMD depicts ES_SES in terms of a standardized mean difference between students whose HISEI score is above/below the median value in the total student population. Positive values for effect sizes related to migration background indicate that students whose parents were both born in Germany outperform students whose parents were both born in other countries. Evaluation of the heterogeneity of effect sizes with random-effects models (when k ≥ 10): σ: standard deviation of the true effect sizes.

Table 3. Lower secondary education (grade 8, 9 and 10): normative expectations for performance gaps between student demographic groups.

		Female students vs. male students				Social Gradient					Migration background: neither vs. both parents
Domain	Subdomain	k	ESGender	SE	σ	k	ESSES	SE	σ	ESSES.SMD	k	ESMig	SE	σ
German		13	0.32	0.03	0.12	13	0.33	0.01	0.02	0.55	13	0.65	0.04	0.11
	Listening	2	0.19	0.02	--	2	0.33	0.01	--	0.55	2	0.73	0.02	--
	Spelling	2	0.46	0.02	--	2	0.30	0.01	--	0.49	2	0.46	0.02	--
	Reading	9	0.29	0.01	--	9	0.33	0.01	--	0.55	9	0.67	0.02	--
English		4	0.16	0.02	--	4	0.31	0.01	--	0.51	4	0.31	0.02	--
	Listening	2	0.13	0.02	--	2	0.33	0.01	--	0.55	2	0.31	0.02	--
	Reading	2	0.18	0.02	--	2	0.28	0.01	--	0.47	2	0.31	0.02	--
Mathematics		9	−0.13	0.01	--	9	0.38	0.01	--	0.63	9	0.64	0.02	--
ICT		2	0.20	0.03	--	2	0.26	0.02	--	0.42	2	0.50	0.06	--
Science		19	0.05	0.03	0.10	19	0.36	0.01	0.01	0.59	19	0.71	0.02	0.07
	Biology	4	0.22	0.02	--	4	0.35	0.01	--	0.58	4	0.63	0.03	--
	Biology: Insights	2	0.22	0.02	--	2	0.36	0.01	--	0.59	2	0.67	0.03	--
	Biology: Knowledge	2	0.23	0.02	--	2	0.35	0.01	--	0.57	2	0.60	0.03	--
	Chemistry	4	0.10	0.02	--	4	0.35	0.01	--	0.57	4	0.60	0.02	--
	Chemistry: Insights	2	0.10	0.02	--	2	0.34	0.01	--	0.57	2	0.59	0.02	--
	Chemistry: Knowledge	2	0.09	0.02	--	2	0.35	0.01	--	0.58	2	0.62	0.03	--
	Physics	4	0.05	0.02	--	4	0.35	0.01	--	0.58	4	0.63	0.02	--
	Physics: Insights	2	0.07	0.02	--	2	0.35	0.01	--	0.58	2	0.62	0.02	--
	Physics: Knowledge	2	0.01	0.02	--	2	0.35	0.01	--	0.58	2	0.63	0.03	--
Problem solving		1	0.06	0.04	--	1	0.36	0.02	--	0.61	1	0.88	0.07	--
Average		48	0.11	0.03	0.18	48	0.34	0.01	0.02	0.56	48	0.63	0.02	0.13

Notes. Further meta-analytic results can be found in Supplemental Tables C.1 to C.4. All effect sizes were standardized by the standard deviation as obtained for the total German student population in a certain grade level. k: number of observed effect sizes; ES_Gender/ES_SES/ES_Mig: average (weighted) standardized effect sizes. Positive values for gender differences indicate that female students outperform male students. ES_SES for the social gradient can be interpreted like a correlation coefficient that indicates how much domain-specific achievement improves (in standard deviation units) when students’ SES increases by one standard deviation. ES_SES.SMD depicts ES_SES in terms of a standardized mean difference between students whose HISEI score is above/below the median value in the total student population. Positive values for effect sizes related to migration background indicate that students whose parents were both born in Germany outperform students whose parents were both born in other countries. Evaluation of the heterogeneity of effect sizes with random-effects models (when k ≥ 10): σ: standard deviation of the true effect sizes.

Table 4. Upper secondary education (grade 11 and 12): normative expectations for performance gaps between student demographic groups.

		Female students vs. male students				Social gradient					Migration background: neither vs. both parents
Domain	Subdomain	k	ESGender	SE	σ	k	ESSES	SE	σ	ESSES.SMD	k	ESMig	SE	σ
German	Reading	2	0.23	0.03	--	2	0.15	0.02	--	0.24	2	0.31	0.08	--
English	Reading	2	0.06	0.03	--	2	0.13	0.02	--	0.22	2	0.01	0.09	--
Mathematics		2	−0.60	0.03	--	2	0.17	0.02	--	0.28	2	0.39	0.09	--
Science		2	−0.36	0.03	--	2	0.13	0.02	--	0.20	2	0.53	0.09	--
ICT		2	−0.45	0.03	--	2	0.18	0.02	--	0.29	2	0.54	0.08	--
Average		10	−0.22	0.11	0.34	10	0.16	0.02	0.03	0.26	10	0.36	0.10	0.22

Notes. Further meta-analytic results can be found in supplemental Tables C.1 to C.4. All effect sizes were standardized by the standard deviation obtained for the total German student population in a certain grade level. k: number of observed effect sizes; ES_Gender/ES_SES/ES_Mig: average (weighted) standardized effect sizes. Positive values for gender differences indicate that female students outperform male students. ES_SES for the social gradient can be interpreted like a correlation coefficient that indicates how much domain-specific achievement improves (in standard deviation units) when students’ SES increases by one standard deviation. ES_SES.SMD depicts ES_SES in terms of a standardized mean difference between students whose HISEI score is above/below the median value in the total student population. Positive values for effect sizes related to migration background indicate that students whose parents were both born in Germany outperform students whose parents were both born in other countries. 95CI lb/95CI ub: lower/upper bound of the 95% confidence interval for the (average) effect size. Evaluation of the heterogeneity of effect sizes with random-effects models (when k ≥ 10): σ: standard deviation of the true effect sizes.

Figure 1. Performance Gap Between Weak and Average Schools (adapted from Bloom et al., 2008, p. 314). Note. To estimate performance gaps between schools, we adapted the approach by Bloom et al. (2008). Specifically, we drew on the standard assumption of multilevel models that the random-coefficients (u_0j) that depict regression-adjusted mean-level differences between schools (i.e., the regression-adjusted school-average achievement levels) are normally distributed with mean zero and standard deviation τ. To obtain an estimate of τ we used a two-level random-intercept model (Level 1: students, Level 2: schools) in which students’ (grand-mean centered) prior achievement, their (grand-mean centered) SES, and information on students’ gender and migration background were entered as predictors at Level 1 for each achievement outcome. Schools with a regression-adjusted school-average achievement level below/at/above zero score worse/same/better than other schools with students of the same prior achievement level and socio-demographic background characteristics. The figure shows the distribution for τ = 1.

Table 5. Performance gaps between weak and average schools: total student population.

Domain	Grade level												Educational stage
	1	2	3	4	5	6	7	8	9	10	11	12	1–4	5–9	5–10	10–12
First Language: German
k	2	1	1	1	--	--	2	--	6	1	--	2	5	8	9	2
(mean) ES	0.40	0.40	0.37	0.26	--	--	0.32	--	0.45	0.38	--	0.26	0.35	0.43	0.42	0.26
SE	0.04	0.04	0.02	0.02	--	--	0.02	--	0.02	0.04	--	0.03	0.03	0.02	0.02	0.03
95CI lb	0.33	0.33	0.32	0.21	--	--	0.27	--	0.41	0.29	--	0.21	0.29	0.39	0.39	0.21
95CI ub	0.47	0.47	0.42	0.31	--	--	0.36	--	0.48	0.46	--	0.31	0.40	0.46	0.45	0.31
Second Language: English
k	--	--	--	--	--	--	--	--	3	--	--	2	--	3	3	2
(mean) ES	--	--	--	--	--	--	--	--	0.51	--	--	0.19	--	0.51	0.51	0.19
SE	--	--	--	--	--	--	--	--	0.02	--	--	0.03	--	0.02	0.02	0.03
95CI lb	--	--	--	--	--	--	--	--	0.47	--	--	0.13	--	0.47	0.47	0.13
95CI ub	--	--	--	--	--	--	--	--	0.55	--	--	0.26	--	0.55	0.55	0.26
Mathematics
k	1	1	--	1	--	--	1	--	1	3	--	2	3	2	5	2
(mean) ES	0.48	0.30	--	0.24	--	--	0.31	--	0.31	0.33	--	0.18	0.30	0.31	0.33	0.18
SE	0.04	0.02	--	0.02	--	--	0.02	--	0.03	0.02	--	0.02	0.02	0.02	0.02	0.02
95CI lb	0.40	0.26	--	0.20	--	--	0.26	--	0.26	0.29	--	0.14	0.25	0.26	0.29	0.14
95CI ub	0.56	0.34	--	0.28	--	--	0.36	--	0.36	0.37	--	0.22	0.34	0.36	0.36	0.22
Science
k	1	--	1	--	--	--	--	--	1	2	2	--	2	1	3	2
(mean) ES	0.41	--	0.31	--	--	--	--	--	0.28	0.32	0.26	--	0.33	0.28	0.30	0.26
SE	0.05	--	0.02	--	--	--	--	--	0.03	0.02	0.02	--	0.03	0.03	0.02	0.02
95CI lb	0.32	--	0.26	--	--	--	--	--	0.22	0.28	0.21	--	0.27	0.22	0.27	0.21
95CI ub	0.51	--	0.35	--	--	--	--	--	0.33	0.37	0.30	--	0.38	0.33	0.34	0.30
ICT
k	--	--	--	--	--	--	--	--	1	--	--	2		1	1	2
(mean) ES	--	--	--	--	--	--	--	--	0.41	--	--	0.21		0.41	0.41	0.21
SE	--	--	--	--	--	--	--	--	0.03	--	--	0.03		0.03	0.03	0.03
95CI lb	--	--	--	--	--	--	--	--	0.35	--	--	0.15		0.35	0.35	0.15
95CI ub	--	--	--	--	--	--	--	--	0.47	--	--	0.27		0.47	0.47	0.27
Average
k	4	2	3	2	--	--	3	--	13	6	2	8	11	16	22	10
(mean) ES	0.42	0.33	0.31	0.25	--	--	0.31	--	0.43	0.34	0.26	0.21	0.32	0.41	0.39	0.23
SE	0.04	0.02	0.02	0.02	--	--	0.02	--	0.04	0.02	0.02	0.02	0.04	0.04	0.03	0.03
95CI lb	0.34	0.28	0.27	0.21	--	--	0.27	--	0.35	0.30	0.21	0.16	0.25	0.34	0.34	0.17
95CI ub	0.50	0.38	0.36	0.29	--	--	0.36	--	0.52	0.38	0.30	0.25	0.40	0.49	0.45	0.29
σ	--	--	--	--	--	--	--	--	0.14	--	--	--	0.08	0.13	0.12	0.05
σ: 95CI lb	--	--	--	--	--	--	--	--	0.10	--	--	--	0.05	0.10	0.09	0.03
σ: 95CI ub	--	--	--	--	--	--	--	--	0.23	--	--	--	0.14	0.20	0.17	0.09
I^2	--	--	--	--	--	--	--	--	97%	--	--	--	89%	97%	95%	67%
95PI lb	--	--	--	--	--	--	--	--	0.14	--	--	--	0.15	0.14	0.15	0.11
95PI ub	--	--	--	--	--	--	--	--	0.72	--	--	--	0.50	0.68	0.63	0.34

Notes. Results for specific school types can be found in supplemental Tables D.1 and D2. The table shows the standardized effect sizes ES_School that were obtained for the total student population across school types. Positive values indicate that the school achievement of average schools is larger than the school-average achievement of weak schools. k: number of effect sizes; (mean) ES: (average) effect size. Average effect sizes were computed by using meta-analytic models with inverse-variance weighting of effect sizes. SE: standard error of the (mean) effect size; 95CI lb/95CI ub: lower/upper bound of the 95% confidence interval for the (mean) effect size. Evaluation of the heterogeneity of effect sizes with random-effects models (when k ≥ 10): σ: total variability (i.e., standard deviation) of the (true) effect sizes; σ: 95CI lb/σ: 95CI ub: lower/upper bound of the 95% confidence interval for σ; I²: proportion of observed heterogeneity in effect sizes that is real and not due to random noise; 95PI lb/95PI ub: lower/upper bound of the 95% prediction interval for the effect sizes which provides a plausible range of values in which the true effect sizes of about 95% of all relevant populations will fall. Educational Stages: Grade levels 1–4: elementary school; grade levels 5–9/grade levels 5–10: phases to prepare students to obtain the lower secondary school leaving certificate/the intermediate secondary school leaving certificate; grade levels 10–12: upper secondary level to prepare students for the general higher education entrance qualification.

^a: The averages comprise additional effect sizes for performance gaps between schools in declarative meta-cognition (as observed in grade levels 3 and 9) that are not shown in this table (see Table D.0).

Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328. https://doi.org/10.1080/19345740802400072

 Google Scholar