Empirical Benchmarks to Interpret Intervention Effects on Student Achievement in Elementary and Secondary School: Meta-Analytic Results from Germany

Abstract

To assess the meaningfulness of an intervention effect on students’ achievement, researchers may apply empirical benchmarks as standards for comparisons, involving normative expectations for students’ academic growth as well as performance gaps between weak and average schools or policy-relevant groups (e.g., male and female students, students from socioeconomically advantaged or disadvantaged families, students with or without a migration background). Previous research made these empirical benchmarks available for students in the United States. We expand this line of research by providing novel meta-analytic evidence on these empirical benchmarks for students attending elementary and secondary schools in Germany for a broad variety of achievement outcomes. Drawing on the results obtained for large probability samples, we observed variations in each kind of benchmark across countries as well as across and within domains and student subpopulations within Germany. Thus, the assessment of the very same intervention effect may depend on the target population and outcome of the intervention. We offer guidelines and illustrations for applying empirical benchmarks to assess the magnitude of intervention effects.

Keywords:

• Effect size
• student performance
• educational evaluation
• intervention
• policy analysis

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://osf.io/g4nad/.

Open Research Statements

Study and Analysis Plan Registration

There is no study and analysis plan registration associated with this manuscript.

Data, Code, and Materials Transparency

The data on effect sizes and R code that support the findings of this study are openly available on the Open Science Framework at https://osf.io/g4nad.

Design and Analysis Reporting Guidelines

There is not a completed reporting guideline checklist included as a supplementary file for this manuscript.

Transparency Declaration

The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Replication Statement

This manuscript reports an original study.

Author Note

This paper uses data from the National Educational Panel Study (NEPS; see Blossfeld & Roßbach, 2019). The NEPS is carried out by the Leibniz Institute for Educational Trajectories (LIfBi, Germany) in cooperation with a nationwide network. Datasets for the Assessment of Student Achievements in German and English as a Foreign Language (DESI) and the longitudinal extensions of the year 2003 (PISA-I + 03) and 2012 (PISA-I + 12) cycles of the Programme for International Student Assessment PISA-I + were made available by the Research Data Center at the Institute for Educational Quality Improvement (FDZ at IQB). Permission from the dataset owners was granted to use these datasets for the research objectives of the present paper. Further, we used the public use files for the German student samples of the year 2000–2018 cycles of PISA, and the year 2013 and 2018 cycles of the International Computer and Information Literacy Study (ICILS) that are publicly available and re-use is permitted via an open license for the research objectives of the present paper. The R code for reproducing all results as well as the data with the effect sizes used in the present paper can be accessed via the Open Science Framework at https://osf.io/x4erk/. We made a preprint of our paper available on edarxiv at https://edarxiv.org/39gbq/. We did not preregister the analyses presented in this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 We do not provide empirical benchmarks that are based on educational intervention effects due to the very small number of randomized trials that have been carried out with student samples in Germany.

2 https://osf.io/x4erk/. Tables and Figures presented in the OSM are indicated by corresponding letters (e.g., Table B.0 in OSM B etc.).

3 It was not possible for us to integrate the effect sizes from previous research by means of meta-analytic models because many original studies did not report the necessary information on sampling variances/standard errors of the effect sizes. We therefore provide minimum, median, and maximum values to summarize these effect sizes in Table B.3.

4 Some of these data have also been used in previous studies to estimate students’ growth in achievement (see Table B.ES). However, these studies used different statistical procedures to estimate effect sizes for students’ growth. Further, none of these studies provided standard errors for these effect sizes, which precludes their meta-analytic integration with fixed-effect or random-effects models.

5 We applied several exclusion criteria to derive the samples for the present analyses (see OSM A.1). Table A.3 itemizes the number of excluded students. Sensitivity analyses showed no systematic differences in the study measures between students that were included and those that were excluded (see Tables A.4–A.7).

6 For example, for SC-3 in grade 9 (at the end of lower secondary education) the standardized mean differences in HISEI/reading/mathematics/science/ICT achievement between the students who entered and did not enter upper education were 0.66/0.83/0.91/0.80/0.81, respectively.

7 For example, for SC-3 in grade 9 (at the end of lower secondary education), the student-level variance of HISEI/reading /mathematics/science/ICT achievement for the students entering upper education was 24%/17%/21%/23%/31% smaller than the variance obtained for the total student population, respectively.

8 This argument was empirically corroborated by the values of ES_School that were obtained for specific school types. Because the underlying analyses were carried out separately for each school type, mean-level differences between school types could not affect performance gaps between schools. Relative to the performance gaps between schools that were observed in the total student population the values of ES_School for specific school types were all smaller in size (see Table D.1 in OSF D).

9 Of note, growth estimates that are based on longitudinal studies should generally be preferred over those based on cross-sectional studies because the latter are more affected by selection effects due to student attrition. Further, growth estimates provided in the present study should be preferred over longitudinal estimates from previous research (provided in the upper panel of Table B.3) because we applied a standardized analysis protocol to control for the quality of the data and statistical analyses to mitigate bias and unwanted heterogeneity in estimates of students’ academic growth (see Riley et al., 2010). When no longitudinal estimate is available, cross-sectional estimates of students’ annual academic growth obtained from previous research (provided in the lower panel of Table B.3) may serve as useful empirical benchmarks.

Additional information

Funding

This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Grant 392108331.