Design Parameter Values for Impact Evaluations of Science and Mathematics Interventions Involving Teacher Outcomes

Abstract

Experimental research in education and training programs typically involves administering treatment to whole groups of individuals. As such, researchers rely on the estimation of design parameter values to conduct power analyses to efficiently plan their studies to detect desired effects. In this study, we present design parameter estimates from a compilation of 11 studies involving teacher-level outcomes. These results expand upon what little is known about the clustering of variance among teacher-level outcomes. The findings inform the design of intervention studies which nest teachers within schools and aim to improve teacher mathematics and science content knowledge or instructional practices. The results show large differences in unconditional intraclass correlation coefficients across studies as well as within outcomes. They also quantify the relative importance of having a pretest measure to promote efficiency. This study highlights a need for improved reporting of this information in the literature to facilitate better experimental designs of interventions involving teacher-level outcomes.

Keywords:

• Experimental design
• design parameters
• teacher professional development
• science education
• intraclass correlation

Disclosure Statement

This manuscript was first submitted to JREE on 28 August 2019. At that time Dr. Sean Reardon was the Editor-in-Chief and Dr. Elizabeth Stuart served as the corresponding editor of this manuscript through its first submission to its acceptance. Per JREE policy, the current editorial team, of which Dr. Unlu is now a part of as a co-editor, was not involved in the peer review and decision process.

Additional information

Funding

This material is based upon work supported by the National Science Foundation [Grant No. 1544236].