Design Parameter Values for Impact Evaluations of Science and Mathematics Interventions Involving Teacher Outcomes

References

• Agodini, R., Harris, B., Thomas, M., Murphy, R., Gallagher, L., & Pendleton, A. (2010). Achievement effects of four early elementary school math curricula: Findings for first and second graders (Report no. NCEE 2011-4001). National Center for Educational Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Appeldoorn, K. (2004). Developing and validating the Collaboratives for Excellence in Teacher Preparation (CETP) core evaluation classroom observation protocol (COP) [Ph.D. dissertation]. University of Minnesota.
   Google Scholar
• Bell, B. A., Ferron, J. M., & Kromrey, J. D. (2008). Cluster size in multilevel models: The impact of sparse data structures on point and interval estimates in two-level models. Journal of Proceedings of the Joint Statistical Meetings, Survey Research Methods Section (pp. 1122–1129). American Statistical Association.
   Google Scholar
• Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59. https://doi.org/10.3102/0162373707299550.
   Web of Science ®Google Scholar
• Brunner, M., Keller, U., Wenger, M., Fischbach, A., & Lüdtke, O. (2018). Between-school variation in students' achievement, motivation, affect, and learning strategies: Results from 81 countries for planning group-randomized trials in education. Journal of Research on Educational Effectiveness, 11(3), 452–478. https://doi.org/10.1080/19345747.2017.1375584.
   Web of Science ®Google Scholar
• Carlisle, J. F., Kelcey, B., Rowan, B., & Phelps, G. (2011). Teachers’ knowledge about early reading: Effects on students’ gains in reading achievement. Journal of Research on Educational Effectiveness, 4(4), 289–321. https://doi.org/10.1080/19345747.2010.539297.
   Web of Science ®Google Scholar
• Coggshall, J. (2015). Title II, Part A: Don’t scrap it, don’t dilute it, fix it. American Institutes for Research. http://educationpolicy.air.org/publications/title-ii-part-dont-scrap-it-dont-dilute-it-fix-it.
   Google Scholar
• Cohen, D. K., & Hill, H. C. (2001). Learning policy: When state education reform works. Yale University Press.
   Google Scholar
• Constantine, J., Player, D., Silva, T., Hallgren, K., Grider, M., Deke, J., & Warner, E. (2009). An evaluation of teachers trained through different routes to certification: Final report (Report no. NCEE 2009-4043). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://files.eric.ed.gov/fulltext/ED504313.pdf.
   Google Scholar
• Donner, A., Birkett, N., & Buck, C. (1981). Randomization by cluster. Sample size requirements and analysis. American Journal of Epidemiology, 114(6), 906–914. https://doi.org/10.1093/oxfordjournals.aje.a113261.
   PubMed Web of Science ®Google Scholar
• Donner, A., & Koval, J. J. (1982). Design considerations in the estimation of intraclass correlation. Annals of Human Genetics, 46(3), 271–277. https://doi.org/10.1111/j.1469-1809.1982.tb00718.x
   PubMed Web of Science ®Google Scholar
• Fahle, E. M., & Reardon, S. F. (2018). How much do test scores vary among school districts? New estimates using population data 2009–2015. Educational Researcher, 47(4), 221–234. https://doi.org/10.3102/0013189X18759524.
   Web of Science ®Google Scholar
• Garet, M. S., Wayne, A. J., Stancavage, F., Taylor, J., Eaton, M., Walters, K., Song, M., Brown, S., Hurlburt, S., Zhu, P., Sepanik, S., & Doolittle, F. (2011). Middle school mathematics professional development impact study: Findings after the second year of implementation (Report no. NCEE 2011-4025). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Hedges, L., & Hedberg, E. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706.
   Web of Science ®Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489. https://doi.org/10.1177/0193841X14529126.
   PubMed Web of Science ®Google Scholar
• Heller, J. I., & Park, O. (2012). Effects of making sense of SCIENCE[TM] professional development on the achievement of middle school students, including English language learners: Final report (Report no. NCEE 2012-4002). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://eric.ed.gov/?id=ED530414
   Google Scholar
• Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers’ mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11–30. https://doi.org/10.1086/428763
   Web of Science ®Google Scholar
• Ho, A. D., & Kane, T. J. (2013). The reliability of classroom observations by school personnel. Measures of effective teaching project. Bill & Melinda Gates Foundation.
   Google Scholar
• Kelcey, B., & Phelps, G. (2013a). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35(3), 370–390. https://doi.org/10.3102/0162373713482766.
   Web of Science ®Google Scholar
• Kelcey, B., & Phelps, G. (2013b). Strategies for improving power in school-randomized studies of professional development. Evaluation Review, 37(6), 520–554. https://doi.org/10.1177/0193841X14528906.
   PubMed Web of Science ®Google Scholar
• Kelcey, B., Spybrook, J., Phelps, G., Jones, N., & Zhang, J. (2017). Designing large-scale multisite and cluster-randomized studies of professional development. The Journal of Experimental Education, 85(3), 389–410. https://doi.org/10.1080/00220973.2016.1220911.
   Web of Science ®Google Scholar
• Kish, L. (1965). Survey sampling. Wiley.
   Google Scholar
• Kowalski, S. M., Taylor, J., Askinas, K., Wang, Q., Zhang, Q., Maddix, W. P., & Tipton, E. (2020). Examining factors contributing to variation in effect size estimates of teacher outcomes from studies of science teacher professional development. Journal of Research on Educational Effectiveness, 13(3), 430–458. https://doi.org/10.1080/19345747.2020.1726538
   Web of Science ®Google Scholar
• Lee, O., & Maerten-Rivera, J. (2012). Teacher change in elementary science instruction with English language learners: Results of a multiyear professional development intervention across multiple grades. Teacher College Record, 114(8), 1–44.
   Web of Science ®Google Scholar
• Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92. https://doi.org/10.1027/1614-1881.1.3.86.
   Google Scholar
• Maeng, J. L., Whitworth, B. A., Bell, R. L., & Sterling, D. R. (2020). The effect of professional development on elementary science teachers’ understanding, confidence, and classroom implementation of reform-based science instruction. Science Education, 104(2), 326–353. https://doi.org/10.1002/sce.21562.
   Web of Science ®Google Scholar
• Matsumura, L. C., Patthey-Chavez, G. G., Valdés, R., & Garnier, H. (2002). Teacher-feedback, writing assignment quality, and third-grade students’ revision in lower- and higher-achieving urban schools. The Elementary School Journal, 103(1), 3–25. https://doi.org/10.1086/499713.
   Web of Science ®Google Scholar
• McNeish, D. M. (2014). Modeling sparsely clustered data: Design-based, model-based, and single-level methods. Psychological Methods, 19(4), 552–563. https://doi.org/10.1037/met0000024.
   PubMed Web of Science ®Google Scholar
• McNeish, D. M., & Stapleton, L. M. (2016). The effect of small sample size on two level model estimates: A review and illustration. Educational Psychology Review, 28(2), 295–314. https://doi.org/10.1007/s10648-014-9287-x.
   Web of Science ®Google Scholar
• Newman, D., Finney, P. B., Bell, S., Turner, H., Jaciw, A. P., Zacamy, J. L., & Gould, L. F. (2012). Evaluation of the effectiveness of the Alabama math, science, and technology initiative (AMSTI) (Report no. NCEE 2012-4008). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://files.eric.ed.gov/fulltext/ED529587.pdf
   Google Scholar
• Phelps, G., Kelcey, B., Jones, N., & Liu, S. (2016). Informing estimates of program effects for studies of mathematics professional development using teacher content knowledge outcomes. Evaluation Review, 40(5), 383–409. https://doi.org/10.1177/0193841X16665024.
   Web of Science ®Google Scholar
• Puma, M. J., Olsen, R. B., Bell, S. H., & Price, C. (2009). What to do when data are missing in group randomized controlled trials (Report no. NCEE 2009-0049). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/pubs/20090049/index.asp.
   Google Scholar
• Randel, B., Apthorp, H., Beesley, A. D., Clark, T. F., & Wang, X. (2016). Impacts of professional development in classroom assessment on teacher and student outcomes. The Journal of Educational Research, 109(5), 491–502. https://doi.org/10.1080/00220671.2014.992581.
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.
   Google Scholar
• Roth, K. J., Wilson, C. D., Taylor, J. A., Stuhlsatz, M., & Hvidsten, C. (2019). Comparing the effects of analysis-of-practice and content-based professional development on teacher and student outcomes in science. American Educational Research Journal, 56(4), 1217–1253. https://doi.org/10.3102/0002831218814759.
   Web of Science ®Google Scholar
• Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245–253. https://doi.org/10.1111/j.1949.8594.2002.tb17883.x.
   Google Scholar
• Schochet, P. Z. (2008). Strategies for power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87. https://doi.org/10.3102/1076998607302714.
   Web of Science ®Google Scholar
• Smith, S. P., & Banilower, E. R. (2006, April). Measuring teachers’ knowledge for teaching force and motion concepts [Paper presentation]. Annual Meeting of the National Association for Research in Science Teaching, San Francisco, CA. http://www.horizon-research.com/atlast/uploads/Papers%20and%20Presentations/NARST%2006%20teacher%20FM%20paper%20final.pdf.
   Google Scholar
• Spybrook, J. (2013). Introduction to special issue on design parameters for cluster randomized trials in education. Evaluation Review, 37(6), 435–444. https://doi.org/10.1177/0193841X14527758.
   PubMed Web of Science ®Google Scholar
• Spybrook, J., Westine, C. D., & Taylor, J. A. (2016). Design parameters for impact research in science education. AERA Open, 2(1), 233285841562597. https://doi.org/10.1177/2332858415625975.
   Web of Science ®Google Scholar
• Taylor, J. A., Getty, S. R., Kowalski, S. M., Wilson, C. D., Carlson, J., & Van Scotter, P. (2015). An efficacy trial of research-based curriculum materials with curriculum-based professional development. American Educational Research Journal, 52(5), 984–1017. https://doi.org/10.3102/0002831215585962
   Web of Science ®Google Scholar
• United States Department of Education. (2016). Findings from the 2015–2016 survey on the use of funds under Title II, Part A. https://www2.ed.gov/programs/teacherqual/leasurveyfundsrpt82016.pdf.
   Google Scholar
• United States Department of Education. (2018). Fiscal years 2017–2019 state tables for the U.S. Department of Education: Funds for state formula – allocated and selected student aid programs. U.S. Department of Education Finance. https://www2.ed.gov/about/overview/budget/statetables/index.html#update.
   Google Scholar
• Westine, C. D., Spybrook, J., & Taylor, J. A. (2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37(6), 490–519. https://doi.org/10.1177/0193841X14531584.
   PubMed Web of Science ®Google Scholar
• Xu, Z., & Nichols, A. (2010). New estimates of design parameters for clustered randomization studies: Findings from North Carolina and Florida. Urban Institute, National Center for Analysis of Longitudinal Data in Education Research.
   Google Scholar
• Zhang, Q., Spybrook, J., & Unlu, F. (2020). Examining design and statistical power for planning cluster randomized trials aimed at improving student science achievement and science teacher outcomes. AERA Open, 6(3), 1–12. https://doi.org/10.1177/233285842093952.
   Web of Science ®Google Scholar
• Zhu, P., Jacob, R., Bloom, H., & Xu, Z. (2012). Designing and analyzing studies that randomized schools to estimate intervention effects on student achievement outcomes without classroom level information. Educational Evaluation and Policy Analysis, 34(1), 45–68. https://doi.org/10.3102/0162373711423786.
   Web of Science ®Google Scholar