35
Views
0
CrossRef citations to date
0
Altmetric
Methodological Studies

Design Parameter Values for Planning Mediation Studies with Teacher and Student Mathematics Outcomes

, , , , &
Received 25 Oct 2023, Accepted 10 Apr 2024, Published online: 23 May 2024

References

  • Association of Mathematics Teacher Educators. (2013). Standards for elementary mathematics specialists: A reference for teacher credentialing and degree programs.
  • Ball, D. L., Thames, M., & Phelps, G. (2008). Content Knowledge for Teaching: What Makes It Special? Journal of Teacher Education, 59(5), 389–407. https://doi.org/10.1177/0022487108324554
  • Ballou, D. (1996). Do public schools hire the best applicants? The Quarterly Journal of Economics, 111(1), 97–133. https://doi.org/10.2307/2946659
  • Bauer, D. J., Preacher, K. J., & Gil, K. M. (2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations. Psychological Methods, 11(2), 142–163. https://doi.org/10.1037/1082-989X.11.2.142
  • Blazar, D., & Kraft, M. (2016). Teacher and teaching effects on students’ attitudes and behaviors. Educational Evaluation and Policy Analysis, 39(1), 146–170. https://doi.org/10.3102/0162373716670260
  • Blazar, D., Braslow, D., Charalambous, C., & Hill, H. (2017). Attending to general and mathematics-specific dimensions of teaching: Exploring factors across two observation instruments. Educational Assessment, 22(2), 71–94. https://doi.org/10.1080/10627197.2017.1309274
  • Brincks, A. M., Enders, C. K., Llabre, M. M., Bulotsky-Shearer, R. J., Prado, G., & Feaster, D. J. (2017). Centering predictor variables in three-level contextual models. Multivariate Behavioral Research, 52(2), 149–163. https://doi.org/10.1080/00273171.2016.1256753
  • Bulus, M., Dong, N., Kelcey, B., & Spybrook, J. (2019). PowerUpR: Power analysis tools for multilevel randomized experiments. R Package. https://powerupr.shinyapps.io/index/
  • Cashin, A. G., Lee, H., Lamb, S. E., Hopewell, S., Mansell, G., Williams, C. M., Kamper, S. J., Henschke, N., & McAuley, J. H. (2019). An overview of systematic reviews found suboptimal reporting and methodological limitations of mediation studies investigating causal mechanisms. Journal of Clinical Epidemiology, 111, 60–68.e1. https://doi.org/10.1016/j.jclinepi.2019.03.005
  • Cox, K., & Kelcey, B. (2019). Robustness of Statistical Power in Group-Randomized Studies of Mediation Under an Optimal Sampling Framework. Methodology, 15(3), 106–118. https://doi.org/10.1027/1614-2241/a000169
  • Desimone, L. (2009). Improving impact studies of teachers’ professional development: toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. https://doi.org/10.3102/0013189X08331140
  • Enochs, L. G., & Riggs, I. M. (1990). Further development of an elementary science teaching efficacy belief instrument: A preservice elementary scale. School Science and Mathematics, 90(8), 694–706. https://doi.org/10.1111/j.1949-8594.1990.tb12048.x
  • Enochs, L. G., Smith, P. L., & Huinker, D. (2000). Establishing factorial validity of the mathematics teaching efficacy beliefs instrument. School Science and Mathematics, 100(4), 194–202. https://doi.org/10.1111/j.1949-8594.2000.tb17256.x
  • Fairchild, A. J., & McDaniel, H. L. (2017). Best (but oft-forgotten) practices: mediation analysis. The American Journal of Clinical Nutrition, 105(6), 1259–1271. https://doi.org/10.3945/ajcn.117.152546
  • Filmer, D., Nahata, V., & Sabarwal, S. (2021). Preparation, practice, and beliefs: A machine learning approach to understanding teacher effectiveness. The World Bank. https://doi.org/10.1596/1813-9450-9847
  • Frenzel, A. C., Daniels, L., & Burić, I. (2021). Teacher emotions in the classroom and their implications for students. Educational Psychologist, 56(4), 250–264. https://doi.org/10.1080/00461520.2021.1985501
  • Frenzel, A. C., Pekrun, R., Goetz, T., Daniels, L. M., Durksen, T. L., Becker-Kurz, B., & Klassen, R. M. (2016). Measuring teachers’ enjoyment, anger, and anxiety: The teacher emotions scales (TES). Contemporary Educational Psychology, 46, 148–163. https://doi.org/10.1016/j.cedpsych.2016.05.003
  • Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science: The Official Journal of the Society for Prevention Research, 16(7), 893–926. https://doi.org/10.1007/s11121-015-0555-x
  • Grouws, D. A., Tarr, J. E., Chávez, Ó., Sears, R., Soria, V. M., & Taylan, R. D. (2013). Curriculum and implementation effects on high school students’ mathematics learning from curricula representing subject-specific and integrated content organizations. Journal for Research in Mathematics Education, 44(2), 416–463. https://doi.org/10.5951/jresematheduc.44.2.0416
  • Hedges, L., & Hedberg, E. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706
  • Hickman, J. J., Fu, J., & Hill, H. C. (2012). Creation and dissemination of upper-elementary mathematics assessment modules. ETS.
  • Hill, C. J., Scher, L., Hamison, J., & Granito, K. (2023). Conducting implementation research in impact studies of education interventions: A guide for researchers. http://ies.ed.gov/ncee
  • Hill, H. C., & Ball, D. L. (2004). Learning mathematics for teaching: Results from California’s Mathematics professional development Institutes. Journal for Research in Mathematics Education, 35(5), 330–351. https://doi.org/10.2307/30034819
  • Hill, H. C., Blunk, M. L., Charalambous, Y., Lewis, J. M., Phelps, G., Sleep, L., & Ball, D. L. (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: an exploratory study. Cognition and Instruction, 26(4), 430–511. https://doi.org/10.1080/07370000802177235
  • Hill, H. C., Charalambous, C. Y., & Chin, M. J. (2018). Teacher characteristics and student learning in mathematics: A comprehensive assessment. Educational Policy, 33(7), 1103–1134. https://doi.org/10.1177/0895904818755468
  • Hill, H. C., Chin, M., & Blazar, D. (2015). Teachers’ Knowledge of Students: Defining a Domain [Paper presentation]. American Educational Research Association Annual Meeting,. Chicago, IL.
  • Hill, H. C., Rowan, B., & Ball, D. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371–406. https://doi.org/10.3102/00028312042002371
  • Hong, G., & Nomi, T. (2012). Weighting methods for assessing policy effects mediated by peer change. Journal of Research on Educational Effectiveness, 5(3), 261–289. https://doi.org/10.1080/15348431.2012.688421
  • IES RFA. (2017). Institute of Education Sciences Education Research Grants Program Request for Applications 84.305A. https://ies.ed.gov/funding/pdf/2017_84305A.pdf
  • IES RFA. (2022). Institute of Education Sciences Education Research Grants Program Request for Applications 84.305A. https://ies.ed.gov/funding/pdf/2022_84305A.pdf
  • IES RFA. (2023). Institute of Education Sciences Education Research Grants Program Request for Applications 84.305A. https://ies.ed.gov/funding/pdf/2024_84305A.pdf
  • Institute of Education Sciences, U.S. Department of Education, & National Science Foundation. (2013). Common guidelines for education research and development. https://www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf
  • Jacob, R., Zhu, P., & Bloom, H. (2010). New empirical evidence for the design of group randomized trials in education. Journal of Research on Educational Effectiveness, 3(2), 157–198. https://doi.org/10.1080/19345741003592428
  • Jacobs, B. R., Hart, K. W., & Rucker, D. W. (2012). Reduction in clinical variance using targeted design changes in computerized provider order entry (CPOE) order sets: Impact on hospitalized children with acute asthma exacerbation. Applied Clinical Informatics, 3(1), 52–63. https://doi.org/10.4338/ACI-2011-01-RA-0002
  • Jacobs, V. R., Lamb, L. L. C., & Philipp, R. A. (2010). Professional noticing of children’s mathematical thinking. Journal for Research in Mathematics Education, 41(2), 169–202. https://doi.org/10.5951/jresematheduc.41.2.0169
  • Jones, N., Camburn, E., Kelcey, B., & Quintero, E. (2022). Teachers’ time use and affect before and after COVID-19 school closures. AERA Open, 8(1), 233285842110680. https://doi.org/10.1177/23328584211068068
  • Juras, R. (2016). Estimates of intraclass correlation coefficients and other design parameters for studies of school-based nutritional interventions. Evaluation Review, 40(4), 314–333. https://doi.org/10.1177/0193841X16675223
  • Kelcey, B., & Phelps, G. (2013a). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35(3), 370–390. https://doi.org/10.3102/0162373713482766
  • Kelcey, B., & Phelps, G. (2013b). Strategies for improving power in school-randomized studies of professional development. Evaluation Review, 37(6), 520–554. https://doi.org/10.1177/0193841X14528906
  • Kelcey, B., & Shen, Z. (2016). Multilevel design of school effectiveness studies in sub-saharan africa. School Effectiveness and School Improvement, 27(4), 492–510. https://doi.org/10.1080/09243453.2016.1168855
  • Kelcey, B., & Shen, Z. (2017a). Planning community-based assessments of HIV educational intervention programs in sub-Saharan Africa. Health Education Research, 32(4), 353–363. https://doi.org/10.1093/her/cyx046
  • Kelcey, B., & Shen, Z. (2017b). Experimental evaluations of school-based HIV programs in sub-Saharan Africa. Studies in Educational Evaluation, 55(2017), 27–34. https://doi.org/10.1016/j.stueduc.2017.04.004
  • Kelcey, B., Cox, K., & Dong, N. (2021). Croon’s bias-corrected factor score path analysis for small- to moderate-sample multilevel structural equation models. Organizational Research Methods, 24(1), 55–77. https://doi.org/10.1177/1094428119879758
  • Kelcey, B., Hill, H. C., & Chin, M. J. (2019). Teacher mathematical knowledge, instructional quality, and student outcomes: A multilevel quantile mediation analysis. School Effectiveness and School Improvement, 30(4), 398–431. https://doi.org/10.1080/09243453.2019.1570944
  • Kelcey, B., McGinn, D., & Hill, H. (2014). Approximate measurement invariance in cross-classified rater-mediated assessments. Frontiers in Psychology, 5, 1469–1469. https://doi.org/10.3389/fpsyg.2014.01469
  • Kelcey, B., Shen, Z., & Spybrook, J. (2016). Intraclass Correlation Coefficients for Designing School Randomized Trials in Education in Sub-Saharan Africa. Evaluation Review, 40(6), 500–525. https://doi.org/10.1177/0193841X16660246
  • Kelcey, B., Spybrook, J., & Dong, N. (2019). Sample size planning in cluster-randomized studies of multilevel mediation. Prevention Science: The Official Journal of the Society for Prevention Research, 20(3), 407–418. https://doi.org/10.1007/s11121-018-0921-6
  • Kelcey, B., Spybrook, J., Dong, N., & Bai, F. (2020). Cross-level mediation in school-randomized studies of teacher development: Experimental design and power. Journal of Research on Educational Effectiveness, 13(3), 459–487. https://doi.org/10.1080/19345747.2020.1726540
  • Kelcey, B., Xie, Y., Dong, N., & Spybrook, J. (2021). Power and sample size determination for multilevel mediation in three-level cluster-randomized. Multivariate Behavioral Research, 56(3), 496–513. https://doi.org/10.1080/00273171.2020.1738910
  • Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally-defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42(5), 499–530. https://doi.org/10.3102/1076998617695506
  • Kowalski, S., Taylor, J., Askinas, K., Wang, Q., Zhang, Q., Maddix, W., & Tipton, E. (2020). Examining factors contributing to variation in effect size estimates of teacher outcomes from studies of science teacher professional development. Journal of Research on Educational Effectiveness, 13(3), 430–458. https://doi.org/10.1080/19345747.2020.1726538
  • Krasnoff, B. (2015). What the research says about class size, professional development, and recruitment, induction and retention of highly qualified teachers: A Compendium of the Evidence on Title II, Part A, Program–Funded Strategies. Northwest Comprehensive Center at Education Northwest. https://files.eric.ed.gov/fulltext/ED558138.pdf
  • Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36(2), 249–277. https://doi.org/10.1207/S15327906MBR3602_06
  • Lee, H., Cashin, A. G., Lamb, S. E., Hopewell, S., Vansteelandt, S., VanderWeele, T. J., MacKinnon, D. P., Mansell, G., Collins, G. S., Golub, R. M., McAuley, J. H., Localio, A. R., van Amelsvoort, L., Guallar, E., Rijnhart, J., Goldsmith, K., Fairchild, A. J., Lewis, C. C., Kamper, S. J., Williams, C. M., & Henschke, N, AGReMA group. (2021). A guideline for reporting mediation analyses of randomized trials and observational studies: the AGReMA statement. Jama, 326(11), 1045–1056. https://doi.org/10.1001/jama.2021.14075
  • Marshall, J. C., Alston, D. M., & Smart, J. B. (2015). TIPS: Teacher intentionality of practice scale. Retrieved from Clemson University’s Server, www.clemson.edu/iim
  • Maslach, C., Jackson, S. E., & Leiter, M. P. (1996). Maslach burnout inventory manual (3rd ed.). CPP, Inc.
  • McLean, L. (in review) Relations between fourth grade teachers’ content-area enjoyment and students’ engagement in mathematics, science, and literacy.
  • Moss, B., Kelcey, B., & Showers, N. (2014). Classrooms as moderators of developmental education effectiveness. Community College Review, 42(3), 201–220. https://doi.org/10.1177/0091552114529153
  • National Science Foundation. (2014). Investing in science, engineering, and education for the nation’s future. https://www.nsf.gov/pubs/2014/nsf14043/nsf14043.pdf
  • National Science Foundation. (2022). Leading the world in discovery and innovation, STEM talent development and the delivery of benefits from research. https://www.nsf.gov/pubs/2022/nsf22068/nsf22068.pdf
  • Nye, B., Konstantopoulous, S., & Hedges, L. V. (2004). How large are teacher effects? Education Evaluation and Policy Analysis, 26, 327–257.
  • Phelps, G., Kelcey, B., Jones, N., & Liu, S. (2016). Informing estimates of program effects for studies of mathematics professional development using teacher content knowledge outcomes. Evaluation Review, 40(5), 383–409. https://doi.org/10.1177/0193841X16665024
  • Pianta, R. C., Hamre, B. K., & Allen, J. P. (2012). Teacher-student relationships and engagement: Conceptualizing, measuring, and improving the capacity of classroom interactions. In Handbook of research on student engagement (pp. 365–386). Springer US. https://doi.org/10.1007/978-1-4614-2018-7_17
  • Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom assessment scoring system (CLASS) manual, K-3. Paul H. Brookes Pub. Co.
  • Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41(4), 630–670. https://doi.org/10.1177/0049124112460380
  • Qin, X., & Hong, G. (2017). A weighting method for assessing between-site heterogeneity in causal mediation mechanism. Journal of Educational and Behavioral Statistics, 42(3), 308–340. https://doi.org/10.3102/1076998617694879
  • Radloff, L. S. (1977). The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385–401. https://doi.org/10.1177/014662167700100306
  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.
  • Raudenbush, S. W., & Sadoff, S. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1(2), 138–154. https://doi.org/10.1080/19345740801982104
  • Rudolph, K. E., Goin, D. E., & Stuart, E. A. (2020). The peril of power: A tutorial on using simulation to better understand when and how we can estimate mediating effects. American Journal of Epidemiology, 189(12), 1559–1567. https://doi.org/10.1093/aje/kwaa083
  • Scher, L., & O’Reilly, F. (2009). Professional development for K-12 math and science teachers: What do we really know? Journal of Research on Educational Effectiveness, 2(3), 209–249. https://doi.org/10.1080/19345740802641527
  • Shen, Z., Curran, F. C., You, Y., Splett, J. W., & Zhang, H. (2023). Intraclass correlations for evaluating the effects of teacher empowerment programs on student educational outcomes. Educational Evaluation and Policy Analysis, 45(1), 134–156. https://doi.org/10.3102/01623737221111400
  • Sim, M., Kim, S.-Y., & Suh, Y. (2022). Sample size requirements for simple and complex mediation models. Educational and Psychological Measurement, 82(1), 76–106. https://doi.org/10.1177/00131644211003261
  • Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092
  • Spybrook, J., & Raudenbush, S. W. (2009). An examination of the precision and technical accuracy of the first wave of group randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318. https://doi.org/10.3102/0162373709339524
  • Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the U.S. institute of education sciences. International Journal of Research & Method in Education, 39(3), 255–267. https://doi.org/10.1080/1743727X.2016.1150454
  • Spybrook, J., Westine, C. D., & Taylor, J. A. (2016). Design parameters for impact research in science education: A multistate analysis. AERA Open, 2(1), 233285841562597. https://doi.org/10.1177/2332858415625975
  • Spybrook, J., Zhang, Q., Kelcey, B., & Dong, N. (2020). Learning from cluster randomized trials in education: An assessment of the capacity of studies to DetermineWhat works, for whom, and under what conditions. Educational Evaluation and Policy Analysis, 42(3), 354–374. https://doi.org/10.3102/0162373720929018
  • Study of Instructional Improvement (SII). (n.d). Retrieved from the Study of Instructional Improvement project website: sii.soe.umich.edu
  • Tarr, J. E., Grouws, D. A., Chávez, Ó., & Soria, V. (2013). The effects of content organization and curriculum implementation on students’ mathematics learning in second-year high school courses. Journal for Research in Mathematics Education, 44(4), 683–729. https://doi.org/10.5951/jresematheduc.44.4.0683
  • University of Michigan. (2010). Study of instructional improvement. https://sii.soe.umich.edu/
  • VanderWeele, T. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38(4), 515–544. https://doi.org/10.1177/0049124110366236
  • VanderWeele, T. J. (2013). A three-way decomposition of a total effect into direct, indirect, and interactive effects. Epidemiology, 24(2), 224–232. https://doi.org/10.1097/EDE.0b013e318281a64e
  • Vanderweele, T. J., Hong, G., Jones, S. M., & Brown, J. L. (2013). Mediation and spillover effects in group-randomized trials: A case study of the 4Rs educational intervention. Grantee Submission, 108(502), 469–482. https://doi.org/10.1080/01621459.2013.779832
  • VanderWeele, T. J., & Chiba, Y. (2014). Sensitivity analysis for direct and indirect effects in the presence of exposure-induced mediator-outcome confounders. Epidemiology, Biostatistics and Public Health, 11(2), e9027. https://doi.org/10.2427/9027
  • Vo, H. M., Zhu, C., & Diep, N. A. (2017). The effect of blended learning on student performance at course-level in higher education: A meta-analysis. Studies in Educational Evaluation, 53, 17–28. https://doi.org/10.1016/j.stueduc.2017.01.002
  • Vo, T. T., Superchi, C., Boutron, I., & Vansteelandt, S. (2020). The conduct and reporting of mediation analysis in recently published randomized controlled trials: results from a methodological systematic review. Journal of Clinical Epidemiology, 117, 78–88. https://doi.org/10.1016/j.jclinepi.2019.10.001
  • Wayne, A. J., Yoon, K. S., Zhu, P., Cronen, S., & Garet, M. S. (2008). Experimenting with teacher professional development: Motives and methods. Educational Researcher, 37(8), 469–479. https://doi.org/10.3102/0013189X08327154
  • Webel, C., Tarr, J., Austin, C., Yeo, S., & Shim, H. (2023). Relationships between elementary mathematics specialist certification, knowledge, beliefs, and classroom learning environments. Journal of Mathematics Teacher Education, https://doi.org/10.1007/s10857-023-09602-6
  • Westine, C., Unlu, F., Taylor, J., Spybrook, J., Zhang, Q., & Anderson, B. (2020). Design Parameter Values for Impact Evaluations of Science and Mathematics Interventions Involving Teacher Outcomes. Journal of Research on Educational Effectiveness, 13(4), 816–839. https://doi.org/10.1080/19345747.2020.1821849
  • White, A. L., Way, J., Perry, B., & Southwell, B. (2005). Mathematical attitudes, beliefs and achievement in primary pre-service mathematics teacher education. Mathematics Teacher Education & Development, 7(2005/2006), 33–52.
  • Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12(4), 695–719. https://doi.org/10.1177/1094428108327450
  • Zhou, X., Shu, L., Xu, Z., & Padrón, Y. (2023). The effect of professional development on in-service STEM teachers’ self-efficacy: a meta-analysis of experimental studies. International Journal of STEM Education, 10(1), 37. https://doi.org/10.1186/s40594-023-00422-x

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.