Experimental Design and Power for Moderation in Multisite Cluster Randomized Trials

Abstract

Multisite cluster randomized trials (MCRTs), in which, the intermediate-level clusters (e.g., classrooms) are randomly assigned to the treatment or control condition within each site (e.g., school), are among the most commonly used experimental designs across a broad range of disciplines. MCRTs often align with the theory that programs are delivered at a cluster-level (e.g., teacher professional development) and provide opportunities to explore treatment effect heterogeneity across sites. In designing experimental studies, a critical step is the statistical power analysis and sample size determination. However, the statistical tools for power analysis of moderator effects in three-level MCRTs are not available. In this study, we derived formulas for calculating the statistical power and the minimum detectable effect size difference (MDESD) with confidence intervals for investigating the effects of various moderators in three-level MCRTs. We considered the levels of the moderators (level-1, −2, and −3), the scales of the moderators (binary and continuous), and random and nonrandomly varying slopes of the (moderated) treatment effects. We validated our formulas through Monte Carlo simulations. Finally, we conclude with directions for future work.

Keywords:

• Minimum detectable effect size difference (MDESD)
• moderator
• multisite cluster randomized trials (MCRTs)
• statistical power
• treatment effect heterogeneity

The quality of study designs in educational and psychological research has been increasingly emphasized in the production of rigorous evidence of the effects of programs and policies. Because the organization of many social structures (e.g., schooling) typically involves multilevel data structure (e.g., students are nested within classrooms, and classrooms are nested within schools), multilevel experiments are widely used in research and program evaluation in these areas. Multisite cluster randomized trials (MCRTs), in which, the intermediate-level clusters (e.g., classrooms) are randomly assigned to the treatment or control condition within each site (e.g., schools), are the most commonly used experimental designs, followed by cluster randomized trials (CRTs), in which, the units for random assignment are the top-level clusters (Spybrook et al., 2016; Spybrook & Raudenbush, 2009). MCRTs align with the theory that many programs are delivered at the cluster-level (e.g., teacher professional development) and provide opportunities to explore treatment effect heterogeneity across sites (Kelcey et al., 2017; Weiss et al., 2014).

In addition to detecting simple average effects to addresses “what works” questions, researchers and policy makers are increasingly interested in additional questions regarding for “whom, and under what circumstances” programs work for, and interpreting and exploring the sources of the treatment effect variation using moderation analyses (Weiss et al., 2014). For example, it is possible that the effects of a teacher development program vary across schools or districts, and that there are heterogeneous responses by students across different subgroups defined by students’ characteristics (e.g., race, gender, SES, and pretest), teachers’ characteristics (e.g., teaching experience, race, and gender), schools’ and districts’ characteristics (e.g., size, urbanity, poverty level, and average achievement), and program implementation. These characteristics may be potential moderators, which are the variables that affect the direction and/or magnitude of the relation between the treatment variable and the outcome variable (Baron & Kenny, 1986). Understanding the context in which an intervention is likely to be (more) effective is fundamental to understanding the extent to which results are applicable and scalable to a wide range of schools and students.

In designing such studies, a critical step is the statistical power analysis. The statistical power analysis is now routinely required to demonstrate sufficient power to detect the treatment effects if they exist (e.g., Kelcey et al., 2019; US DoE & NSF, 2013). There exist the tools for power analyses of the main effects of multisite randomized trials (MRTs) (e.g., Borenstein & Hedges, 2012; Dong & Maynard, 2013; Konstantopoulos, 2008; Raudenbush et al., 2011) and for power analyses of moderator effects in two- and three-level CRTs (e.g., Dong et al., 2018, 2021; Spybrook et al., 2016). Although Dong et al. (2021) and Raudenbush and Liu (2000) provided a framework for power analysis of moderator effects in two-level multisite individual randomized trials, and Bloom and Spybrook (2017) developed formulas for the minimum detectable effect size difference (MDESD) for the site-level binary moderator in MRTs, there is no comprehensive statistical tool for power analyses of moderator effects in three-level MCRTs. It is still not clear how the intraclass correlations at levels 2 and 3, the sample size allocations, the covariates, the scales and levels of moderators, and the treatment effect variation/heterogeneity coefficients affect the statistical power of the moderator effects in three-level MCRTs. Given the increasing uses of three-level MCRTs in program evaluation, the statistical tools and software for power analyses of the effects of moderators at different levels would enhance the capacity of researchers to design rigorous studies to answer research questions related to the treatment effect heterogeneity.

To address this gap, the purpose of this study is to develop a statistical and empirical framework for designing three-level MCRTs to investigate the moderated treatment effect. Specifically, we will derive formulas for calculating the statistical power and the minimum detectable effect size difference (MDESD) with confidence intervals for investigating the effects of various potential moderators in three-level MCRTs. In the following, we first provide an illustrative example of teacher professional development for investigating moderation effects in three-level MCRTs. We then present the formulas for the standard error (SE), statistical power, and the MDESD and its confidence intervals for the moderator effect at level 1 followed by levels 2 and 3. Within this scope, we begin by detailing the case of continuous moderators with random slopes and then extend these cases to allow for binary moderators and nonrandomly varying slope models. We use Monte Carlo simulations to assess the validity of the formulas we derived. Finally, we conclude with directions for future work.

An illustrative example for investigating moderation effects in MCRTs

Universal prevention interventions have been implemented in schools to reduce student problem behaviors. For instance, using MCRTs, several classroom management programs have been found effective in reducing students’ emotional dysregulation (Reinke et al., 2018) and concentration problems (Herman, Reinke et al., 2022). In addition, the results of these two projects indicated that the treatment effects were moderated by the pretest (Reinke et al., 2018), by the special education status (Reinke et al., 2021), and by the student risk of behavior problems (Sinclair et al., 2021). However, the demographic information such as gender, race, and free lunch status were not found to be significant moderators in some universal interventions (e.g., Domitrovich et al., 2016; Herman, Dong et al., 2022; Ialongo et al., 2019; Reinke et al., 2018).

Although researchers have begun examining whether the effects of teacher professional development on student outcomes are moderated by the student level variables, there is no comprehensive framework to guide the design and analyses of moderator effects. The alignment of the analytic design with the substantive theory underlying the program can inform the rigorous design of teacher development studies (e.g., Rossi et al., 2004; Wayne et al., 2008). Recent literature has highlighted the importance of the designs and data collection regimes in the studies of professional development because they affect the scale of data collections and the types of research questions that we can address (Kelcey & Phelps, 2013; Schochet, 2011).

To provide an illustration of the complexity of the designs and the capacity to answer research questions, a simplified conceptual framework for investigating moderation effects of professional development using three-level MCRTs is illustrated in Figure 1. This is a common study of a teacher professional development program (treatment) that is designed to improve student outcomes (e.g., social and behavioral outcome). The treatment (teacher professional development) is at the intermediate or second level. In a three-level MCRT, where teachers within each school are randomly assigned to receive professional development, teacher characteristics are not related to the treatment; in non-experimental designs, the teacher characteristics may be related to the treatment status. The schools are at level 3, i.e., the schools serve as blocks or sites under which there are two treatment conditions (receiving professional development or not) in each school. The students are at level 1. The characteristics of students, teachers, and schools may be related to the student outcome (black arrows), which will not affect the accuracy of the main effect estimates of the professional development under random assignment but may affect the precision (e.g., standard error, power) of the treatment effect estimates. The key research questions may include: (1) what are the average/main effects of the professional development on the student outcome, and (2) is there any variation in the effects of the professional development across the schools (sites/blocks), and do the effects differ by the characteristics of students, teachers, and schools (red arrows indicating the moderation)? Note that it is also common to probe the mediation effect, e.g., the effect of the teacher professional development is mediated by teacher knowledge or instruction (Kelcey et al., 2019; 2020), however, this article focuses on Research Question 2 above by studying multisite moderation analyses.

Figure 1. A conceptual framework for investigating moderation effects of professional development.

Statistical framework

In designing moderation studies in three-level MCRTs (Figure 1), there are multiple considerations concerning the potential form of the moderation including (a) level(s) of the moderator variables, (b) the random or nonrandomly varying slopes (coefficients) of the treatment and moderator variables, and (c) the scale of the moderator(s) (e.g., continuous or categorical). Table 1 presents the list of moderation designs in three-level MCRTs. For example, Model MRT3-2R-1 refers to a three-level multisite randomized trial with the treatment at level 2 and a moderator at level 1, and with a random slope for the moderation. Below we describe how we develop the formulas for the standard error (SE), statistical power, and the MDESD and its CIs for the moderator effect at level 1 followed by levels 2 and 3. Within this scope, we first detail the case of continuous moderators with random slopes and then extend these cases to allow for binary moderators and nonrandomly varying slope models.

Table 1. List of design and software modules of three-level multisite cluster randomized trials (MCRTs).

1	2	3	4	5	6	7
Model Number	Level of Moderator	Slope of Treatment or Moderation	Binary Moderator		Continuous Moderator
			MDESD Calculation	Power Calculation	MDESD Calculation	Power Calculation
MRT3-2R-1	1	Random
MRT3-2R-2	2	Random	MRT32R_MDESD	MRT32R_Power	MRT32Rc_MDESD	MRT32Rc_Power
MRT3-2R-3	3	Random
MRT3-2N-1	1	Nonrandomly Varying
MRT3-2N-2	2	Nonrandomly Varying	MRT32N_MDESD	MRT32N_Power	MRT32Nc_MDESD	MRT32Nc_Power
MRT3-2N-3	3	Nonrandomly Varying

The random slope Model

A random slope model assumes that the moderator effect randomly varies across sites when the moderator is at levels 1 or 2, or the treatment effect randomly varies across sites after controlling for the level-3 moderator.

Moderator at level 1

Suppose there are n students in each teacher’s classroom. There are J teachers per school, where a proportion (P) of the teachers within each school are randomly assigned to the treatment group to receive professional development, and there are total K schools which serve as blocks or sites. For example, a research question may probe the extent to which the effects of a professional development program on a student outcome vary by the students’ pretest or gender (i.e., the moderated treatment effects) while the moderated treatment effects also varying randomly across schools. This design corresponds to Model MRT3-2R-1 in Table 1.

To test for this cross-level moderation in the presence of heterogeneous effects across schools, we use three-level random slope hierarchical linear modeling (HLM) (Raudenbush & Bryk, 2002): (1) $$\mathrm{\text{Level}}\mathrm{ }1:Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{M}_{\mathit{\text{ijk}}}^{\left(1\right)}+{\pi }_{2\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}},e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|M,X}^2).$$(1) (2) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }2:{\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}\\ {\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}+{\beta }_{11k}{T}_{\mathit{\text{jk}}}+r_{1\mathit{\text{jk}}}\\ \begin{array}{c}{\pi }_{2\mathit{\text{jk}}}={\beta }_{20k}\\ \left(\begin{array}{c}{r}_{0\mathit{\text{jk}}}\\ r_{1\mathit{\text{jk}}}\end{array}\right)\sim N\left[\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}{\tau }_{00|T,W}^2& {\tau }_{01|T,W}\\ & {\tau }_{11|T}^2\end{array}\right)\right].\end{array}\end{array}$$(2) (3) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }3:{\beta }_{00k}={\gamma }_{000}+u_{00k}\\ {\beta }_{01k}={\gamma }_{010}+u_{01k}\\ \begin{array}{c}{\beta }_{02k}={\gamma }_{020}\\ {\beta }_{10k}={\gamma }_{100}+u_{10k}\\ \begin{array}{c}{\beta }_{11k}={\gamma }_{110}+u_{11k}\\ \begin{array}{c}{\beta }_{20k}={\gamma }_{200}\\ \left(\begin{array}{c}\begin{array}{c}{u}_{00k}\\ u_{01k}\\ u_{10k}\end{array}\\ u_{11k}\end{array}\right)\sim N\left[\left(\begin{array}{c}\begin{array}{c}0\\ 0\\ 0\end{array}\\ 0\end{array}\right),\left(\begin{array}{cccc}{\tau }_{0000}^2& {\tau }_{0001}& {\tau }_{0010}& {\tau }_{0011}\\ & {\tau }_{0101}^2& {\tau }_{0110}& {\tau }_{0111}\\ & & {\tau }_{1010}^2& {\tau }_{1011}\\ & & & {\tau }_{1111}^2\end{array}\right)\right].\end{array}\end{array}\end{array}\end{array}$$(3) $Y_{\mathit{\text{ijk}}}$ is the outcome for student i with teacher j in school k. The treatment variable, $T_{\mathit{\text{jk}}},$ is a binary variable indicating whether the teachers receive the professional development. $X_{\mathit{\text{ijk}}}$ is a level-1 covariate and $W_{\mathit{\text{jk}}}$ is a level-2 covariate. $M_{\mathit{\text{ijk}}}^{(1)}$ is a continuous level-1 moderator, and $M_{\mathit{\text{ijk}}}^{(1)}\sim N(0,{\sigma }_{M^{(1)}}^2).$ $M_{\mathit{\text{ijk}}}^{(1)}$ can be viewed as a grand-mean centered variable. Of interest for the moderator analysis is the parameter ${\beta }_{11k},$ which represents the site-specific moderation effect and consists of the cross-site average moderated treatment effect (${\gamma }_{110}$) and the random site-specific deviation from that average ($u_{11k}$). ${\gamma }_{110}$ can also be interpreted as the average difference on the association of the variable $M_{\mathit{\text{ijk}}}^{(1)}$ and the outcome between the treatment conditions.

By extending Snijders (2001, 2005) work on the variance of the estimated regression coefficients of a level-1 variable with a random slope, Dong et al. (2021) showed that the variance of the cross-level moderation effect estimate in two-level multisite randomized trials is associated with the residual variance in the level-1 slope, the variance of the level-1 residuals, and the variances of the moderator and treatment variables in addition to sample sizes (Dong et al., 2021, Equation (8)(8) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }2:{\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{T}_{\mathit{\text{jk}}}{M}_{\mathit{\text{jk}}}^{\left(2\right)}+{\beta }_{03k}{M}_{\mathit{\text{jk}}}^{\left(2\right)}+{\beta }_{04k}{}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}},r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,M,W}^2),\\ {\pi }_{1\mathit{\text{jk}}}={\beta }_{10k},\end{array}$$(8) ). We extend Dong et al. (2021) and Snijders (2001, 2005) to three-level MCRTs, and the standard error of the moderator effect estimate (${\widehat{\gamma }}_{110}$) can be expressed as: (4) $${\mathit{\text{SE}}(\widehat{\gamma }}_{110})=\sqrt{\frac{{\tau }_{1111}^2}{K}+\frac{{\tau }_{11|T}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|M,X}^2}{{\sigma }_{M^{\left(1\right)}}^{2}P(1-P)\mathit{\text{KJn}}}}.$$(4)

The standard error of the moderation effect is associated with the variance (${\tau }_{1111}^2$) of the random slope of the moderation ($M_{\mathit{\text{ijk}}}^{(1)}\times T_{\mathit{\text{jk}}}$) across sites/blocks (level 3), the variance (${\tau }_{11|T}^2$) of the random slope of the moderator ($M_{\mathit{\text{ijk}}}^{(1)}$) among level-2 clusters conditional on treatment, the variance (${\sigma }_{1|M,X}^2$) of level-1 residuals conditional on $M_{\mathit{\text{ijk}}}^{(1)}$ and $X_{\mathit{\text{ijk}}},$ the variance of $M_{\mathit{\text{ijk}}}^{(1)},$ the variance [$P(1-P)$] of the treatment variable, and the sample sizes (K, J, n).

Note that when $M_{\mathit{\text{ijk}}}^{(1)}$ is a binary variable with a proportion of $Q_1$ in one subgroup and (1-$Q_1$) in another subgroup, $M_{\mathit{\text{ijk}}}^{(1)}$ ∼ Bernoulli ($Q_1$): (5) $$\mathit{\text{VAR}}\left(M_{\mathit{\text{ijk}}}^{\left(1\right)}\right)={\sigma }_{M^{\left(1\right)}}^2=Q_1(1-Q_1).$$(5)

We insert Expression 5 into Expression 4, and the standard error of the moderator effect estimate for a binary level-1 moderator can be expressed as: (6) $${\mathit{\text{SE}}(\widehat{\gamma }}_{110})=\sqrt{\frac{{\tau }_{1111}^2}{K}+\frac{{\tau }_{11|T}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|M,X}^2}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}.$$(6)

Moderator at level 2

The research question is whether the effects of the professional development on student outcome vary by the teachers’ teaching experience or gender (i.e., the moderated treatment effects) while the moderated treatment effects also varying randomly across schools. This design corresponds to Model MRT3-2R-2 in Table 1.

To test for this same-level moderation, we use three-level random slope HLM (Raudenbush & Bryk, 2002): (7) $$\mathrm{\text{Level}}\mathrm{ }1:Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}},e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|X}^2).$$(7) (8) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }2:{\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{T}_{\mathit{\text{jk}}}{M}_{\mathit{\text{jk}}}^{\left(2\right)}+{\beta }_{03k}{M}_{\mathit{\text{jk}}}^{\left(2\right)}+{\beta }_{04k}{}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}},r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,M,W}^2),\\ {\pi }_{1\mathit{\text{jk}}}={\beta }_{10k},\end{array}$$(8) (9) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }3:{\beta }_{00k}={\gamma }_{000}+u_{00k}\\ {\beta }_{01k}={\gamma }_{010}+u_{01k}\\ \begin{array}{c}{\beta }_{02k}={\gamma }_{020}+u_{02k}\\ {\beta }_{03k}={\gamma }_{030}\\ \begin{array}{c}{\beta }_{04k}={\gamma }_{040}\\ {\beta }_{10k}={\gamma }_{100}\\ \left(\begin{array}{c}{u}_{00k}\\ u_{01k}\\ u_{02k}\end{array}\right)\sim N\left[\left(\begin{array}{c}0\\ 0\\ 0\end{array}\right),\left(\begin{array}{ccc}{\tau }_{0000}^2& {\tau }_{0001}& {\tau }_{0002}\\ & {\tau }_{0101}^2& {\tau }_{0102}\\ & & {\tau }_{0202}^2\end{array}\right)\right].\end{array}\end{array}\end{array}$$(9) $M_{\mathit{\text{jk}}}^{\left(2\right)}$ is a continuous level-2 moderator, and $M_{\mathit{\text{jk}}}^{(2)}\sim N(0,{\sigma }_{M^{(2)}}^2).$ The parameter, ${\beta }_{02k},$ represents the site-specific moderation effect and consists of the cross-site average moderated treatment effect (${\gamma }_{020}$) and the random site-specific deviation from that average ($u_{02k}$).

By extending Dong et al. (2021) and Snijders (2001, 2005) to three-level MCRTs, the standard error of the moderator effect point estimate (${\widehat{\gamma }}_{110}$) can be expressed as: (10) $${\mathit{\text{SE}}(\widehat{\gamma }}_{020})=\sqrt{\frac{{\tau }_{0202}^2}{K}+\frac{{\tau }_{00|T,M,W}^2}{{\sigma }_{M^{\left(2\right)}}^{2}P(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|X}^2}{{\sigma }_{M^{\left(2\right)}}^{2}P(1-P)\mathit{\text{KJn}}}}.$$(10)

Similarly, when the level-2 moderator $M_{\mathit{\text{jk}}}^{\left(2\right)}$ is a binary variable with a proportion of $Q_2$ in one subgroup, $M_{\mathit{\text{jk}}}^{\left(2\right)}$∼ Bernoulli ($Q_2$): $\mathit{\text{VAR}}\left(M_{\mathit{\text{jk}}}^{\left(2\right)}\right)={\sigma }_{M^{(2)}}^2=Q_2(1-Q_2).$

The standard error of the moderator point estimate for a binary level-2 moderator can be expressed as: (11) $${\mathit{\text{SE}}(\widehat{\gamma }}_{020})=\sqrt{\frac{{\tau }_{0202}^2}{K}+\frac{{\tau }_{00|T,M,W}^2}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|X}^2}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}.$$(11)

Moderator at level 3

A level-3 moderator example research question in the current context is whether the effects of the professional development program on a student outcome vary by the site-level characteristics (e.g., school size, urbanity) while also varying randomly across schools. This design corresponds to Model MRT3-2R-3 in Table 1.

To test for this cross-level moderation, we use three-level random slope HLM (Raudenbush & Bryk, 2002): (12) $$\mathrm{\text{Level}}\mathrm{ }1:Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}},e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|X}^2).$$(12) (13) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }2:{\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}},r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,W}^2),\\ {\pi }_{1\mathit{\text{jk}}}={\beta }_{10k},\end{array}$$(13) (14) $$\begin{array}{c}\mathrm{\text{Level}}\mathrm{ }3:{\beta }_{00k}={\gamma }_{000}+{\gamma }_{001}{M}_k^{\left(3\right)}+u_{00k}\\ {\beta }_{01k}={\gamma }_{010}+{\gamma }_{011}{M}_k^{\left(3\right)}+u_{01k}\\ \begin{array}{c}{\beta }_{02k}={\gamma }_{020}\\ {\beta }_{10k}={\gamma }_{100}\\ \left(\begin{array}{c}{u}_{00k}\\ u_{01k}\end{array}\right)\sim N\left[\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}{\tau }_{0000|M}^2& {\tau }_{0001|M}\\ & {\tau }_{0101|M}^2\end{array}\right)\right].\end{array}\end{array}$$(14) $M_k^{\left(3\right)}$ is a continuous level-3 moderator, and $M_k^{(3)}\sim N(0,{\sigma }_{M^{(3)}}^2).$ The parameter, ${\beta }_{01k},$ represents the site-specific treatment effects that include three components: (1) the average treatment effects across sites (${\gamma }_{010}$), (2) the average moderation effect (${\gamma }_{011}$) across sites, and (3) the random treatment effects across sites ($u_{01k}$).

By extending Dong et al. (2021) and Snijders (2001, 2005) to three-level MCRTs, the standard error of the moderator effect estimate (${\widehat{\gamma }}_{011}$) can be expressed as: (15) $${\mathit{\text{SE}}(\widehat{\gamma }}_{011})=\sqrt{\frac{{\tau }_{0101|M}^2}{{\sigma }_{M^{\left(3\right)}}^{2}K}+\frac{{\tau }_{00|T,W}^2}{{\sigma }_{M^{\left(3\right)}}^2\mathrm{P}(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|X}^2}{{\sigma }_{M^{\left(3\right)}}^{2}P(1-P)\mathit{\text{KJn}}}}.$$(15)

Similarly, when the level-3 moderator $M_k^{\left(3\right)}$ is a binary variable with a proportion of $Q_3$ in one subgroup, $M_k^{\left(3\right)}$∼ Bernoulli ($Q_3$): $\mathit{\text{VAR}}\left(M_k^{\left(3\right)}\right)={\sigma }_{M^{(3)}}^2=Q_3(1-Q_3).$

The standard error of the moderator point estimate for a binary level-3 moderator can be expressed as: (16) $${\mathit{\text{SE}}(\widehat{\gamma }}_{011})=\sqrt{\frac{{\tau }_{0101|M}^2}{Q_3(1-Q_3)K}+\frac{{\tau }_{00|T,W}^2}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|X}^2}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}}.$$(16)

Power formulas

We can test ${\gamma }_{110},$ ${\gamma }_{020},$ and ${\gamma }_{011}$ using a t-test. Assuming the alternative hypothesis is true, the test statistic follows a non-central t-distribution, T’, and the noncentrality parameters (unstandardized) for the continuous moderators are: (17) $${\lambda }_{|M^{\left(1\right)}}={\widehat{\gamma }}_{110}/\sqrt{\frac{{\tau }_{1111}^2}{K}+\frac{{\tau }_{11|T}^2}{\mathrm{P}(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|M,X}^2}{{\sigma }_{M^{\left(1\right)}}^{2}P(1-P)\mathit{\text{KJn}}}},$$(17) (18) $${\lambda }_{|M^{\left(2\right)}}={\widehat{\gamma }}_{020}/\sqrt{\frac{{\tau }_{0202}^2}{K}+\frac{{\tau }_{00|T,M,}^2}{{\sigma }_{M^{\left(2\right)}}^{2}P(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|X}^2}{{\sigma }_{M^{\left(2\right)}}^{2}P(1-P)\mathit{\text{KJn}}}},$$(18) and (19) $${\lambda }_{|M^{\left(3\right)}}={\widehat{\gamma }}_{011}/\sqrt{\frac{{\tau }_{0101|M}^2}{{\sigma }_{M^{\left(3\right)}}^{2}K}+\frac{{\tau }_{00|T,W}^2}{{\sigma }_{M^{\left(3\right)}}^{2}P(1-P)\mathit{\text{KJ}}}+\frac{{\sigma }_{1|X}^2}{{\sigma }_{M^{\left(3\right)}}^{2}P(1-P)\mathit{\text{KJn}}}}.$$(19)

Note that the treatment may affect the variance of the outcome and the association of the outcome with the covariate and moderator for the treatment group. We use the common assumption of homogeneous variance of residuals between the treatment and control groups; that is, the variances are equal between the treatment and control groups after the treatment effect is accounted for. For instance, this assumption implies that the variance (${\tau }_{11|T}^2$) of the level-1 moderator slope among level-2 clusters conditional on treatment from the analysis of data including both treatment and control groups is equal to the variance (${\tau }_{11}^2$) from the analysis of data only from the control group. In addition, we use the variance components (${\tau }_3^2,$ ${\tau }_2^2,$ and ${\sigma }_1^2$ are levels-3, 2, and 1 variances) in the three-level unconditional HLM for standardization. (20) $$\mathrm{\text{Level}}\mathrm{ }1:Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+e_{\mathit{\text{ijk}}},e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_1^2).$$(20) (21) $$\mathrm{\text{Level}}\mathrm{ }2:{\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+r_{0\mathit{\text{jk}}},r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_2^2).$$(21) (22) $$\mathrm{\text{Level}}\mathrm{ }3:{\beta }_{00k}={\gamma }_{000}+u_{00k},u_{00k}\sim N(0,{\tau }_3^2).$$(22)

Let ${\sigma }_{M^{(1)}}^2=1$ and ${\delta }_{1c}={\gamma }_{110}/\sqrt{{\tau }_3^2+{\tau }_2^2+{\sigma }_1^2},$ the noncentrality parameter (standardized) for the continuous level-1 moderator is: (23) $${\lambda }_{|M^{\left(1\right)}}={\widehat{\delta }}_{1c}/\sqrt{\frac{{\omega }_{3T{M}^{\left(1\right)}}^2}{K}+\frac{{\omega }_{2M^{\left(1\right)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}.$$(23) ${\rho }_3$ is the unconditional intraclass correlation coefficient (ICC) at level 3, ${\rho }_3={\tau }_3^2/({\tau }_3^2+{\tau }_2^2+{\sigma }_1^2).$ ${\rho }_2$ is the unconditional ICC at level 2, ${\rho }_2={\tau }_2^2/({\tau }_3^2+{\tau }_2^2+{\sigma }_1^2).$ ${\omega }_{3T{M}^{(1)}}^2={\tau }_{1111}^2/({\tau }_3^2+{\tau }_2^2+{\sigma }_1^2)$ indicates the standardized effect variability of the moderation ($M_{\mathit{\text{ijk}}}^{(1)}\times T_{\mathit{\text{jk}}}$) across blocks (level 3). ${\omega }_{2M^{(1)}}^2={\tau }_{11|T}^2/({\tau }_3^2+{\tau }_2^2+{\sigma }_1^2)$ indicates the standardized effect variability of the random slope of the level-1 moderator ($M_{\mathit{\text{ijk}}}^{(1)}$) among level-2 clusters conditional on the treatment variable ($T_{\mathit{\text{jk}}}$). $R_1^2$ is the proportion of variance at level 1 that is explained by the level-1 covariate ($X_{\mathit{\text{ijk}}}$) and moderator ($M_{\mathit{\text{ijk}}}^{(1)}$): $R_1^2=1-{\sigma }_{1|M,X}^2/{\sigma }_1^2.$

The standardized noncentrality parameter for the binary level-1 moderator is: (24) $${\lambda }_{|M^{\left(1\right)}}={\widehat{\delta }}_{1b}/\sqrt{\frac{{\omega }_{3T{M}^{\left(1\right)}}^2}{K}+\frac{{\mathrm{\omega }}_{2M^{\left(1\right)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}.$$(24)

The statistical power for a two-sided test with the degrees of freedom of $v=K-1$ is: $$1-\beta =1-P\left[T^{\prime }\left(K-1{,\lambda }_{|M^{\left(1\right)}}\right)<t_0\right]+P\left[T^{\prime }\right(K-1{,\lambda }_{|M^{\left(1\right)}})\le {-t}_0],\text{where}{t}_0=t_{1-\frac{\alpha }{2},K-1}.$$

Let ${\sigma }_{M^{(2)}}^2=1$ and ${\delta }_{2c}={\gamma }_{020}/\sqrt{{\tau }_3^2+{\tau }_2^2+{\sigma }_1^2},$ the noncentrality parameter (standardized) for the continuous level-2 moderator is: (25) $${\lambda }_{|M^{\left(2\right)}}={\widehat{\delta }}_{2c}/\sqrt{\frac{{\omega }_{3T{M}^{\left(2\right)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}.$$(25) ${\omega }_{3T{M}^{(2)}}^2={\tau }_{0202}^2/({\tau }_3^2+{\tau }_2^2+{\sigma }_1^2)$ indicates the standardized effect variability of the random slope of the moderation ($T_{\mathit{\text{jk}}}{M}_{\mathit{\text{jk}}}^{\left(2\right)}$) across sites (level 3). $R_2^2$ is the proportion of variance at the level-2 intercept that is explained by the level-2 covariate ($W_{\mathit{\text{jk}}}$), moderator ($M_{\mathit{\text{jk}}}^{\left(2\right)}$), treatment variable ($T_{\mathit{\text{jk}}}$), and the interaction ($T_{\mathit{\text{jk}}}{M}_{\mathit{\text{jk}}}^{\left(2\right)}$): $R_2^2=1-{\tau }_{00|T,M,W}^2/{\tau }_2^2.$

The standardized noncentrality parameter for the binary level-2 moderator is: (26) $${\lambda }_{|M^{\left(2\right)}}={\widehat{\delta }}_{2b}/\sqrt{\frac{{\omega }_{3T{M}^{\left(2\right)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}.$$(26)

The statistical power for a two-sided test with the degrees of freedom of $v=K-1$ is: $$1-\beta =1-P\left[T^{\prime }\left(K-1{,\lambda }_{|M^{\left(2\right)}}\right)<t_0\right]+P\left[T^{\prime }\right(K-1{,\lambda }_{|M^{\left(2\right)}})\le {-t}_0],\text{where}{t}_0=t_{1-\frac{\alpha }{2},K-1}.$$

Let ${\sigma }_{M^{(3)}}^2=1$ and ${\delta }_{3c}={\gamma }_{011}/\sqrt{{\tau }_3^2+{\tau }_2^2+{\sigma }_1^2},$ the noncentrality parameter (standardized) for the continuous level-3 moderator is: (27) $${\lambda }_{|M^{\left(3\right)}}={\widehat{\delta }}_{3c}/\sqrt{\frac{{\omega }_{3T}^2-{\widehat{\delta }}_{3c}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}.$$(27) ${\omega }_{3T}^2={\tau }_{0101}^2/({\tau }_3^2+{\tau }_2^2+{\sigma }_1^2)$ indicates the standardized effect variability of the treatment effect ($T_{\mathit{\text{jk}}}$) across blocks (level 3), where ${\tau }_{0101}^2$ is the variance of the treatment effect that is unconditional on any moderator. ${\tau }_{0101|M}^2={\tau }_{0101}^2-{\widehat{\gamma }}_{011}^2{\sigma }_{M^{(3)}}^2.$

The standardized noncentrality parameter for the binary level-3 moderator is: (28) $${\lambda }_{|M^{\left(3\right)}}={\widehat{\delta }}_{3b}/\sqrt{\frac{{\omega }_{3T}^2-{\widehat{\delta }}_{3b}^2{Q}_3(1-Q_3)}{K{Q}_3(1-Q_3)}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}}.$$(28)

The statistical power for a two-sided test with the degrees of freedom of $v=K-2$ is: $$1-\beta =1-P\left[T^{\prime }\left(K-2{,\lambda }_{|M^{\left(3\right)}}\right)<t_0\right]+P\left[T^{\prime }\right(K-2{,\lambda }_{|M^{\left(3\right)}})\le {-t}_0],\text{where}{t}_0=t_{1-\frac{\alpha }{2},K-2}.$$

Note that the effect size of the moderation effect is defined as the standardized coefficient for a continuous moderator in Expressions 23, 25, and 27, and as the standardized mean difference for a binary moderator in Expressions 24, 26, and 28. The parameters (${\rho }_3,$ ${\rho }_2,$ ${\omega }_{2M^{(1)}}^2,$ and $R_1^2$) can be estimated from the data in observational studies without any interventions, or data from the intervention studies controlling for the treatment variables, while the parameters (${\omega }_{3T{M}^{(1)}}^2,$ ${\omega }_{3T{M}^{(2)}}^2,$ and ${\omega }_{3T}^2$) must be estimated from the intervention studies.

Formulas for the minimum detectable effect size difference with confidence interval

In addition to knowing the statistical power for a study to detect a desired effect size, it is useful to know the minimum effect size difference that a moderation study can detect with sufficient power (e.g., 80%) given sample sizes. The minimum detectable effect size difference (MDESD) can be expressed as (Bloom, 1995, 2005, 2006; Dong et al., 2018; Dong, Spybrook, & Kelcey et al., 2020; Murray, 1998; Spybrook, Kelcey, & Dong et al., 2016): (29) $$\mathit{\text{MDESD}}\left(\left|\widehat{\delta }\right|\right)=M_v\times \mathit{\text{SE}}\left(\widehat{\gamma }\right)/{\mathit{\text{SD}}}_Y,$$(29) where $M_v=t_{\alpha }+t_{1-\beta }$ for one-tailed tests with v degrees of freedom, and $M_v=t_{\alpha /2}+t_{1-\beta }$ for two-tailed tests. $\mathit{\text{SE}}(\widehat{\gamma })$ is the standard error of the moderation effect estimate as in Expressions 4, 6, 10, 11, 15 & 16. ${\mathit{\text{SD}}}_Y$ is the standard deviation of the outcome measure (Y), and is defined as the square root of the total unconditional variance in Expressions 20-22, ${\mathit{\text{SD}}}_Y=\sqrt{{\tau }_3^2+{\tau }_2^2+{\sigma }_1^2}.$

Hence, the MDESD regarding the standardized coefficient for a continuous level-1 moderator is: (30) $$\mathit{\text{MDESD}}\left(\left|{\widehat{\delta }}_{1c}\right|\right)=M_v\sqrt{\frac{{\omega }_{3T{M}^{\left(1\right)}}^2}{K}+\frac{{\omega }_{2M^{\left(1\right)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}},$$(30) where the degrees of freedom of $v=K-1.$

The 100*(1-α)% confidence interval for $\mathit{\text{MDESD}}\left(\left|{\widehat{\delta }}_{1c}\right|\right)$ is given by: (31) $${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\omega }_{3T{M}^{\left(1\right)}}^2}{K}+\frac{{\omega }_{2M^{\left(1\right)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}.$$(31)

The MDESD regarding the standardized mean difference for a binary moderator is: (32) $$\mathit{\text{MDESD}}\left(\left|{\widehat{\delta }}_{1b}\right|\right)=M_v\sqrt{\frac{{\omega }_{3T{M}^{\left(1\right)}}^2}{K}+\frac{{\omega }_{2M^{\left(1\right)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}.$$(32)

The 100*(1-α)% confidence interval for $\mathit{\text{MDESD}}\left(\left|{\widehat{\delta }}_{1b}\right|\right)$ is given by: (33) $${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\omega }_{3T{M}^{\left(1\right)}}^2}{K}+\frac{{\omega }_{2M^{\left(1\right)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}.$$(33)

Similarly, the MDESD and its 100*(1-α)% confidence interval for a continuous or binary level-2 or −3 moderator are presented in Table 2.

Table 2. Summary of standardized noncentrality parameters, MDESD and 100*(1-α)% confidence intervals for three-level MCRTs.

Model Number	HLM	Standardized Noncentrality Parameter (λ)	MDESD and 100*(1-α)% Confidence Interval	Degree of Freedom (v)
MCRT3-2R-1	L1: $Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{M}_{\mathit{\text{ijk}}}^{(1)}+{\pi }_{2\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}},$ $e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|M,X}^2).$ L2: ${\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}$ ${\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}+{\beta }_{11k}{T}_{\mathit{\text{jk}}}+r_{1\mathit{\text{jk}}}$ ${\pi }_{2\mathit{\text{jk}}}={\beta }_{20k}$ $\left(\begin{array}{c}{r}_{0\mathit{\text{jk}}}\\ r_{1\mathit{\text{jk}}}\end{array}\right)\sim N\left[\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}{\tau }_{00|T,W}^2& {\tau }_{01|T,W}\\ & {\tau }_{11|T}^2\end{array}\right)\right].$ L3: ${\beta }_{00k}={\gamma }_{000}+u_{00k}$ ${\beta }_{01k}={\gamma }_{010}+u_{01k}$ ${\beta }_{02k}={\gamma }_{020}$ ${\beta }_{10k}={\gamma }_{100}+u_{10k}$ ${\beta }_{11k}={\gamma }_{110}+u_{11k}$ ${\beta }_{20k}={\gamma }_{200}$ $\left(\begin{array}{c}\begin{array}{c}{u}_{00k}\\ u_{01k}\\ u_{10k}\end{array}\\ u_{11k}\end{array}\right)\sim N\left[\left(\begin{array}{c}\begin{array}{c}0\\ 0\\ 0\end{array}\\ 0\end{array}\right),\left(\begin{array}{cccc}{\tau }_{0000}^2& {\tau }_{0001}& {\tau }_{0010}& {\tau }_{0011}\\ & {\tau }_{0101}^2& {\tau }_{0110}& {\tau }_{0111}\\ & & {\tau }_{1010}^2& {\tau }_{1011}\\ & & & {\tau }_{1111}^2\end{array}\right)\right].$	Binary Moderator: ${\widehat{\delta }}_{1b}/\sqrt{\frac{{\omega }_{3T{M}^{(1)}}^2}{K}+\frac{{\omega }_{2M^{(1)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}$ Continuous Moderator: ${\widehat{\delta }}_{1c}/\sqrt{\frac{{\omega }_{3T{M}^{(1)}}^2}{K}+\frac{{\omega }_{2M^{(1)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	Binary Moderator: $M_v\sqrt{\frac{{\omega }_{3T{M}^{(1)}}^2}{K}+\frac{{\omega }_{2M^{(1)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\omega }_{3T{M}^{(1)}}^2}{K}+\frac{{\omega }_{2M^{(1)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}$ Continuous Moderator: $M_v\sqrt{\frac{{\omega }_{3T{M}^{(1)}}^2}{K}+\frac{{\omega }_{2M^{(1)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\omega }_{3T{M}^{(1)}}^2}{K}+\frac{{\omega }_{2M^{(1)}}^2}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	K − 1
MCRT3-2R-2	L1: $Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}}$, $e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|X}^2)$. L2: ${\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{T}_{\mathit{\text{jk}}}{M}_{\mathit{\text{jk}}}^{(2)}+{\beta }_{03k}{M}_{\mathit{\text{jk}}}^{(2)}+{\beta }_{04k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}$, $r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,M,W}^2$), ${\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}$, L3: ${\beta }_{00k}={\gamma }_{000}+u_{00k}$ ${\beta }_{01k}={\gamma }_{010}+u_{01k}$ ${\beta }_{02k}={\gamma }_{020}+u_{02k}$ ${\beta }_{03k}={\gamma }_{030}$ ${\beta }_{04k}={\gamma }_{040}$ ${\beta }_{10k}={\gamma }_{100}$ $\left(\begin{array}{c}{u}_{00k}\\ u_{01k}\\ u_{02k}\end{array}\right)\sim N\left[\left(\begin{array}{c}0\\ 0\\ 0\end{array}\right),\left(\begin{array}{ccc}{\tau }_{0000}^2& {\tau }_{0001}& {\tau }_{0002}\\ & {\tau }_{0101}^2& {\tau }_{0102}\\ & & {\tau }_{0202}^2\end{array}\right)\right]$.	Binary Moderator: ${\widehat{\delta }}_{2b}/\sqrt{\frac{{\omega }_{3T{M}^{(2)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}$ Continuous Moderator: ${\widehat{\delta }}_{2c}/\sqrt{\frac{{\omega }_{3T{M}^{(2)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	Binary Moderator: $M_v\sqrt{\frac{{\omega }_{3T{M}^{(2)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\omega }_{3T{M}^{(2)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}$ Continuous Moderator: $M_v\sqrt{\frac{{\omega }_{3T{M}^{(2)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\omega }_{3T{M}^{(2)}}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	K − 1
MCRT3-2R-3	L1: $Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}}$, $e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|X}^2)$. L2: ${\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}$, $r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,W}^2$), ${\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}$, L3: ${\beta }_{00k}={\gamma }_{000}+{\gamma }_{001}{M}_k^{\left(3\right)}+u_{00k}$ ${\beta }_{01k}={\gamma }_{010}+{\gamma }_{011}{M}_k^{\left(3\right)}+u_{01k}$ ${\beta }_{02k}={\gamma }_{020}$ ${\beta }_{10k}={\gamma }_{100}$ $\left(\begin{array}{c}{u}_{00k}\\ u_{01k}\end{array}\right)\sim N\left[\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}{\tau }_{0000|M}^2& {\tau }_{0001|M}\\ & {\tau }_{0101|M}^2\end{array}\right)\right]$.	Binary Moderator: ${\widehat{\delta }}_{3b}/\sqrt{\frac{{\omega }_{3T}^2-{\widehat{\delta }}_{3b}^2{Q}_3(1-Q_3)}{K{Q}_3(1-Q_3)}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}}$ Continuous Moderator: ${\widehat{\delta }}_{3c}/\sqrt{\frac{{\omega }_{3T}^2-{\widehat{\delta }}_{3c}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	Binary Moderator: $M_v\sqrt{\left(\frac{{\omega }_{3T}^2}{K{Q}_3(1-Q_3)}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}\right)/(1+\frac{M_v^2}{K})}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\left(\frac{{\omega }_{3T}^2}{K{Q}_3(1-Q_3)}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}\right)/(1+\frac{M_v^2}{K})}$ Continuous Moderator: $M_v\sqrt{\left(\frac{{\omega }_{3T}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}\right)/(1+\frac{M_v^2}{K})}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\left(\frac{{\omega }_{3T}^2}{K}+\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}\right)/(1+\frac{M_v^2}{K})}$	K − 2
MCRT3-2N-1	L1: $Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{M}_{\mathit{\text{ijk}}}^{(1)}+{\pi }_{2\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}}$, $e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|M,X}^2)$. L2: ${\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}$ ${\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}+{\beta }_{11k}{T}_{\mathit{\text{jk}}}$ ${\pi }_{2\mathit{\text{jk}}}={\beta }_{20k}$ $r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,W}^2)$. L3: ${\beta }_{00k}={\gamma }_{000}+u_{00k}$ ${\beta }_{01k}={\gamma }_{010}$ ${\beta }_{02k}={\gamma }_{020}$ ${\beta }_{10k}={\gamma }_{100}$ ${\beta }_{11k}={\gamma }_{110}$ ${\beta }_{20k}={\gamma }_{200}$ $u_{00k}\sim N(0,{\tau }_{0000}^2)$	Binary Moderator: ${\widehat{\delta }}_{1b}/\sqrt{\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}$ Continuous Moderator: ${\widehat{\delta }}_{1c}/\sqrt{\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	Binary Moderator: $M_v\sqrt{\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_1(1-Q_1)\mathit{\text{KJn}}}}$ Continuous Moderator: $M_v\sqrt{\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	KJ(n-1)-3
MCRT3-2N-2	L: $Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}}$, $e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|X}^2)$. L2: ${\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{T}_{\mathit{\text{jk}}}{M}_{\mathit{\text{jk}}}^{(2)}+{\beta }_{03k}{M}_{\mathit{\text{jk}}}^{(2)}+{\beta }_{04k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}$, $r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,M,W}^2$), ${\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}$, L3: ${\beta }_{00k}={\gamma }_{000}+u_{00k}$ ${\beta }_{01k}={\gamma }_{010}$ ${\beta }_{02k}={\gamma }_{020}$ ${\beta }_{03k}={\gamma }_{030}$ ${\beta }_{04k}={\gamma }_{040}$ ${\beta }_{10k}={\gamma }_{100}$ $u_{00k}\sim N(0,{\tau }_{0000}^2)$	Binary Moderator: ${\widehat{\delta }}_{2b}/\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}$ Continuous Moderator: ${\widehat{\delta }}_{2c}/\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	Binary Moderator: $M_v\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_2(1-Q_2)\mathit{\text{KJn}}}}$ Continuous Moderator: $M_v\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	K(J-1)-4
MCRT3-2N-3	L1: $Y_{\mathit{\text{ijk}}}={\pi }_{0\mathit{\text{jk}}}+{\pi }_{1\mathit{\text{jk}}}{X}_{\mathit{\text{ijk}}}+e_{\mathit{\text{ijk}}}$, $e_{\mathit{\text{ijk}}}\sim N(0,{\sigma }_{1|X}^2)$. L2: ${\pi }_{0\mathit{\text{jk}}}={\beta }_{00k}+{\beta }_{01k}{T}_{\mathit{\text{jk}}}+{\beta }_{02k}{W}_{\mathit{\text{jk}}}+r_{0\mathit{\text{jk}}}$, $r_{0\mathit{\text{jk}}}\sim N(0,{\tau }_{00|T,W}^2$), ${\pi }_{1\mathit{\text{jk}}}={\beta }_{10k}$, L3: ${\beta }_{00k}={\gamma }_{000}+{\gamma }_{001}{M}_k^{\left(3\right)}+u_{00k}$ ${\beta }_{01k}={\gamma }_{010}+{\gamma }_{011}{M}_k^{\left(3\right)}$ ${\beta }_{02k}={\gamma }_{020}$ ${\beta }_{10k}={\gamma }_{100}$ $u_{00k}\sim N(0,{\tau }_{0000|M}^2)$	Binary Moderator: ${\widehat{\delta }}_{3b}/\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}}$ Continuous Moderator: ${\widehat{\delta }}_{3c}/\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	Binary Moderator: $M_v\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)Q_3(1-Q_3)\mathit{\text{KJn}}}}$ Continuous Moderator: $M_v\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$ ${(M}_v\pm t_{\alpha /2})\sqrt{\frac{{\rho }_2(1-R_2^2)}{P(1-P)\mathit{\text{KJ}}}+\frac{(1-{\rho }_3-{\rho }_2)(1-R_1^2)}{P(1-P)\mathit{\text{KJn}}}}$	K(J-1)-3

Note. MCRT3-2N-1, MCRT3-2N-2, and MCRT3-2N-3 stand for three-level MCRTs where treatment is at level 2 with a level-1, -2, -3 moderator with nonrandomly varying slopes, respectively. MCRT3-2R-1, MCRT3-2R-2, and MCRT3-2R-3 stand for three-level MRTs where treatment is at level 2 with a level-1, -2, -3 moderator with random slopes, respectively.

The nonrandomly varying slope model

A nonrandomly varying slope model assumes that the moderator effect does not randomly varies across sites when the moderator is at level 1 or 2, and/or the treatment effect does not randomly vary across sites after controlling for the level-3 moderator. An approach similar to that of the random slope model can be used for deriving formulas for the power and the MDESD for the designs in Table 1. For example, for Model MCRT3-2N-1, where the treatment effect is assumed to nonrandomly varying, i.e., only varying by the moderator but not randomly varying across schools, and the coefficient of the level-1 moderator ($M_{\mathit{\text{ijk}}}^{(1)}$) is not randomly varying across level-2 clusters, hence, only the third component under the square root in Expressions 4 & 6 is left for the standard error of the moderator effect estimate (${\widehat{\gamma }}_{110}$).

Similarly, the moderation effect does not randomly vary across sites for Model MCRT3-2N-2 and the treatment effect does not randomly vary across sites after controlling for level-3 moderator for MCRT3-2N-3, the standard error of the moderator effect estimate is only associated with the second and third components under the square root in Expressions 10, 11, 15, and 16. It results in the same power and MDESD formulas for MCRT3-2N-2 and MCRT3-2N-3 (Table 2). All the formulas for power and MDESD for the nonrandomly varying slope model in three-level MCRTs are presented in Table 2.

Monte Carlo simulations

We conducted Monte Carlo simulations to examine whether the standard error and power formulas that we derived were consistent with the simulated results. The procedures for the Monte Carlo simulation are below:

1. We generated data using six HLMs in Table 2. For each model, we generated data for a continuous and a binary moderator separately, and with a nonzero and zero moderator effect separately. Hence, there were 24 scenarios: 3 (levels of moderators) × 2 (random and nonrandomly varying slopes) × 2 (scales of the moderator: continuous and binary) × 2 (nonzero and zero moderator effect).

2. We used SAS PROC MIXED to analyze the datasets. We estimated the unconditional ICCs at school and teacher levels using the unconditional HLMs. We calculated the moderator effect, the standardized effect variability of the moderation across sites for level-1 moderator, ${\omega }_{{3\mathit{\text{TM}}}^{(1)}}^2,$ the standardized effect variability of the moderation across sites for level-2 moderator, ${\omega }_{{3\mathit{\text{TM}}}^{(2)}}^2,$ the standardized effect variability of level-1 moderator across level-2 units, ${\omega }_{{2M}^{(1)}}^2,$ and the proportions of variances at level 1 and level 2 explained by covariates ($R_1^2$ and $R_2^2$) using the same estimation models as the models for generating data. The standardized variability of the treatment effect across sites for level-3 moderator, ${\omega }_{3T}^2,$ was estimated using the models that only included the treatment variable.

3. The moderator effect was standardized to the standardized mean difference for the binary moderators or the standardized coefficient for the continuous moderators; a p-value of the moderator effect that is less than 0.05 was coded a rejection of the null hypothesis of no moderation.

4. We replicated Steps (1) to (3) 2,000 times and calculated the means of the moderator effect size and the other parameters; The standard deviation of 2,000 moderator effect sizes served as the standard error estimate based on the empirical distribution of the moderator effect; We also calculated the standard error based on our formulas, and constructed the 95% confidence interval (CI) for each point estimate; we calculate the absolute difference and relative difference between the standard errors based on our formulas and that from the empirical distribution; we calculate the coverage rate of the 95% CI as the percentage of the 95% CI based on our formulas covering the true moderator effect. The proportion of times the null was rejected across the 2,000 replications estimated the Type I error rate when the moderation effect was set to 0 and the empirical power when the moderation effect was not set as 0; We compared the power and Type I error rate calculated from our derived formulas with those estimated from simulation.

The results provided evidence of the close correspondence on standard errors and power (or Type I error) between our formulas and the empirical distribution from the simulation. For example, in all scenarios the absolute difference and relative difference between the SE based on the empirical distribution of the moderator effect estimates and SE calculated from our formulas range from 0.000 to 0.005 and from −6.1% to 0.0%, respectively. The coverage rate of the 95% confidence interval (CI) ranges from 0.94 to 0.97. The absolute difference between the Type I error calculated from the formulas and that estimated from simulation ranges from 0.001 to 0.012; The absolute difference between the power calculated from the formulas and that estimated from simulation ranges from 0.004 to 0.043. Figure 2 presents the plots of power of levels-1, −2, and −3 moderators from simulation and formulas varying by the scales of moderators (continuous and binary) and slopes (random or nonrandom). Black circles represent power from simulation, and red triangles represent power from formulas.

Figure 2. Power from simulation and formulas.

Note. Simulation results were based on 2,000 replications. Under the assumptions: The intraclass correlation coefficients at Level-2 and 3: ${\rho }_2$ = 0.1, and ${\rho }_3$ = 0.2; The proportions of variances at level 1 and level 2 explained by covariates: $R_1^2$ = $R_2^2$ = 0.50; The proportion of clusters assigned to the treatment group, P = 0.5; $Q_1$ = $Q_2$ = $Q_3$ = 0.5 for binary moderators. For random slope models, the standardized effect variability of the moderation across sites for Level-1 moderator (${\omega }_{{3\mathit{\text{TM}}}^{(1)}}^2$) and Level-2 moderator (${\omega }_{{3\mathit{\text{TM}}}^{(2)}}^2$) are 0.05, the standardized variability of the treatment effect across sites for Level-3 moderator (${\omega }_{3T}^2$) is 0.09, the standardized effect variability of Level-1 moderator across Level-2 units (${\omega }_{{2M}^{(1)}}^2$) is 0.05, the moderator effect size = 0.20, sample size per level-2 unit (n) is 20, sample size per site (J) is 10, and total sample size of sites (K) is 20. For nonrandom slope models, the moderator effect size = 0.10, sample size per level-2 unit (n) is 20, sample size per site (J) is 4 for level 1 moderator and 10 for levels-2 and 3 moderators, and total sample size of sites (K) is 20 (except for binary level-1 moderator, K = 40).

Conclusion

This study provided a conceptual and statistical framework to guide the design and analysis of MCRTs. We derived the formulas to calculate the statistical power and MDESD with confidence intervals, and validated our formulas with Monte Carlo simulation. The results will have the potential to substantively impact our understanding of treatment effect variation by providing comprehensive tools to researchers to design rigorous multisite moderation studies. The framework and statistical formulas are expected to expand the scope and enhance the quality of evidence in examining the programs “work for whom and under what circumstances”. Some suggestions are below.

First, when we design a three-level MCRT to investigate the treatment effect heterogeneity, we have multiple options to consider: the individual or contextual factors that may moderate the treatment effect, and these moderators can be at either level-1, −2, or −3; the moderators can be either binary or continuous; the (moderated) treatment effect can be either random or nonrandomly varying. It is important to be inclusive by focusing on confirmatory hypothesis testing based on program theory as well as additional exploratory hypothesis testing of moderation. For example, a MCRT with adequate power for the main effect analysis will likely also be adequately powered to for the level-1 moderator effect analysis. Similarly, a MCRT with adequate power for the level-2 moderator effect analysis will be very likely for the main effect and level-1 moderator effect analysis with adequate power.

Second, our formulas assume that the sample size n is same across level-2 clusters and j is the same across sites, $Q_1$ is the same across level-2 clusters, and P and $Q_2$ are the same across sites. In practice, it is more likely to have an unbalanced design. Dong et al. (2021) conducted a simulation in two-level MRTs and found that the power calculation based on the geometric means of these sample sizes approximated the power from the simulation very well, the geometric means performed better than the harmonic means which underestimated the actual power and the arithmetic means which overestimated the actual power. We suggest using the geometric means of the sample sizes in the calculation of power or MDESD for the unbalanced designs.

Finally, it is important to justify the parameters used for a power analysis based on the literature or pilot studies (Bloom et al., 2007; Dong et al., 2016, 2022; Hedges & Hedberg, 2007, 2013; Phelps et al., 2016). The power is sensitive to some parameters (e.g., $R_1^2$) and less sensitive to other parameters (e.g., P, $Q_1,$ $Q_2,$ and $Q_3$ when they are close to 0.5). The researchers may conduct a series of power analysis with a range of parameter values, especially for the parameter values that are less documented in the literature such as the treatment/moderator effect heterogeneity.

Hence, one direction for future research is to conduct more empirical studies to report the design parameters in three-level design such as ICCs, treatment/moderator effect heterogeneity, and meaningful size regarding the moderator effects. Furthermore, this study focused on three-level MCRTs where treatment is at level 2. The extension to three-level MRTs with treatment is at level 1 may enhance researchers’ capacity to investigate multisite moderation with level-1, −2, and −3 moderators. In addition, although we expect that the power formulas are fairly robust to violation of normality assumptions for moderators based on prior research (Dong et al., 2021), the power can be affected by unreliability in the measurement of the moderator and outcome (Kelcey, Cox, & Dong et al., 2021), additional partially nested data structures (Cox et al., 2022), and missing data. Future studies to address these aspects are also needed. Finally, based on the formulas presented in this article, creating free publicly available software packages with a tutorial would facilitate applied researchers in their design of multisite moderation studies.

Additional information

Funding

This project has been funded by the National Science Foundation [1913563, 1552535, and 1760884]. The opinions expressed herein are those of the authors and not the funding agency.