Sample Size Planning for Detecting Mediation Effects: A Power Analysis Procedure Considering Uncertainty in Effect Size Estimates

Abstract

When planning mediation studies, researchers are often interested in the sample size needed to achieve adequate power for testing mediation. Power depends on population effect sizes, which are unknown in practice. In conventional power analysis, effect size estimates, however, are often used as population values, which could result in underpowered studies. Uncertainty in effect size estimates has been considered in other sample size planning contexts (e.g., t-test, ANOVA), but has not been handled properly for planning mediation studies. In the current study, we proposed an easy-to-use sample size planning method for testing mediation with uncertainty in effect size estimates considered. We conducted simulation studies to demonstrate the impact of uncertainty in effect size estimates on power of testing mediation, and to provide sample size suggestions under different levels of uncertainty. Empirical examples were provided to illustrate the application of our method. R functions and a web application were developed to facilitate implementation.

Keywords:

• Sample size planning
• mediation analysis
• uncertainty
• effect size estimates
• power analysis

Notes

Notes

1 Other mediation effect size measures have also been proposed (e.g., Berry & Mielke, 2002; Cheung, 2009; de Heus, 2012; Fairchild, MacKinnon, Taborga, & Taylor, 2009; Kraemer, 2014; Preacher & Kelley, 2011; Wen & Fan, 2015).

2 For more detailed descriptions of the six methods, see Fritz and MacKinnon (2007).

3 See https://schoemanna.shinyapps.io/mc_power_med/.

4 However, for a given value of a and a given value of b, the power of testing ab from a testing method is fixed.

5 “Conventional” power analysis in the article means the power analysis procedure that uses the face values of effect size estimates as if they were true values.

6 Detailed data generation process is as follows: X is simulated from N(0, 1); then M from $N(\widehat{a}X,{\widehat{\sigma }}_{e_M}^2);$ and then Y from $N(\widehat{b}M+\widehat{c\prime }X,{\widehat{\sigma }}_{e_Y}^2)$ with ${\widehat{\sigma }}_{e_M}^2=1-{\widehat{a}}^2,{\widehat{\sigma }}_{e_Y}^2=1-{\widehat{c\prime }}^2-{\widehat{b}}^2-2\widehat{a}\widehat{b}\widehat{c\hspace{0.17em}\prime }.$

7 Steps 1 and 2 are related to the parametric bootstrap idea (Monte Carlo is also referred to as parametric resampling; Efron & Tibshirani, 1993). Thus, K can be seen as the number of generated parametric bootstrap samples, and larger K leads to more accurate approximations. Note that in Steps 1 and 2 of our method, the size of a generated sample, N_pilot, is for specifying the level of uncertainty in the effect size estimates, and can be different from that of the original sample used in the prior study; but in parametric bootstrap, the two are often the same.

8 Besides the joint significance test, the percentile bootstrap test is also frequently used for testing mediation. It has been shown that the percentile bootstrap test has similar satisfactory performance as the joint significance test in terms of both Type I error rates and power (e.g., Fritz & MacKinnon, 2007; MacKinnon et al., 2004). Indeed, the power distributions and the sample size suggestions we obtained using the percentile bootstrap test were close to those obtained using the joint significance test for both simulation studies. In addition, an anonymous reviewer suggested that the percentile bootstrap test lacks a unique null distribution for testing ab and thus may not be appropriate for power analysis of testing ab. Therefore, we decided to report only the results from the joint significance test in the paper to save space and avoid potential controversy. The results from the percentile bootstrap test are available upon request.

9 Skewness is calculated as ${[(K-1)/K]}^{3/2}{m}_3/m_2^{3/2},$ where K is the number of power values used to form the power distribution and $m_r=[{\sum }_{k=1}^K{(\mathit{Powe}{r}_k-\frac{1}{K}{\sum }_{k=1}^{K}Po\mathit{we}{r}_k)}^r]/K,r=2,3$ (Joanes & Gill, 1998).

10 The values in the rows of $N_{\mathit{pilot}}=\infty$ in Table 4 are slightly different from the sample sizes needed for 0.8 power provided in Table 3 of Fritz and MacKinnon (2007). This is because we considered a standardized mediation model where M and Y in Eqs. 1(1) $$Y=\mathit{aX}+e_M$$(1) and 2(2) $$Y=\mathit{bM}+c\prime X+e_Y,$$(2) have unit variances; and in Fritz and MacKinnon (2007) the residual terms e_M and e_Y in Eqs. 1(1) $$Y=\mathit{aX}+e_M$$(1) and 2(2) $$Y=\mathit{bM}+c\prime X+e_Y,$$(2) have unit variances.