Indirect Effects in Sequential Mediation Models: Evaluating Methods for Hypothesis Testing and Confidence Interval Formation

Abstract

Complex mediation models, such as a two-mediator sequential model, have become more prevalent in the literature. To test an indirect effect in a two-mediator model, we conducted a large-scale Monte Carlo simulation study of the Type I error, statistical power, and confidence interval coverage rates of 10 frequentist and Bayesian confidence/credible intervals (CIs) for normally and nonnormally distributed data. The simulation included never-studied methods and conditions (e.g., Bayesian CI with flat and weakly informative prior methods, two model-based bootstrap methods, and two nonnormality conditions) as well as understudied methods (e.g., profile-likelihood, Monte Carlo with maximum likelihood standard error [MC-ML] and robust standard error [MC-Robust]). The popular BC bootstrap showed inflated Type I error rates and CI under-coverage. We recommend different methods depending on the purpose of the analysis. For testing the null hypothesis of no mediation, we recommend MC-ML, profile-likelihood, and two Bayesian methods. To report a CI, if data has a multivariate normal distribution, we recommend MC-ML, profile-likelihood, and the two Bayesian methods; otherwise, for multivariate nonnormal data we recommend the percentile bootstrap. We argue that the best method for testing hypotheses is not necessarily the best method for CI construction, which is consistent with the findings we present.

Keywords:

• Indirect effect
• confidence interval
• sequential mediation
• Bayesian credible interval

Article Information

Conflict of interest disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was partially supported by NIAAA (R01AA025539, D. Tofighi and K. Witkiewitz, PIs).

Role of the funders/sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The authors would like to thank Benjamin B. Dunford, Krannert School of Management, Purdue University, and Katie Witkiewitz, Department of Psychology, University of New Mexico, for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors' institutions is not intended and should not be inferred.

Notes

1 We will use the MC-Robust CI with Huber-White (Huber, 1967; White, 1980) standard errors (and robust covariance of the parameter estimates) to adjust for the potential non-normality of data; however, Falk (2018) used MC-Robust with robust Satorra-Bentler (2010) standard error correction.

2 Although the terms “diffuse” or “uninformative” might be more appropriate in referring to a noninformative prior in our context, we use the term “flat” prior to be consistent with the terminology used in the rstanarm package. In our context, a flat prior for a regression coefficient does not mean a uniform prior, but it is a normal distribution with the mean of 0 and standard deviation of 10.

3 To our knowledge, there is no established guideline for the number of the Monte Carlo samples in mediation analysis. We used RMediation to calculate the desired precision of the estimates of the standard errors of the indirect effect. We then conducted preliminary analyses to decide on the number of Monte Carlo samples, conservatively choosing 100,000 Monte Carlo samples to insure stable results.

4 To our knowledge, software packages such as OpenMx and lavaan do not have built-in functions to produce case residuals. Instead, these packages compute a variety of the residuals that are a function of the difference between the sample and model implied covariance between the dependent (endogenous) variables in the model.

5 Generally, we do not recommend removing outliers when robust estimators that down weight the outliers are available. To date, OpenMx and lavaan do not have an estimator robust to outliers.