The Effect of Noninvariance on the Estimation of the Mediated Effect in the Two-Wave Mediation Model

Abstract

The two-wave mediation model is the most suitable model for examining mediation effects in a randomized intervention and includes measures taken at pretest and posttest. When using self-report measures, the meaning of responses may change for the treatment group over the course of the intervention and result in noninvariance across groups at posttest, a phenomenon referred to as response shift. We investigate how the mediated effect would be impacted by noninvariance when using sum scores (i.e., assuming invariance). In a Monte Carlo simulation study, the magnitude and proportion of items that had noninvariant intercepts, the direction of noninvariance, number of items, effect size of the mediated effect and sample size were varied. Results showed increased Type I and Type II errors due to a biased estimate of the intervention effect on the mediator resulting from noninvariance. Thus, measurement noninvariance could lead to erroneous conclusions about the process underlying the intervention.

Keywords:

• Longitudinal mediation
• mediation
• measurement invariance
• pretest-posttest design

Notes

1 Prior to treatment, M₁ and Y₁ are assumed to be correlated to form a joint confounder of the M₂ to Y₂ relation (i.e., the b-path component of the mediated effect ab₅) via the temporal stabilities of the mediator (b₂) and the outcome (b₁) (Mayer et al., 2014; Valente and MacKinnon, 2020), and if present, via the cross-lags (b₃ and b₄). It is important to adjust for M₁ and Y₁ to adjust for potential confounder bias of the mediated effect.

2 The literature on measurement invariance methodology largely focuses on reflective measurement models (Bollen & Diamantopoulos, 2017, p. 607) in which changes in the latent construct result in changes to the observed indicators rather than the other way around. Therefore, under a reflective model, differences in any of the measurement parameters are understood as measurement noninvariance rather than indicating change in the construct. Throughout this paper, we make the explicit assumption that the measurement model for the construct is reflective. If the measurement model is not reflective, but a reflective model is fit, true change could appear as noninvariance. Therefore, we urge researchers to ensure that a reflective measurement model is justifiable for their mediator construct. Alternative measurement models include causal-formative indicators (e.g., Bollen & Bauldry, 2011; Rhemtulla, van Bork & Borsboom, 2020) and psychometric network models (e.g., Epskamp, Rhemtulla, & Borsboom, 2017).

3 For example, if the intercepts for the control group were (0.9, 0.4, 0.5, 0.5, 0.7, 0.4), and the intercepts for the treatment group were (0.9, 0.4, 0.6, 0.4, 0.7, 0.4), there is noninvariance for items 3 and 4 in different directions. However, because ${\tau }_C^{*}=3.4$ and ${\tau }_T^{*}$ = 3.4, per Equation 6, the a-path will not be biased.

4 In using multiple group model for data generation, the a-path was required to enter the model as an intercept in the regression of M₂ on M₁ because the X variable representing group is not a variable in the structural model. This intercept was then allowed to differ between groups. Thus, if the true a=0.285, the regression in the data-generating model for the treatment group was: M₂=0.285 +0 .985M₁ and for the control group M₂=0.985M₁.

5 This would apply best for situations in which it is not possible to fit a MIMIC model due to a small sample size and a large number of items. If it is not possible to fit a latent variable model to test for invariance, researchers could consider simpler techniques for testing for noninvariance, such as observed variable methods (Millsap, 2011, Ch. 8).