Handling effect size dependency in meta-analysis

ABSTRACT

The statistical synthesis of quantitative effects within primary studies via meta-analysis is an important analytical technique in the scientific toolkit of modern researchers. As with any scientific method or technique, knowledge of the weaknesses that might render findings limited or potentially erroneous as well as strategies by which to mitigate these biases is essential for high-quality scientific evidence. In this paper, we focus on one prevalent consideration for meta-analytical investigations, namely dependency among effects. We provide readers with a non-technical introduction to and overview of statistical solutions for handling dependent effects for their efforts to integrate evidence within primary studies. This goal is achieved via a series of seven reflective questions that scholars might consider when planning and executing a meta-analysis in which some degree of dependency among effect sizes from primary studies may exist. We also provide an example application of the recommendations with real-world data, including an analytical script that readers can adapt for their own purposes.

KEYWORDS:

• multivariate meta-analysis
• open science
• research synthesis
• robust variance estimation
• systematic review
• three-level meta-analysis

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 A detailed analysis of the philosophical foundations of meta-analysis is beyond the scope of the current article. Interested readers are referred elsewhere for critiques (Stegenga, 2009, 2011). Given the numerous subjective decisions that are applied in the design and execution of meta-analyses, we prioritise a post-positivist paradigmatic perspective.

2 Network meta-analysis is another type of multivariate approach in which analysts compare multiple interventions against a common comparator group. Interested readers are referred elsewhere for overviews of network meta-analysis (Molloy et al., 2018; Salanti, 2012).

3 Traditional univariate meta-analysis is technically multilevel in nature with participants in primary studies located at level 1 and the studies at level 2.

4 Open source programs such as Jamovi (The jamovi project, 2020) and JASP (JASP Team, 2020) have capabilities for researchers interested in conducting univariate meta-analyses with ‘point and click’ software. It is likely that multilevel meta-analysis will be incorporated in future updates to these programs (e.g., https://github.com/kylehamilton/MAJOR/blob/master/README.md).

5 Interested readers are referred elsewhere for an excellent R package and Shiny web app to visualise risk of bias (McGuinness & Higgins, 2020).

6 Ultimately, the approach one adopts to extending Egger’s test beyond the traditional two-level meta-analytic framework depends on the nature of the effect sizes adopted in one’s analysis. For example, there is an inherent association between correlations and their sampling variances, so one may prefer to use the inverse of the sample size as the predictor in the moderation analysis (https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2020-May/002086.html).