Quantifying Replicability and Consistency in Systematic Reviews

Abstract

Systematic reviews and meta-analyses are important tools for synthesizing evidence from multiple studies. They serve to increase power and improve precision, in the same way that large studies can do, but also to establish the consistency of effects and replicability of results across studies. In this work we propose statistical tools to quantify replicability of effect signs (or directions) and their consistency. We suggest that these tools accompany the fixed-effect or random-effects meta-analysis, and we show that they convey important information for the assessment of the intervention under investigation. We motivate and demonstrate our approach and its implications by examples from systematic reviews from the Cochrane Library. Our tools make no assumptions on the distribution of the true effect sizes, so their inferential guarantees continue to hold even if the assumptions of the fixed-effect or random-effects models do not hold. We also develop a version of this tool under the fixed-effect assumption for cases where it is crucial and justified.

Keywords:

• Cochrane collaboration
• Drug discovery
• Heterogeneity
• Meta-analysis
• Partial conjunction analysis
• r-value

Supplementary Materials

An R package implementing the methods proposed in this article is now available for download at CRAN, under the name “metarep” (https://cran.r-project.org/web/packages/metarep/index.html). R-codes for generating the reported simulations and reproducing the examples are available on GitHub (https://github.com/IJaljuli/r-value). Supplemental file includes (a) proof for Proposition 6.1, and (b) results of simulations like in Section 3 for n = 4, 20 and both equal and unequal group sizes.

Additional information

Funding

Iman Jaljuli gratefully acknowledges The Baroness Ariane de Rothschild Women’s Doctoral grant. Yoav Benjamini gratefully acknowledges Joint US-NSF and US-Israel BSF [grant no. 2016746]. Orestis A. Panagiotou gratefully acknowledges U.S. Agency for Healthcare Research and Quality [grant no. 1R03HS025840] and reports personal fees from International Consulting Associates Inc. outside the submitted work. Ruth Heller gratefully acknowledges Israeli Science Foundation [grant 1049/16].