Assessing rationales for inclusiveness in meta-analytic samples

Abstract

The desirability of including all relevant studies in a meta-analysis has been defended by at least six arguments: (1) Any inclusion/exclusion criteria would inevitably reflect unique biases of the individual meta-analyst; (2) The global methodological quality of studies cannot be reliably assessed; (3) There is evidence to suggest that effect size and methodological quality do not covary; (4) Flaws in the inclusive sample are likely to be distributed unsystematically; (5) Inclusiveness combined with describing the characteristics of the individual studies allows the meta-analyst to determine the effect that any number of moderating variables has on overall effect size estimates; and (6) A number of studies with nonsignificant findings cannot be retrieved, and further inclusion/exclusion criteria would exacerbate the “file drawer” problem. Each of these six rationales will be reconsidered. Results indicate that these rationales receive little empirical support, and thus deserve to be empirically assessed on prospective meta-analytic samples. We conclude that meta-analytic inclusiveness should not be thoughtlessly pursued, and that alternative approaches can be validly adopted.