Meta-analysis

ABSTRACT

The sheer volume of available research and shifts toward evidence-based practice has led researchers and practitioners in sport and exercise psychology to rely increasingly on meta-analyses to summarize current knowledge, provide future research directions, and inform policy and practice. These issues highlight the imperative of precision and integrity in the conduct of meta-analyses in the discipline. This review provides a summary of meta-analytic methods relevant to sport and exercise psychology, identifies important issues and advances in meta-analytic methods, and provides best practice guidelines for meta-analysts to consider when synthesizing research in the discipline. In Part I, I provide an overview of the basic principles of meta-analysis and direct readers to accessible, non-technical treatments of the topic. In Part II, I introduce several key issues in meta-analysis and summarize the latest advances in each: effective assessment of heterogeneity; testing for moderators; dealing with dependency; evaluating publication bias and tracking down ‘fugitive literature’; and assessing sample size in meta-analysis. I also cover two emerging topics: testing theories using meta-analysis and open science and transparency practices in meta-analysis. I conclude the discussion of each issue by providing best practice guidelines, and refer the reader to further accessible texts to augment knowledge and understanding.

KEYWORDS:

• Research synthesis
• publication bias
• multi-level meta-analysis
• meta-analytic structural equation modeling
• open science

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Researchers interested in evaluating effects across a research literature have a number of methods at their disposal such as narrative review, systematic review with or without descriptive effect sizes from a meta-analysis, scoping review, and realist review. Decisions on which method to select is dependent on a number of factors including the goal of the review and the form of the available literature. An in-depth evaluation of the process involved in deciding whether or not a meta-analysis is appropriate and feasible for a review is beyond the scope of this article. Interested readers are directed to sources that provide greater detail on the considerations involved (e.g., Cooper et al., 2019a), and the ‘big picture’ methods and analysis articles in the current special issue.

2 Effect sizes reported in primary studies are likely to vary in the metric used, or the effect size to be used in the analysis may need to be computed from available data. In practice, the meta-analyst usually selects the effect size metric to use for the meta-analysis, and applies statistical formulae to convert available data and effect size information into the selected metric prior to analysis.

3 In some instances, researchers might opt to focus their analysis only on studies adopting a particular study design (e.g., randomized controlled trials, RCTs), rather than use it as a moderator. Such decisions, however, need to be clearly justified a priori when deciding on inclusion/exclusion criteria (e.g., the researcher might argue that RCTs have the most evidential value) as to omit studies would mean losing data on the effect size of interest.

4 A caveat when conducting meta-regression moderator analyses with categorical moderators is that results can be difficult to interpret when there is more than two categories or a combination of categorical moderators is used. It is also possible that it may not be possible to allocate all studies to a category in these kinds of moderator producing empty cells, which, when deleted listwise from a data set, reduces the power of the analysis.

5 It is important to note that in regression methods used to detect bias in meta-analysis the standard error of each effect size is naturally correlated with the effect size itself (Pustejovsky & Rodgers, 2019). This dependency may increase the likelihood of concluding that bias is present. Researchers have therefore recommended using alternative measures of precision to the standard error in these analyses such as the inverse sample size (e.g., Macaskill et al., 2001; Pustejovsky & Rodgers, 2019). For a more detailed explanation, see Rodgers and Pustejovsky (2021).