The (Non)Comparability of the Correlation Effect Size Across Different Measurement Procedures: A Challenge to Meta-Analysis as a Tool for Identifying “Evidence Based Practices”

Abstract

Meta-analysis is becoming a principal tool for research synthesis and for the identification and justification of evidence based practices. A fundamental assumption in meta-analysis is that effect sizes based upon different measures are comparable. Recent work has challenged this assumption in the case of the standardized mean difference. In this article it is shown that population universe (true) score level correlation effect sizes, for the relationship between two constructs A and B, based upon different measures will be comparable only if construct validity invariance holds across the measures used to make inferences to A and the measures used to make inferences to B. The results of a simulation study are also reported which show that the results of a meta-analysis may be significantly and adversely affected by violations of construct validity invariance. Finally, it is concluded that the theoretical results obtained in this article, and the results of the simulation study, combine to suggest that the role of meta-analysis in the synthesis of social work research, and in the identification of evidence based practices, be de-emphasized until important questions about the sensitivity of meta-analysis to violations of construct validity invariance are answered.

KEYWORDS:

• Meta-analysis
• effect sizes
• construct validity invariance
• correlation effect size invariance