How Evidence Clearinghouses Can Avoid the Winner’s Curse

Abstract

Published studies of intervention effects probably report effect sizes that are larger than the true effect size. There are probably many reasons for this, but one can be thought of as a “winner’s curse.” In this essay, I discuss evidence from two recent studies that highlight how evidence clearinghouses might inadvertently expose themselves to the winner’s curse, describe reasonable and unreasonable expectations for evidence clearinghouses, then suggest strategies clearinghouses might adopt to reduce their exposure to the winner’s curse.

Keywords:

• Evidence clearinghouse
• systematic review
• winner’s curse

This article refers to:

A Recipe for Disappointment: Policy, Effect Size, and the Winner’s Curse

Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education

Notes

1 Prior versions of the standards required that if an RCT experienced a high degree of attrition according to the WWC’s attrition algorithm, then (a) the WWC needed to be able to assess baseline balance on protocol-specified covariates for the sample of participants in the analysis, (b) imbalance on these covariates could not exceed a threshold of .25 standard deviations, and (c) if imbalance was .05 standard deviations or greater, then the researchers must have included the protocol-specified covariates in their estimation model. The change to the standards involves RCTs for which attrition and other types of compositional change are not thought to represent a large risk of bias: these studies no longer need to allow the WWC to assess balance – including the protocol-specified covariates in the estimation model is sufficient.