A Strong Case for Rethinking Causal Inference

Abstract

Using compelling empirical strategies and sound reasoning, Simpson and Sims et al. ably contribute to our growing understanding of inferential errors that arise when filtering research findings using statistical significance. Drawing on the “Type M” (magnitude) and “Type S” (sign) errors described by Gelman and Carlin, both articles demonstrate that large Type M errors almost certainly exist when applying the statistical-significance filter in the field of education. In this comment, I discuss the pros and cons of four of the more actionable suggestions offered by the authors of both articles in their discussion sections. I also provide my own recommendations for drawing inferences about causal relationships that I believe will empower researchers and decision makers to more productively use evidence while avoiding the types of inferential mistakes examined by Sims et al. and Simpson.

Keywords:

• Bayesian inference
• null hypothesis significance testing
• randomized controlled trials

This article refers to:

A Recipe for Disappointment: Policy, Effect Size, and the Winner’s Curse

Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education

Notes

1 Based on formulas from Schochet (2008).

2 Figure 1 calculations are based on the formula $\mathit{MDES}=2.8*\sqrt{\frac{4}{n}}$