Some Counterclaims Undermine Themselves in Observational Studies

Abstract

Claims based on observational studies that a treatment has certain effects are often met with counterclaims asserting that the treatment is without effect, that associations are produced by biased treatment assignment. Some counterclaims undermine themselves in the following specific sense: presuming the counterclaim to be true may strengthen the support that the original data provide for the original claim, so that the counterclaim fails in its role as a critique of the original claim. In mathematics, a proof by contradiction supposes a proposition to be true en route to proving that the proposition is false. Analogously, the supposition that a particular counterclaim is true may justify an otherwise unjustified statistical analysis, and this added analysis may interpret the original data as providing even stronger support for the original claim. More precisely, the original study is sensitive to unmeasured biases of a particular magnitude, but an analysis that supposes the counterclaim to be true may be insensitive to much larger unmeasured biases. The issues are illustrated using data from the U.S. Fatal Accident Reporting System. Supplementary materials for this article are available online.

• Design sensitivity; Segment; Sensitivity analysis; Truncated product of P-values

Additional information

Notes on contributors

Paul R. Rosenbaum

Paul R. Rosenbaum, Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104-6340 (E-mail: [email protected]). Supported by a grant from the MMS Program of the U.S. National Science Foundation.