Is precise econometrics an illusion?¹

Abstract

Many empirical economists say that the teaching of econometrics is unbalanced, and students are not well-prepared for the serious problems they will encounter with real data. Here, the author considers the problem of noisy data, which is present in most econometric studies, but receives far too little attention. Most econometric studies are done in a world of low signal-to-noise ratios, and educated common sense suggests that we cannot expect precise results in such an environment. Sensitivity analysis shows that the apparent precision of reported econometric results is generally an illusion, because it is highly dependent on error term independence assumptions.^1,2

Keywords:

• Econometric estimates
• noisy data
• precision

JEL codes:

• B41
• C13
• C52

Notes

Notes

1 Much of this article is based on Chapter 2 of my book, Economics as Anatomy: Radical Change in Empirical Economics, Edward Elgar Publishing, 2019, and I am grateful to Edward Elgar Publishing for permission to re-use that material in this article. I am especially grateful to David Colander for suggesting the idea of this symposium, and for some incisive comments and thoughtful suggestions on an earlier draft of this article.

2 Leamer (1978) coined the term, “sinning in the basement.”

3 McCloskey and Ziliak (1996), Ziliak and McCloskey (2004, 2008).

4 By “prejudice,” Kalman is referring to assumptions that are not (and often cannot be) checked against data. Some econometricians have taken exception to Kalman’s use of the laden word, “prejudice,” in this context, but as Los (1989, 1269) points out, Isaac Newton used the very same word: “Praejudicia sunt et scientiam non pariunt.” (“They are prejudices and do not produce [good] science.”) If this scientific use of the term was good enough for Newton, then I think Kalman should be allowed to use it.

5 See Frisch (1934), and more modern statements of Frisch’s approach by Patefield (1981), Klepper and Leamer (1984) and others. See also the more demanding treatment by (Kalman 1982a, 1982b) and his associates, e.g., Los (1989).

6 The sample included only econometric studies using data relating to the actual economy. Studies of classroom or laboratory experiments were excluded, because we would expect much higher S/N ratios in this experimental context, which are not typical of studies based on data from the real economy. However, natural experiments and quasi-natural experiments using randomized control trials were not excluded.

7 I note, however, that several very influential econometricians do not entirely agree with Angrist and Pischke on the merits of RCTs—notably, Leamer (2010), Sims (2010), Keane (2010) and more recently, Deaton and Cartwright (2018).

8 Just six of the 100 articles in my sample used RCTs. As a rough “back of the envelope” calculation, therefore, I have combined the data from these six papers with similar data from four other studies based on RCTs, giving a total of 229 parameter estimates from 10 journal articles. The distribution of values for ψ in this smaller sample is very similar to table 2. For the 10 RCT studies, the median value of ψ is 0.03 (the same as table 2), and the maximum value is 0.84 (1.33 in table 2).

9 However, just because econometrics offers the power to make broad generalizations, that does not mean that it is correct for us to do so. If a small group of case studies shows a great deal of idiosyncracy, then that means idiosyncracy is a feature of the data, and in that case, it does not make sense to estimate a “one size fits all” econometric model (Swann 2019, 64–66).