A history of statistical methods in experimental economics

Abstract

Statistics is a minor topic in historical and methodological writings on experimental economics. This article aims to address this lacuna. To do so, we conduct a quantitative analysis of papers published in the 1970–2010 period. We also provide qualitative insights through comparisons with econometrics and psychology. Our results reveal a significant change in experimental economics’s statistical methods, namely an evolution from purely descriptive methods to more sophisticated and standardized techniques. We highlight that, by contrast with psychology and econometrics, this evolution was not accompanied with explicit methodological discussions about the role of statistics in empirical research.

Keywords:

• Experimental economics
• statistics
• econometrics
• history of economic thought
• methodology

JEL CODES:

• B20
• C83
• A14
• C90

Acknowledgements

We gratefully acknowledge comments from the two anonymous referees. We also thank Andrej Svorenčík, Roselinde Kessels, Agnès Labrousse, Stéphane Longuet, Nikolay Nenovsky and Jaime Marques Pereira for commenting on the paper's earlier versions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 Two exceptions are a chapter (on “noise and variability in experimental data”) in Bardsley’s (2010) book and a book by Moffatt (2015) entitled Experimetrics. The latter is discussed in the last section of this paper.

2 Both Hoover (2006) and Qin (2013) systematically use a similar distinction between technical discussions of methods and methodological discussions in their methodological and historical accounts of econometrics. McCloskey (1998) and Ross and Kincaid (2009, 10) use a similar distinction regarding economics more generally – although Ross and Kincaid argue that the distinction forms two intuitive poles of a continuum instead of a strict dichotomy.

3 The first chapter is an introduction by Al Roth.

4 Yet technically speaking, ANOVA is very similar to linear regression. Statisticians frequently consider the usual ANOVA set-up as a special case of OLS regression (Gelman 2005). But we chose to classify ANOVA as statistical testing from a methodological perspective, because it is most commonly used to test the difference between the means of several groups, and is thus analogous to a multiple t-test. Also, our classification is based on the idea of a general evolution in experimental economics' statistical methodology, that goes from descriptive statistics to structural modeling. Regression is regarded as a step toward structural modeling; it relies indeed on an equation to estimate, which suggests a plausible theoretical model. The crucial difference between ANOVA and regression in this gard is that basic applications of ANOVA mention F-stats and p-values only (ANOVA tables are rare) and do not provide a general model to estimate. This probably explains why ANOVA is relatively rare in economics, while it has been so important in experimental psychology (Rucci and Tweney 1980). Similarly, we found that ANOVA was frequent in individual DM in psychology and almost absent in the other economic subfields.

5 In our initial classification, maximum-likelihood estimates were required for articles to be classified as structural modeling. But some articles apply linear OLS regression and simulation methods to estimate theoretical models (e.g., Fischbacher and Gächter 2010) Such studies seem to correspond to our definition of structural modeling. We changed therefore our criteria for inclusion in the structural modeling category. The main criterion is the purpose of the regression. If regression techniques are applied to test a previous hypothesis, or merely provide support for an observation, the article was classified in the regression category. Alternatively, when the equation to estimate in the regression is considered as a general model (for instance, utility functions, bid functions in auction experiments, learning models), the article was classified in the structural modeling category. Our classification might be criticized because we included in the structural modeling category some articles using very basic statistical tools, for instance linear OLS regression of bid functions in auction experiments. We insist that all of our data and justifications are available on request.

6 We had not enough observations for each subfield and each method (some n_i,j < 5) to perform the parametric Chi-square test. Another reason for choosing non-parametric tests was that such tests are distribution-free and we did not want to make specific assumptions about the sampling distribution of methods in each subfield (in particular, there might be subgroups of experiments using very similar methods, yielding skewed and non-normal distributions).

7 The exchange started with McCloskey arguing that “none of the statistical tests of significance that you, Betsy, et al. have done in ‘Preferences, Property Rights and Anonymity in Bargaining Games’ [article subsequently published as Hoffman et al. 1996] is sensible rhetoric” (McCloskey to Smith, November 8, 1993, Box 18, Folder Sept–Dec 1993 of Smith's papers at the Rubenstein library, Duke University). Despite their initial disagreement, both Smith and McCloskey consider that their scientific views have converged in the past ten years or so (Smith and McCloskey, personal communication to the authors, 27/05/2017).

8 There is no contemporary handbook that mimics the structure of Kagel and Roth’s (1995) in terms of types of experiments, including their second volume of the handbook (Kagel and Roth 2015), in which only one chapter corresponds to one type of experiment that was also a chapter in the first volume: Kagel and Levin’s (2015) chapter on auctions, which we selected as the survey for auctions for the period covered in this section. Note that the previous section had similar sampling sources (chapters taken from the same handbook) while this one has different sampling sources (various books, chapters and articles). If we had observed a weak tendency toward homogenisation of statistical methods, it could have been due to the variation in our sampling procedures. Given the relatively strong tendency that we observe, we do not believe that the variation in our sampling procedure has created a problem.

9 The reason was that there are actually very few behavioral economics’ articles published in psychological journals after 1995.

10 The same significance level was used in the previous section. We found significant differences for auctions, asset markets and IO at the 5% significance level only. T-stats from Cramer von Mises test are clearly decreasing between the two periods, thus confirming this process of homogenization in statistical methods across domains.

11 Moffatt is also well in line with the tendency identified between this and the previous section because he is a strong supporter of structural modeling