Conceptual tools for assessing experiments: some well-entrenched confusions regarding the internal/external validity distinction

Abstract

The notions of internal and external validity of an experiment, coined by Donald T. Campbell in the context of social scientific quasi-experimentation more than 50 years ago, are still central in the debates around the experimental method, both for practitioners and for philosophers of science. This paper points at the more problematic aspects of the distinction between the internal and external validity of experiments and, with a focus on the field of behavioural economics, traces the many misunderstandings that surround the internal/external dyad in the philosophical and social scientific literature.

Keywords:

• validity
• internal validity
• external validity
• experiments
• behavioral economics
• experimental economics

Notes

1. See, for example, Sullivan (2009) for an application of the categories to the understanding of experiments in neurobiology.

2. Statistical conclusion validity has been defined as the validity of inferences about the correlation between treatment and outcome, and construct validity refers to the inferences about the higher order constructs that represent sampling particulars (Shadish et al. 2002). These concepts, in particular that of construct validity, are common currency in social psychology and related fields in the social sciences. It is interesting to note that Shadish et al. have grouped together statistical conclusion and internal validity, opposing them to construct and external validity, suggesting the existence of a broader ‘internal’ and ‘external’ validity dichotomy subsuming the dyad under the four kinds validity categorization.

3. Mayo (2008, p. 637), for example, thinks of ‘it is preferable to speak of general methodological flaws or potential sources of invalidity or bias rather than keep to the (often ambiguous) external/internal validity terminology(.)’

4. To cite just one well-known example in the field of behavioural economics: the Ultimatum Game was initially devised for purposes other than the measuring of nonstandard preferences associated with altruist concerns or norm-following behaviour (Sugden 2008; Guala 2008).

5. At a minimum, this range of conditions will always include instances like those of the experimental setting, warranting, at least, that confidence in the causal character of the relationship between variables will translate into the replicability of results. At the extreme, the experimental conditions could be so rigorously defined as to include, for example, the particular date and time in which the experiment had been performed.

6. See, for example, Reiss (2008) and Cartwright (2006) for interesting appeals to discussing more openly the practical implications and potential for public intervention of much of what is produced in Economics departments. Though they emphasize the importance of the external validity of research, their ideas can be conveyed by referring to concepts such as relevance and applicability.