The Experimenter-as-Fixed-Effect Fallacy

ABSTRACT

Few studies in psychology are designed to provide a valid basis for generalizing beyond the specific—fixed—sample of experimenters who conducted the study; that is, they commit the experimenter-as-fixed-effect fallacy. We conducted a model experiment to demonstrate remedies to this fallacy by (a) creating an experimenter factor with each experimenter construed as one level of the factor and by (b) statistically treating the experimenter factor as random. Results are compared, using the appropriate design and statistics, with results that would have been reported if the experimenter factor had been treated as fixed or ignored altogether. In this article, we also discuss the use of a random experimenter factor to increase generalization and to avoid failure to replicate.