Bias neglect: A blind spot in the evaluation of scientific results

Abstract

Experimenter bias occurs when scientists' hypotheses influence their results, even if involuntarily. Meta-analyses have suggested that in some domains, such as psychology, up to a third of the studies could be unreliable due to such biases. A series of experiments demonstrates that while people are aware of the possibility that scientists can be more biased when the conclusions of their experiments fit their initial hypotheses, they robustly fail to appreciate that they should also be more sceptical of such results. This is true even when participants read descriptions of studies that have been shown to be biased. Moreover, participants take other sources of bias—such as financial incentives—into account, showing that this bias neglect may be specific to theory-driven hypothesis testing. In combination with a common style of scientific reporting, bias neglect could lead the public to accept premature conclusions.

Keywords:

• Experimenter bias
• Scientific reasoning
• Decision making
• Epistemic vigilance

We thank Alex Shaw, Angie Johnston, Joshua Knobe, Matthew Fisher, and Frank Keil for helpful comments. This study was supported by the Center for the Experiment of Mind in Nature (Oslo).

Notes

¹ We asked the “bias” question first in order to make the perception of bias salient and thus to give participants every chance to use this information. We are of course aware that this introduces the possibility of order effects, and we therefore also counterbalance the order of questions in subsequent studies below.