Battle in the Planning Office: Field Experts versus Normative Statisticians

Abstract

Generally, rational decision‐making is conceived as arriving at a decision by a correct application of the rules of logic and statistics. If not, the conclusions are called biased. After an impressive series of experiments and tests carried out in the last few decades, the view arose that rationality is tough for all, skilled field experts not excluded. A new type of planner’s counsellor is called for: the normative statistician, the expert in reasoning with uncertainty par excellence. To unravel this view, the paper explores a specific practice of clinical decision‐making, namely Evidence‐Based Medicine. This practice is chosen, because it is very explicit about how to rationalize practice. The paper shows that whether a decision‐making process is rational cannot be assessed without taking into account the environment in which the decisions have to be taken. To be more specific, the decision to call for new evidence should be rational too. This decision and the way in which this evidence is obtained are crucial to validate the base rates. Rationality should be model‐based, which means that not only the isolated decision‐making process should take a Bayesian updating process as its norm, but should also model the acquisition of evidence (priors and tests results) as a rational process.

Keywords:

• Expert
• Rationality
• Bayesian Statistics
• Evidence‐Based Medicine

Acknowledgements

This paper was presented at the “Biased experts versus plain facts” session of the Annual Meeting of the Society for Social Studies of Science (November 2006), Vancouver, Canada and at the International Congress “The Social Sciences and Democracy: A Philosophy of Science Perspective” (September 2006), Ghent University, Belgium. I am grateful for the comments received from participants at both sessions. I am also grateful for the comments and encouragements from the participants of the project “The Nature of Evidence: How Well Do ‘Facts’ Travel?” of the Department of Economic History, London School of Economics. In particular I would like to thank Jacqueline Krol for providing her EBM practice‐based materials and Jon Adams for his editorial support.

Notes

[1] This concept of ‘packaging’ is borrowed from Leonelli (forthcoming), where she explores the idea that packaging is needed to make facts travel.

[2] where P: disease is present, A: disease is absent, and +: positive test result.

[3]

[4] This group consisted of the following members: P. Brill‐Edwards, J. Cairns, D. Churchill, D. Cook, A. Detsky, M. Enkin, P. Frid, M. Gerrity, H. Gerstein, J. Gibson, B. Haynes, J. Hirsch, J. Irvine, R. Jaeschke, A. Kerigan, A. Laupacis, V. Lawrence, Mark Levine, Mitchell Levine, J. Menard, V. Moyer, C. Mulrow, P. Links, A. Neville, J. Nishikawa, A. Oxman, A. Panju, D. Sackett, J. Sinclair, and P. Tugwell.

[5] A third edition by S.E. Straus, W.S. Richardson, P. Glasziou, and R.B. Haynes was published in 2005. The 2^nd edition is however used for this paper.

[6] If Pr(P) < T_trx , and

[7] This can be seen by maximizing Pr(P | +)·Pr(A | −) for Pr(P).

[8] When Pr(+ | P) ≈ Pr(− | A), then also Pr(− | P) ≈ Pr(+ | A), and thus LR(+)LR(−) ≈ 1.

[9] First edition by Alexander Mood was published in 1950. The 2^nd edition coauthored by Franklin Graybill appeared in 1963, and the 3^rd edition with Duane Boes as third author was published in 1974.

[10] A “posterior Bayes estimator” is defined as E[Y | X], where X is a random variable with probability Pr(X | Y = y), and Y a random variable with probability Pr(Y). A posterior Bayes estimator is an “unbiased” estimator of y when E[E[Y | X] | y] = y. It is shown that a posterior Bayes estimator is unbiased only when this estimator correctly estimates y with probability one. In all other cases the estimator is not unbiased.

[11] This problem raised a good deal of commotion, even among mathematicians, when discussed by vos Savant (1990). She phrased the problem as follows: “Suppose you’re on a game show, and you’re given the choice of three doors. Behind one door is a car, behind the others, goats. You pick a door, say #1, and the host, who knows what’s behind the doors, opens another door, say #3, which has a goat. He says to you, ‘Do you want to pick door #2?’ Is it to your advantage to switch your choice of doors?” (1990, 13).