Empirical Evaluation of Prior Beliefs about Frequencies: Methodology and a Case Study in Congenital Heart Disease

Abstract

We consider the problem of critiquing prior beliefs concerning the distribution of a discrete random variable in the light of a sequentially obtained sample. A topical application concerns a probabilistic expert system for the diagnosis of congenital heart disease, which requires specification of a large number of conditional probabilities that are initially imprecisely estimated by a suitable “expert.” These prior beliefs may be formally updated as data become available, but it would seem essential to contrast the original expert assessments with the data obtained to quickly identify inappropriate subjective inputs. We consider both Bayes factor and significance testing techniques for such a prior/data comparison, both in nonsequential and sequential forms. The common basis as alternative standardizations of the logarithm of the predictive ordinate of the observed data is emphasised, and a Bayesian discrepancy statistic with a variety of interpretations provides a formal means of discounting the expert's judgments in the light of the data. The judgments are found to be of generally high quality, and procedures for automatic monitoring and adaptation are recommended.

Key Words:

• Bayes factor
• Dirichlet distribution
• Expert system
• Prequential assessment
• Scoring rule
• Subjective probabilities