Expert knowledge elicitation using item response theory

ABSTRACT

In an expert knowledge elicitation exercise, experts face a carefully constructed list of questions that they answer according to their knowledge. The elicitation process concludes when a probability distribution is found that adequately captures the experts' beliefs in the light of those answers. In many situations, it is very difficult to create a set of questions that will efficiently capture the experts' knowledge, since experts might not be able to make precise probabilistic statements about the parameter of interest. We present an approach for capturing expert knowledge based on item response theory, in which a set of binary response questions is proposed to the expert, trying to capture responses directly related to the quantity of interest. As a result, the posterior distribution of the parameter of interest will represent the elicited prior distribution that does not assume any particular parametric form. The method is illustrated by a simulated example and by an application involving the elicitation of rain prophets' predictions for the rainy season in the north-east of Brazil.

Keywords:

• Subjective probability
• item response theory
• latent trait
• prior information
• nonparametric elicitation
• rain prophets

Acknowledgments

We thank the Associate Editor and the Referees for the comments and suggestions which improved considerably the presentation of our work.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

J. A. A. Andrade http://orcid.org/0000-0002-8017-7572

J. P. Gosling http://orcid.org/0000-0002-4072-3022