Expert Knowledge Elicitation: Subjective but Scientific

Figures & data

Table 1 Average probability judgments in the Muslims exercise.

	2 million first	8 million first
$P(M>2$ million$)$	0.696	0.804
$P(M>8$ million$)$	0.303	0.403

Table 2 Average bounds in the taxi journey exercise.

	95% Bound first	Plausible bound first
Upper 95% bound	93 mins	67 mins
Upper plausible bound	126 mins	84 mins

Table 3 H2035 SHELF2 record, preliminaries.

Elicitation title	Horizon 2035: Physical long-term conditions
Workshop	SHELF workshop 1
Date	7th July 2014
Quantity	% change in the demand for care for physical LTCs per 1000 persons aged 85+ by 2035
Anonymity	In this record, the experts are identified by letters A, B, E, F, and G, and the Facilitator by Z
Start time	14:35
Definition	X = % change in the demand for care for physical LTCs per 1000 persons aged 85+ by 2035 relative to current. “Business as now” in terms of technology changes, changes in training, changes in organization. Ruling out radical technology changes and no major policy interventions or restructuring. Essentially, we are assuming that current trends are continuing.

Table 4 H2035 SHELF2 record, individual elicitation.

Individual elicitation	Method: Quartile
	Judgments: Z asked the experts to write down their median (M), to be obtained by consideration of a value for which there was a 50/50 chance of being more or less. If they were offered a prize for guessing whether X was above or below their M value, they should not have a preference for betting either way.
	The lower and upper quartiles (Q1 and Q3 respectively) were to be selected on the basis that the experts should judge it to be equally likely for X to be below Q1 or between Q1 and M, and that they should also judge it equally likely that X would be between M and Q3 or above Q3. In training, they had been advised that usually they would both be closer to M than to L or U and that the distance from M depends on confidence the expert has in their value for M.
	Experts were finally asked to make a coherence check that L < Q1 < M < Q3 < U and a “flip-of-the-coin” test on the true value being inside or outside the range (Q1,Q3).

Fig. 1 Distributions fitted to the judgments in Table 3.

Table 5 Individual judgments in the H2035 elicitation.

Expert	L	Q1	M	Q3	U
A	0	2.5	3	4	10
B	0	4	6	8	20
E	1	2.25	3	4	6
F	1	1.8	2	3	5
G	–10	–4	0	4	10

Table 6 H2035 SHELF2 record, group discussion.

Group discussion	Z initiated group discussion by asking B and G to explain why they thought X might be, respectively, so high or so low. B felt that diabetes is on the increase and cancer mortality is down so LTCs might be set for a substantial increase. It was also suggested that there could be an increase in social care demands for this age group because fewer families may be able to cope with the care provision.
	G said that there was far too much uncertainty to rule out very low, that is highly negative, or very high values for X. At this point, the other experts contributed their strong feelings that although X could conceivably be negative, in current conditions a decrease in LTCs in this age group was very unlikely. A also argued that a large increase was equally unlikely, simply because the numbers of LTCs in the 85+ age group was already high and it was hard to see how it could increase much further.
	There was a discussion around whether the differences in the demands might be taken up by younger age groups.
	A wide-ranging discussion continued, with A, E, and F being challenged over their distributions being much narrower, representing greater certainty about X, than those of B and G. However, no further material arguments being presented, Z drew the discussion to a close.

Table 7 H2035 SHELF2 record, group elicitation.

Group elicitation	Method: Probabilities
	Judgments: Z first asked the experts to consider RIO’s probability that X would be less than 2%, then that X would be more than 4%, and finally that X would be negative. After discussion, the following values were agreed.
	Pr(X < 2%) = 0.6
	Pr(X > 4%) = 0.2
	Pr(X < 0%) = 0.1
	For Pr(X < 2%), experts agreed that the true value is more likely to be below than above 2% and that 60% probability seems to convey this.
	Z pointed out that only one expert had originally given probability more than 0.5 to this, but the experts felt that the discussion had generally led them to give more weight to lower values of X than they had initially thought.
	Z also questioned their judgment of Pr(X < 0%) = 0.1, pointing out that G had initially given probability 0.5 to X being negative. G confirmed that he/she would give this a lower probability after the discussion, and that 0.1 is a reasonable judgment for RIO.

Table 8 Original and fitted probabilities in the H2035 case study.

	$P(X<2\%)$	$P(X>4\%)$	$P(X<0\%)$
Original judgments	0.6	0.2	0.1
Fitted distribution	0.57	0.17	0.12

Fig. 2 The final fitted distribution in the H2035 elicitation.