The Brier Rule Is not a Good Measure of Epistemic Utility (and Other Useful Facts about Epistemic Betterness)

ABSTRACT

Measures of epistemic utility are used by formal epistemologists to make determinations of epistemic betterness among cognitive states. The Brier rule is the most popular choice (by far) among formal epistemologists for such a measure. In this paper, however, we show that the Brier rule is sometimes seriously wrong about whether one cognitive state is epistemically better than another. In particular, there are cases where an agent gets evidence that definitively eliminates a false hypothesis (and the probabilities assigned to the other hypotheses stay in the same ratios), but where the Brier rule says that things have become epistemically worse. Along the way to this ‘elimination experiment’ counter-example to the Brier rule as a measure of epistemic utility, we identify several useful monotonicity principles for epistemic betterness. We also reply to several potential objections to this counter-example.

KEYWORDS:

• Bayesianism
• Brier score
• epistemic utility
• formal epistemology
• law of likelihoods
• proper scoring rules

Notes

¹ Don Fallis and Dennis Whitcomb [2009] consider how epistemic values beyond truth might fit into the framework of epistemic consequentialism.

² For the sake of simplicity, in this paper we set aside the possibility that the different hypotheses might have different degrees of informativeness. We also set aside the possibility that the different false hypotheses might have different degrees of verisimilitude. See Graham Oddie [2014] for a discussion of formal models of verisimilitude or ‘truthlikeness’.

³ Since we are focusing here on the degree to which an agent's cognitive state gets at the truth, we might speak instead about measures of inaccuracy and about minimizing expected inaccuracy as many formal epistemologists do (e.g. Joyce [2009] and Leitgeb and Pettigrew [2010a]).

⁴ Strictly speaking, Joyce's result is even stronger than this. For measures of epistemic utility that have certain independently desirable properties (such as propriety and separability), Joyce shows that, for any credences c that are not coherent, there are coherent credences c′ that have a higher epistemic utility than c regardless of which hypothesis happens to be true (see Pettigrew [2013: 900–2]). In other words, he uses utility dominance rather than expected utility maximization to vindicate probabilism.

⁵ Goldman [1999: 115–23] argues that conditionalizing on new evidence maximizes objectively expected epistemic utility and not just subjectively expected epistemic utility. But, as we discuss in the following section, the measure of epistemic utility that he uses to prove this result is not a proper scoring rule. So, it is not an appropriate measure of epistemic utility. Moreover, Goldman's result about conditionalization does not hold for any bounded proper scoring rule [Fallis and Liddell 2002].

⁶ Note that this statement of propriety is not restricted to coherent credences. Also, some epistemologists (e.g. Oddie [1997: 539] and Joyce [2009: 276]) require that measures of epistemic utility be strictly proper scoring rules. That is, , for all r and s. All of the scoring rules under discussion in this paper are strictly proper.

⁷ Note that the following statements of these rules presuppose that credences are over a partition. See Joyce [2009: 275] for statements of these rules for credences over a Boolean algebra. We discuss the issue of partitions versus Boolean algebras below. Also, the Brier rule is often given as a measure of inaccuracy (with the sign reversed) rather than as a measure of epistemic utility, as it is here. It is sometimes referred to as the quadratic rule.

⁸ For coherent credences, this sum is minimized when a probability of 1/n is assigned to all of the hypotheses in the partition. It is maximized when a probability of 1 is assigned to one of the hypotheses.

⁹ Since they handle this term a bit differently, the two rules do give a different weight to this epistemic cost.

¹⁰ If the probability assigned to the true hypothesis and the probability assigned to a false hypothesis increase, epistemic utility may increase or decrease depending on the details of the case.

¹¹ In ‘A Strange Thing about the Brier Score’, Brian Knab and Miriam Schoenfield [2015] suggest that ‘falsity distributions don't matter.’ Lewis and Fallis [2014] argue that they do matter. But this debate is orthogonal to our concerns here. It is the specific way in which the Brier rule handles falsity distributions that we object to in this paper.

¹² Like other PSRs, the Brier rule is typically used as a method for eliciting probability estimates [Bickel 2007]. Our criticism, though, is just of the Brier rule as a measure of epistemic utility.

¹³ According to the Brier rule, u₁(r) = 0.625 and u₁(s) = 0.333. According to the logarithmic rule, u₁(r) = −0.301 and u₁(s) = −0.477. According to the spherical rule, u₁(r) = 0.817 and u₁(s) = 0.577.

¹⁴ Since the probabilities have to sum to 1 for coherent credences, the other probabilities cannot all be exactly the same. That is, if r_i is different than s_i, then some of the other probabilities in r and s have to be different as well.

¹⁵ According to the Brier rule, u₁(r) = 0.875 and u₁(s) = 0.625. According to the logarithmic rule, u₁(r) = −0.125 and u₁(s) = −0.301. According to the spherical rule, u₁(r) = 0.949 and u₁(s) = 0.817.

¹⁶ This claim is trivial for the logarithmic rule. Proofs of this claim for the Brier rule and for the spherical rule are in the Appendix. It may not be the case, though, that all PSRs endorse this principle.

¹⁷ According to the Brier rule, u₁(r) = 0.500 and u₁(s) = 0.333. According to the logarithmic rule, u₁(r) = −0.301 and u₁(s) = −0.477. According to the spherical rule, u₁(r) = 0.707 and u₁(s) = 0.577.

¹⁸ At least, it represents progress unless ‘the Bayesian agent has been so unfortunate as to assign the true hypothesis a zero prior’ [Earman 1992: 163]. Since we assume that the hypotheses under consideration are jointly exhaustive, the true hypothesis is included. In addition, in our ‘elimination experiment’ examples, the agent starts out assigning a non-zero probability to each of the hypotheses.

¹⁹ See Godfrey-Smith [2011: 1294–5] and Fallis [2015: 384–6] for further discussion of why these philosophers have themselves been misled.

²⁰ If an agent shifts from s to r on the basis of evidence e, then r_k/s_k = pr(h_k|e)/pr(h_k) = pr(e|h_k)/pr(e). So, r_i/s_i ≥ r_j/s_j if and only if pr(e|h_i) ≥ pr(e|h_j). Thus, according to Ian Hacking's ‘Law of Likelihoods’, e supports h_i at least as much as it does h_j if and only if r_i/s_i ≥ r_j/s_j.

²¹ More formally, what M3 says is that, if r_j < s_j for some j ≠ i, and there is a real number α such that αr_k = s_k for all k ≠ j, then u_i(r) > u_i(s).

²² This claim is again trivial for the logarithmic rule. A proof of this claim for the spherical rule is in the Appendix.

²³ As noted above, all other things being equal, epistemic utility on the Brier rule decreases as the total probability assigned to the false hypotheses is concentrated on fewer false hypotheses. In many cases, including our ‘elimination experiment’ counter-example, this ‘epistemic cost’ is (according to the Brier rule) enough to outweigh the epistemic benefit of an increase in the probability assigned to the true hypothesis. Knab and Schoenfield [2015] give a different example that also displays this effect. In offering an analysis of misleading evidence, Lewis and Fallis [2014] independently discuss a very similar case. However, these examples are not as obviously in conflict with scientific practice as our ‘elimination experiment’ counter-example is.

²⁴ A full Boolean algebra includes conjunctions, as well as disjunctions, of the basic hypotheses. However, since we are assuming here that the hypotheses under consideration are mutually exclusive, any conjunctions must be assigned a probability of zero.

²⁵ It is possible to modify some of these principles so that they are applicable to a full Boolean algebra of hypotheses in which more than one hypothesis can be true. For instance, we could rewrite M4 as M4*: If r_i/s_i ≥ r_j/s_j for all i and j such that h_i is true and h_j is false, then r is epistemically at least as good as s. Although this principle is true, it is far too weak for our purposes. Elimination experiments are always epistemically beneficial, but M4* does not entail this.

²⁶ Thanks to an anonymous referee for suggesting this sort of objection.

²⁷ The standard versions of the logarithmic rule and the spherical rule (as stated above) do not vindicate probabilism. Joyce [2009: 275] proposes alternative versions of these rules that do. Unfortunately, unlike the standard versions, Joyce's versions of these rules are (just like the Brier rule) subject to ‘elimination experiment’ counter-examples. (His version of the logarithmic rule does say that r = (1/3, 2/3, 0) is epistemically better than s = (1/4, 1/2, 1/4) when h₁ is true, but it says that s = (1/10, 8/10, 1/10) is epistemically better than r = (1/9, 8/9, 0) when h₁ is true.) So, vindicating probabilism while avoiding ‘elimination experiment’ counter-examples might not be trivial.

²⁸ We would like to thank Kobus Barnard, David Black, Kenny Easwaran, Branden Fitelson, Will Fleisher, Martin Frické, Peter Godfrey-Smith, Simon Goldstein, Stephen Hetherington, Terry Horgan, Jenann Ismael, James Joyce, Gerrard Liddell, Kay Mathiesen, Richard Pettigrew, Daniel Rubio, Jonah Schupbach, Elliott Sober, Julia Staffel, Dan Zelinski, and two anonymous referees for helpful feedback on earlier versions of this material. Much of this work was carried out while the first author was a visiting fellow at the Tanner Humanities Center at the University of Utah.