Calibrated Bayes Factors Should Not Be Used: A Reply to Hoijtink, van Kooten, and Hulsker

ABSTRACT

Hoijtink, Kooten, and Hulsker (2016) present a method for choosing the prior distribution for an analysis with Bayes factor that is based on controlling error rates, which they advocate as an alternative to our more subjective methods (Morey & Rouder, 2014; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011). We show that the method they advocate amounts to a simple significance test, and that the resulting Bayes factors are not interpretable. Additionally, their method fails in common circumstances, and has the potential to yield arbitrarily high Type II error rates. After critiquing their method, we outline the position on subjectivity that underlies our advocacy of Bayes factors.

KEYWORDS:

• Statistical inference; Bayesian statistics; Bayes factors; error rates

Article information

Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was supported by Grant “Bayes or Bust” from the European Research Council.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The ideas and opinions expressed herein are those of the authors alone, and endorsement by the author's institutions or the European Research Council is not intended and should not be inferred.

Notes

2 HKH present their Definition 2 with a fixed α = .05. Because.05 is arbitrary, we use a generalization of Definition 2 that allows any α.

3 In this situation, we would argue that a one-sided test with this τ is appropriate. However, the question of one-sided vs. two-sided test is irrelevant to the point we are making; to remain consistent with the reported t test and the HKH-calibrated test, we report a two-sided Bayes factor.

4 Confusingly, HKH appear to admit that their reading is unfair. They write that “[i]t has to be noted that Rouder et al. (2009) and Wagenmakers et al. (2011) either in the publications referred to in the current paper or in other publications, also note that the choice τ = 1 is to some degree arbitrary and other choices could/should be considered.” (p. 9 of the manuscript)

5 Typically, a t prior with 1 degree of freedom is used instead of the normal prior described here, and the prior scale is called r (de Vries & Morey, 2013; Morey & Rouder, 2011; Rouder et al., 2009). The prior probabilities described here are for the t prior; the prior probabilities will be slightly different for the scaled-information Bayes factor’s normal prior.