Comparing Objective and Subjective Bayes Factors for the Two-Sample Comparison: The Classification Theorem in Action

ABSTRACT

Many Bayes factors have been proposed for comparing population means in two-sample (independent samples) studies. Recently, Wang and Liu presented an “objective” Bayes factor (BF) as an alternative to a “subjective” one presented by Gönen et al. Their report was evidently intended to show the superiority of their BF based on “undesirable behavior” of the latter. A wonderful aspect of Bayesian models is that they provide an opportunity to “lay all cards on the table.” What distinguishes the various BFs in the two-sample problem is the choice of priors (cards) for the model parameters. This article discusses desiderata of BFs that have been proposed, and proposes a new criterion to compare BFs, no matter whether subjectively or objectively determined. A BF may be preferred if it correctly classifies the data as coming from the correct model most often. The criterion is based on a famous result in classification theory to minimize the total probability of misclassification. This criterion is objective, easily verified by simulation, shows clearly the effects (positive or negative) of assuming particular priors, provides new insights into the appropriateness of BFs in general, and provides a new answer to the question, “Which BF is best?”

KEYWORDS:

• Decision theory
• Effect size
• Optimal classification
• Prior probability

Supplementary Material

R code files: R_Code_Figures1234 provides the complete R code for creating Figures 1, 2, 3 and 4. (.txt file). R_Code_analytical_TPM provides the R code for analytical calculation of TPMs of GBF₀ and WBF as described in the Appendix.

Effect size datasets: Data used to create Figures 3 and 4 are publicly available for download at https://osf.io/fgjvw/?action=download and http://www.ruudwetzels.com/data/effectsize_data.zip, respectively.

Additional information

Funding

Dr. Gönen's work was supported by NCI grant P30 CA008748.