From theories to models to predictions: A Bayesian model comparison approach

ABSTRACT

A key goal in research is to use data to assess competing hypotheses or theories. An alternative to the conventional significance testing is Bayesian model comparison. The main idea is that competing theories are represented by statistical models. In the Bayesian framework, these models then yield predictions about data even before the data are seen. How well the data match the predictions under competing models may be calculated, and the ratio of these matches – the Bayes factor – is used to assess the evidence for one model compared to another. We illustrate the process of going from theories to models and to predictions in the context of two hypothetical examples about how exposure to media affects attitudes toward refugees.

KEYWORDS:

• Bayesian analysis
• Bayesian model comparison
• Bayes factors
• Bayesian ANOVA

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Jeffrey Rouder (Ph.D.) is the Jean Claude Falmagne Chair of Mathematical Psychology at the University of California, Irvine. He develops targeted statistical and mathematical models to study perception and cognition.

Julia Haaf (M.Sc.) is a Ph.D. student at the University of Missouri. Her research focuses on the development of Bayesian models to account for individual differences in cognition.

Frederik Aust (Dipl.-Psych, University of Cologne) is a Ph.D. student in the Department of Psychology. His research focuses on formal models of learning and memory as well as research methodology.

ORCID

Jeffrey N. Rouder http://orcid.org/0000-0003-2023-3891

Julia M. Haaf http://orcid.org/0000-0001-5122-706X

Frederik Aust http://orcid.org/0000-0003-4900-788X

Notes

1. The density of the positive-half normal with mean µ and variance b² is given by where Φ is the CDF of a standard normal. In practice, we set µ to 0 and b to 0.7 make this specification appropriate for Likert scale ratings.

2. More technically, the predictions are the integral f(Y |θ)π(θ)dθ where f(Y |θ) is the probability density of observations conditional on parameter values and π(θ) is the the probability density of the parameters.

3. Throughout this paper, we used R (3.4.0; R Core Team, 2016) and the R-packages afex (0.18.0; Singmann, Bolker, Westfall, & Aust, 2016), BayesFactor (0.9.12.2; Morey & Rouder, 2015), cowplot (0.8.0; Wilke, 2016), dplyr (0.5.0; Wickham & Francois, 2016), ggplot2 (2.2.1; Wickham, 2009), Hmisc (4.0.3; Harrell & Dupont, 2016), msm (1.6.4; Jackson, 2011), mvtnorm (1.0.6; Genz & Bretz, 2009; Wilhelm & Manjunath, 2015), papaja (0.1.0.9492; Aust & Barth, 2016), shape (1.4.2; Soetaert, 2014), spatialfil (0.15; Dinapoli & Gatta, 2015), and tmvtnorm (1.4.10; Wilhelm & Manjunath, 2015) for all our analyses and figures.

4. The notation is expanded as follows. Let u_j be an indicator of whether the jth participant is conservative or liberal, with u_j = 1 if a participant is conservative and u_j = 0 if a participant is liberal, respectively. Likewise, let w_i be an indicator of the condition with w_i = −1/2 and w_i = 1/2 for control and refugee-plight story condition, respectively. We model the data with four parameters: an overall mean, µ, an overall main effect of political orientation, η, an effect of story condition for conservatives, α, and an effect of story condition for liberals, β. With these parameters, a model on data may be expressed as:

The first term, µ, is the grand mean; the second term, ()η, takes on values of η/2 when the person is conservative, and −η/2 when the person is liberal. The third term, u_jw_iα, only comes into play for conservatives, and it is −α/2 for the control story and α/2 for the refugee story. The last term is the analogous term for liberals.