Multiple Perspectives on Inference for Two Simple Statistical Scenarios

ABSTRACT

When data analysts operate within different statistical frameworks (e.g., frequentist versus Bayesian, emphasis on estimation versus emphasis on testing), how does this impact the qualitative conclusions that are drawn for real data? To study this question empirically we selected from the literature two simple scenarios—involving a comparison of two proportions and a Pearson correlation—and asked four teams of statisticians to provide a concise analysis and a qualitative interpretation of the outcome. The results showed considerable overall agreement; nevertheless, this agreement did not appear to diminish the intensity of the subsequent debate over which statistical framework is more appropriate to address the questions at hand.

KEYWORDS:

• Frequentist or Bayesian
• Multilab analysis
• Statistical paradigms
• Testing or estimation

Notes

Notes

1 In contrast to the current approach, Silberzahn et al. (in press) used a relatively complex dataset and did not emphasize the differences in interpretation caused by the adoption of dissimilar statistical paradigms.

2 The original study focused on cetirizine-induced differences in major birth defects, spontaneous abortions, and preterm deliveries. We decided to look at all birth defects, because the sample sizes were larger for this comparison and we deemed the data more interesting.

3 The dataset is made available on the OSF repository: https://osf.io/hykmz/.

4 Clarification: the original article does not provide a rationale for why several participants were excluded from the analysis.

5 The dataset is made available on the OSF repository: https://osf.io/hykmz/.

6 JASP is an open-source statistical software program with a graphical user interface that supports both frequentist and Bayesian analyses.

7 The statement is based on Jeffreys’ claims “[a]s a matter of fact I have applied my significance tests to numerous applications that have also been worked out by Fisher’s, and have not yet found a disagreement in the actual decisions reached” (Jeffreys 1939, p. 365) and “it would only be once in a blue moon that we [Fisher and Jeffreys] would disagree about the inference to be drawn in any particular case, and that in the exceptional cases we would both be a bit doubtful” (Bennett 1990, p. 162).

Additional information

Funding

This work was supported in part by a Vici grant from the Netherlands Organisation of Scientific Research awarded to EJW (016.Vici.170.083) and a Vidi grant awarded to DL (452.17.013). NvD’s and JS’s work was supported by ERC Starting Investigator Grant No. 640638.