Statistical Guidance to Authors at Top-Ranked Journals across Scientific Disciplines

Abstract

Scientific journals may counter the misuse, misreporting, and misinterpretation of statistics by providing guidance to authors. We described the nature and prevalence of statistical guidance at 15 journals (top-ranked by Impact Factor) in each of 22 scientific disciplines across five high-level domains (N = 330 journals). The frequency of statistical guidance varied across domains (Health & Life Sciences: 122/165 journals, 74%; Multidisciplinary: 9/15 journals, 60%; Social Sciences: 8/30 journals, 27%; Physical Sciences: 21/90 journals, 23%; Formal Sciences: 0/30 journals, 0%). In one discipline (Clinical Medicine), statistical guidance was provided by all examined journals and in two disciplines (Mathematics and Computer Science) no examined journals provided statistical guidance. Of the 160 journals providing statistical guidance, 93 had a dedicated statistics section in their author instructions. The most frequently mentioned topics were confidence intervals (90 journals) and p-values (88 journals). For six “hotly debated” topics (statistical significance, p-values, Bayesian statistics, effect sizes, confidence intervals, and sample size planning/justification) journals typically offered implicit or explicit endorsement and rarely provided opposition. The heterogeneity of statistical guidance provided by top-ranked journals within and between disciplines highlights a need for further research and debate about the role journals can play in improving statistical practice.

KEYWORDS:

• Author instructions
• Journal policy
• Meta-research
• Statistics
• Statistical guidelines

1 Introduction

The validity of scientific claims often depends on the appropriate selection, implementation, reporting, and interpretation of statistical analyses. However, serious concerns have been raised about the frequent misuse, misreporting, and misinterpretation of statistical methods in several scientific disciplines (Altman 1982; Gigerenzer 2018; Nieuwenhuis, Forstmann, and Wagenmakers 2011; Sedlmeier and Gigerenzer 1989; Strasak et al. 2007; Wasserstein and Lazar 2016). Academic journals may attempt to counter these problems by providing guidance on statistical issues in their instructions to authors (Altman et al. 1983; Bailar and Mosteller 1988; Smith 2005).

It is unclear whether journal-based statistical guidance, advocacy, or regulation is helpful or detrimental, especially in the context of controversial statistical topics (Mayo 2021). There continues to be heated debate among statisticians and methodologists about fundamental statistical issues, as exemplified by the diverse commentaries in a 2019 special issue of The American Statistician (Wasserstein, Schirm, and Lazar 2019). p-values continue to be vigorously attacked (Wagenmakers 2007) and staunchly defended (Mayo 2018), and recent years have seen competing calls to either abandon (McShane et al. 2019), redefine (Benjamin et al. 2018), or justify (Lakens et al. 2018) statistical significance. Others have advocated for a so-called “New Statistics” centered on effect size estimation and confidence intervals (Cumming 2014) or promoted wider adoption of Bayesian statistics (Goodman 1999; Kruschke and Liddell 2018).

In some cases, these debates have influenced journal policy and controversial changes in policy have sometimes backfired. For example, in 2005, in an attempt to avoid the “pitfalls of traditional statistical inference,” the journal Psychological Science endorsed a novel statistic $p_{\text{rep}}$ (Killeen 2005) which was subsequently adopted by the majority of publishing authors, and then abandoned under heavy criticism a few years later (Iverson et al. 2009). In 2015, the journal Basic and Applied Social Psychology introduced an outright ban on p-values and confidence intervals “because the state of the art remains uncertain” (Trafimow and Marks 2015). An assessment of articles published in the first year after the ban argued that the absence of statistical safeguards led many authors to overstate their findings (Fricker et al. 2019). Less controversial statistical guidance may help to reduce basic statistical errors; however, beyond case studies, we are not aware of any rigorous trials assessing the impact of journal-based statistical guidance on author behavior. Some evidence suggests that more general methodological reporting guidelines, such as Consolidated Standards of Reporting Trials (CONSORT), can be modestly effective (Turner et al. 2012). Understanding the current landscape of journal-based statistical guidance will help to inform the design of future trials that assess journal policies intended to improve statistical practice.

Currently there is limited empirical data documenting the prevalence and content of statistical guidance for authors issued by scientific journals. Such data will help to guide debates about the appropriate role of journal policy in addressing statistical issues (Mayo 2021) as well as identify gaps in journal-based statistical guidance within and across scientific disciplines. Some previous research has addressed this topic and suggested that there is probably large room for improvement. For example, Schriger, Arora, and Altman (2006) reported that 39% of 166 leading medical journals provided statistical guidance to authors. Topics addressed included p-values (20%), confidence intervals (14%), statistical power (10%), multiple testing (5%), modeling (4%), sensitivity analysis (4%), and Bayesian methods (1%). Recently, Malički et al. (2019) used a text-mining approach to search for a limited set of terms related to Bayes factors, confidence intervals, effect sizes, and sample size calculations in instructions to authors provided by 835 journals across a range of scientific disciplines. The prevalence of any statistical guidance, defined by these terms, was 6% across all journals. Another empirical evaluation (Malički et al. 2020) found that none of 57 preprint servers provided any statistical advice. These three studies provide some insight into the prevalence and content of statistical guidance offered to authors; however, Schriger, Arora, and Altman (2006) was limited to medical journals, Malički et al. (2019) focused on only a few statistical topics, and Malički et al. (2020) was limited to preprint servers.

The goal of the present study was to describe the nature and prevalence of statistical guidance offered to authors by 15 journals (top-ranked by Impact Factor) in each of 22 scientific disciplines (N = 330 journals) across five high-level domains (Health & Life Sciences, Multidisciplinary, Social Sciences, Physical Sciences, Formal Sciences). We manually inspected journal websites to identify any author-facing guidance related to statistics. We noted how often guidance pertained to 20 prespecified statistical issues and described whether journals endorsed or opposed statistical methods related to six topics that we considered in advance to be “hotly debated” (p-values, statistical significance, effect sizes, confidence intervals, sample size planning/justification, and Bayesian statistics).

2 Materials and Methods

The study protocol (rationale, methods, and analysis plan) was preregistered on the Open Science Framework study registry on November 23rd, 2019 (https://osf.io/cz9g3/). All departures from that protocol are explicitly acknowledged in supplementary material A. All data exclusions and measures conducted during this study are reported in this manuscript. Our reporting adheres to The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (supplementary material B).

2.1 Design

The study had a cross-sectional design. We examined whether each journal provided any statistical guidance, included a dedicated statistical guidance section in its instructions to authors, or referred to any external statistical guidance (e.g., reporting guideline). For journals that provided their own statistical guidance, we recorded whether the guidance addressed each of 20 prespecified (https://osf.io/cz9g3/) topics (see Figure 2).

2.2 Sample

The sample consisted of the top 15 journals ranked by 2017 Journal Impact Factor (JIF) in each of 22 high-level scientific disciplines across five domains (N = 330 journals). Journals that only publish reviews were not included. This represents the entire population of interest. The sample size of 15 journals per discipline was based on our intuition that this would likely capture the most influential journals in each field. JIFs were obtained from Journal Citation Reports (https://jcr.clarivate.com) and scientific disciplines were defined by Essential Science Indicators (https://perma.cc/MD4V-A5X5; for details see supplementary material C). To help summarize the data, we assigned individual disciplines to higher level scientific domains (Health & Life Sciences, Multidisciplinary, Social Sciences, Physical Sciences, and Formal Sciences.

2.3 Procedure

1. Two investigators (T.B. & T.E.H.) manually searched each journal’s website as of November 2019 and made copies of any statistical guidance intended for authors (for details see supplementary material D). Our operational definition of statistical guidance was: any advice or instruction related to the appropriate selection, implementation, reporting, or interpretation of statistical analyses.

2. A group of investigators (M.M., D.S., M.S.H.) then manually examined and classified the extracted statistical guidance. All classifications were verified by a second investigator (T.E.H.) and any differences were resolved through discussion. Investigators were each randomly assigned (using the R function “sample”) to five ranks and coded journals at that rank in each of the 22 scientific disciplines (i.e., 110 journals per coder).

3. Investigators recorded whether journals provided a dedicated statistical guidance section in their author instructions and whether references were provided to external statistical guidance, such as reporting guidelines. We did not count general instructions to follow external guidelines (e.g., “follow EQUATOR reporting guidelines”) that did not refer to any specific guidelines.

4. For guidance provided directly by journals, investigators recorded whether each of 20 prespecified topics were mentioned and extracted all relevant verbatim guidance pertaining to those topics. For the topic “prespecification of analyses,” we did not count general instructions to register clinical trials unless they specifically referred to statistical analyses in some way, for example, a requirement to share the original statistical analysis plan.

5. For six topics that we considered in advance to be “hotly debated” in the statistical literature (p-values, statistical significance, confidence intervals, effect sizes, sample size planning/justification, and Bayesian statistics), one investigator (T.E.H.) examined the extracted verbatim guidance and classified the level of endorsement offered by the journal according to four categories:

   1. Explicit endorsement—the journal advises or instructs authors to use this method whenever possible/appropriate (e.g., “Authors should report p-values”).

   2. Implicit endorsement—the journal provides advice on the method, implying endorsement, but does not explicitly advise that the method should be used (e.g., “Articles that use significance testing should report the alpha level for all tests”).

   3. Implicit opposition—the journal advises that they would prefer the method is not used, but does not explicitly rule it out (e.g., “articles should not usually contain statistical significance testing”).

   4. Explicit opposition—the journal advises that the method should not be used (e.g., “Authors should replace p-values with effect size estimates and 95% confidence intervals”).

6. For guidance pertaining to p-values, statistical significance, and confidence intervals, one investigator (T.E.H.) recorded whether there were instructions to report exact p-values/alpha levels/confidence levels or use specific p-value thresholds/alpha/confidence levels (e.g., p < 0.05, $\alpha =0.05$, 95% confidence interval).

2.4 Data Analysis

We report descriptive statistics (number and proportion of journals) stratified by domains and disciplines. Analyses were performed using custom code written in the programming language R (R Core Team 2021).

3 Results

3.1 Journal Impact Factors

Across all 330 journals, the median 2017 Journal Impact Factor was 7.54 (IQR = 6.56; range 2.45–79.26). Journal Impact Factors for each scientific discipline are available in supplementary Table E1.

3.2 Frequency of Statistical Guidance in General

Overall, we found that 160 journals offered some statistical guidance. This includes 32 journals that only referred to statistical guidance in external sources (reporting guidelines or academic papers) and 128 journals that provided their own guidance (of which 105 also referred to external sources). Of the 160 journals offering statistical guidance, 93 had a dedicated statistics section in their author instructions. Some journals shared the same publisher-level guidance, including 31 Nature journals, 12 Cell journals, and 2 Frontiers journals.

There was considerable heterogeneity across disciplines and domains (Figure 1; for equivalent tabular data, see https://osf.io/pkjhf/). The frequency of statistical guidance was highest in the Health & Life Sciences (122 of 165 journals, 74%), followed by Multidisciplinary journals (9 of 15 journals, 60%), Social Sciences (8 of 30 journals, 27%), Physical Sciences (21 of 90 journals, 23%), and the Formal Sciences (0 of 30 journals, 0%). In one discipline (Clinical Medicine) Statistical guidance was provided by all examined journals and in two disciplines (Mathematics and Computer Science) no examined journals provided statistical guidance.

Fig. 1 Frequency of journals offering some statistical guidance (circles) or having a dedicated statistical guidance section (diamonds) in each scientific discipline (bars not highlighted, n = 15 journals per discipline) and each higher-level scientific domain (bars highlighted; Health & Life Sciences: n = 165 journals; Multidisciplinary: n = 15; Social Sciences: n = 30; Physical Sciences: n = 90; Formal Sciences, n = 30). Data is presented in descending order of frequency first by domain, then by discipline within domains.

3.3 Frequency of Statistical Guidance on Specific Topics

Of the 128 journals that provided their own guidance, 117 mentioned at least one of the 20 prespecified statistical topics (supplementary material Figures F1–F22) and 11 did not mention any of these topics (but did provide guidance on other miscellaneous statistical issues). The maximum number of topics mentioned by an individual journal was 15. The median topics mentioned was 6 (IQR = 8; supplementary material Figure F23).

The frequency of guidance pertaining to each of 20 prespecified statistical topics is shown stratified by domain in Figure 2 and stratified by discipline in supplementary material Figures G1–G4. Overall, guidance was most common for confidence intervals (n = 90 journals), p-values (n = 88), sample size planning/justification (n = 72), effect sizes (n = 66), prespecification of analyses (n = 65), and data exclusions (n = 64). Guidance was less common for checking model assumptions (n = 61), multiple comparisons (n = 60), statistical significance (n = 53), one-tailed hypothesis tests (n = 51), Bayesian statistics (n = 39), null hypotheses (n = 33), subgroup analyses (n = 30), secondary outcomes (n = 25), missing data (n = 18), sensitivity analyses (n = 11), nonparametric tests (n = 10), baseline covariates (n = 9), outliers (n = 7), and categorization of continuous outcomes (n = 2). Guidance on the 20 topics tended to be more common in the Health & Life Sciences and Multidisciplinary journals compared to other domains (Figure 2) and was especially common in the Clinical Medicine journals compared to other disciplines (supplementary material Figures G1–G4). Examples of verbatim guidance are provided in supplementary Table H1. A sensitivity analyses excluding journals that shared publisher-level guidance is available in supplementary material I.

Fig. 2 Frequency of journals offering guidance on 20 prespecified statistical topics. Data is stratified by scientific domain (Health & Life Sciences: n = 165 journals; Multidisciplinary: n = 15; Social Sciences: n = 30; Physical Sciences: n = 90); the Formal Sciences are not shown because no journals provided statistical guidance. Data is presented in descending order of overall frequency across all domains. CCV = categorizing continuous variables.

3.4 External Statistical Guidance

One-hundred and thirty-seven journals referred authors to statistical guidance in external sources (supplementary material J). Of these, 32 only referred to statistical guidance in external sources, whereas the other 105 also provided their own additional guidance. Seventy-nine different external sources were referred to, of which 49 were reporting guidelines and 30 were other types of documents, such as academic papers, textbooks, or society guidelines. Referring to external guidance was most common in the Health & Life Sciences (109 of 165 journals, 66%) followed by Multidisciplinary journals (8 of 15 journals, 53%), Physical Sciences (17 of 90 journals, 19%), Social Sciences (3 of 30 journals, 10%), and Formal Sciences (0 of 30 journals, 0%).

The most commonly referred to reporting guidelines were CONSORT (n = 95 journals), ARRIVE (n = 80), PRISMA (n = 47), REMARK (n = 40), and STARD (n = 36), with all other reporting guidelines being referred to by 21 or fewer journals. All other types of external guidance were referred to by 3 or fewer journals (supplementary material J).

3.5 Statistical Guidance on “Hotly Debated” Topics

Table 1 shows how many of the examined journals offered different levels of endorsement for six statistical topics that we considered in advance to be “hotly debated” in the statistical literature (note that these only partly overlap with the most commonly mentioned statistical topics in journal guidance, as shown in Figure 2). Ninety journals provided guidance on confidence intervals, of which the vast majority (n = 85) offered explicit endorsement. Most of these journals (n = 69) did not provide any guidance on confidence levels, but 16 advised authors to use a 95% confidence level and 5 advised authors to report their chosen confidence level.

Table 1 Number of journals offering various levels of endorsement for six “hotly debated” statistical topics.

Topic	Explicit	Implicit	Implicit	Explicit
	endorsement	endorsement	opposition	opposition
Confidence intervals	85	4	1^#	0
Sample size planning/	67	5	0	0
justification
Effect sizes	62	4	0	0
p-values	10	77	0	1*
Statistical significance	9	35	5	3
Bayesian statistics	0	39	0	0

Explicit endorsement = the journal advises or instructs authors to use this method whenever possible/appropriate. Implicit endorsement = the journal provides advice on the method, implying endorsement, but does not explicitly advise that the method should be used. Implicit opposition = the journal advises that they would prefer the method is not used, but does not explicitly rule it out. Explicit opposition = the journal advises that the method should not be used.

NOTE: Some opposition was caveated as indicated by the symbols below.

^# Confidence intervals are implicitly opposed by this journal only in the context of small sample sizes.

* p-values are opposed by this journal only when there were no prespecified multiplicity corrections.

Eighty-eight journals provided guidance on p-values, of which the vast majority offered implicit endorsement (n = 77). One journal explicitly opposed the use of p-values, but only in the absence of prespecified multiplicity corrections. Fifty-two journals advised reporting exact p-values, 21 advised reporting of exact p-values unless they were very small (e.g., p < 0.001), 5 advised authors to use reporting thresholds (e.g., p < 0.05, p < 0.01, p < 0.001), and 10 offered no guidance on whether to report exact p-values or use thresholds.

Fifty-three journals provided guidance on statistical significance, of which the majority (n = 35) offered implicit endorsement. Three journals explicitly opposed the use of statistical significance, instructing authors to report exact p-values and/or standard errors/confidence intervals instead. Thirty journals advised authors to report the alpha level they used, 2 advised using an alpha level of 0.05, 1 advised using an alpha level of 0.01, and 20 offered no guidance on alpha levels. Seventy-two journals provided guidance on sample size planning/justification, of which the vast majority (n = 67) offered explicit endorsement. 66 journals offered guidance on effect sizes, of which the vast majority (n = 62) offered explicit endorsement. All 39 journals that provided guidance about Bayesian statistics provided implicit endorsement.

4 Discussion

Our investigation revealed considerable heterogeneity in the provision of statistical guidance by 330 top-ranked journals across scientific disciplines and domains. The vast majority of the examined Health & Life Sciences journals provided statistical guidance (including all Clinical Medicine journals), as did the majority of the examined Multidisciplinary journals. Less than half of the examined journals in the Social Sciences and Physical Sciences provided statistical guidance, and none of the examined journals in the Formal Sciences (Mathematics and Computer Science) did. Of the total 160 journals that offered some statistical guidance, 67 did not have a dedicated statistical guidance section in their author instructions. Statistical guidance addressed a range of topics, and was most common for hotly debated topics (p-values, confidence intervals, sample size planning/justification, and effect sizes) with the emphasis varying across disciplines. Few of the examined journals appeared to take particularly controversial positions, such as banning p-values or the use of statistical significance.

Our finding that 48% of the 330 examined journals across disciplines provided statistical guidance is substantially higher than observed in prior research on this topic. Malički et al. (2019) found that statistical guidance was provided by only 6% of 835 journals across a similarly broad range of scientific disciplines; however, this estimate was based on a text-mining approach sensitive to only a narrow range of statistical topics (specifically, Bayes factors, confidence intervals, effect sizes, and power/sample size calculations). Additionally, our sample only included journals with high Impact Factors whereas Malički et al. examined a representative (randomly selected) sample of journals across a range of citation impact. It is possible that journals with lower citation impact are less likely to provide statistical guidance (as is the case for authorship and conflict of interest guidance according to a recent systematic review, Malički et al. 2021). Our methodology is more comparable to Schriger, Arora, and Altman (2006) who reported that 39% of 166 major medical journals provided statistical guidance. By contrast, all of the medical journals we examined provided statistical guidance; however, we only examined the 15 journals with the highest Impact Factor. It is possible that there has been an increase in the provision of statistical guidance over time (as has been shown for guidance related to authorship, conflicts of interest, and data sharing, Malički et al. 2021), but empirical comparisons are not possible because journal-level information is not provided in Schriger et al. Our study contributes to this literature by establishing a benchmark against which future studies of journal-based statistical guidance can be compared.

Although there was substantial variation across disciplines in the provision of statistical guidance, our study was not designed to detect reasons for these differences. It is plausible that the examined Health & Life Sciences journals were more likely to provide statistical guidance because (a) it seems especially important to avoid errors with highly proximal repercussions for human health; (b) biomedical and health-related research is often subject to stricter regulatory oversight; (c) there is a plethora of empirical data highlighting the severity and prevalence of statistical misuse, misreporting, and misinterpretation in health research (Altman 1994; Strasak et al. 2007); (d) journals may feel they need to compensate for lower statistical competency among Health & Life Sciences researchers due to inadequate educational programs or statistical support services (Aiken, West, and Millsap 2008; Windish, Huot, and Green 2007). Differences in the use of statistics between disciplines is also likely to be a major factor; for example, statistics are likely to be less useful in the Formal Sciences relative to more applied sciences and may explain the absence of statistical guidance in the examined Computer Science and Mathematics journals. Cultural or historic factors related to the role that journals traditionally play in the oversight of research may also be important, though we are not aware of relevant empirical evidence that documents this.

Moreover, because statistical methods and their optimal use differ across scientific subfields it is difficult to score or rank journal-based statistical guidance. It may be useful to assess the accuracy, thoroughness, and consistency of statistical guidance at the level of individual journals; however, that was beyond the scope of our study. Future work may try to assess such performance by examining not only statistical guidance but also the use and misuse of statistics in specific papers published by each journal.

It is unclear whether journal-based statistical guidance is an effective means to address misuse, misreporting, and misinterpretation of statistics. Indeed, one may be skeptical about whether instructions to authors are even commonly read. Some evidence does suggest that, at least in particular scenarios, editorial advocacy for specific statistical practices has been effective. For example, several journals have successfully encouraged authors to abandon null-hypothesis significance testing in favor of an estimation approach centered on reporting effect sizes and confidence intervals (Fidler et al. 2004; Finch et al. 2004; Giofrè et al. 2017). However, these effects seem at least partly dependent on the active campaigning and oversight of specific editors, rather than written guidance to authors alone. A coordinated approach in which the journal’s expectations about statistical practice are made explicit in author instructions, then reinforced by editors and reviewers is likely important. Soliciting dedicated statistical reviewers may be more effective at addressing statistical issues beyond simple reporting and previous studies have found that the use of statistical review is rare in some fields, such as psychology (Hardwicke et al. 2019) and relatively common in other fields, such as biomedicine (Hardwicke and Goodman 2020).

Journals are only one interventional lever for improving statistical practice and an exclusive focus on journals is likely to be insufficient. In one study, Fidler et al. (2004) observed that although authors tended to follow instructions to report confidence intervals, they often showed little evidence of using them to aid their interpretation of the research results. Ultimately, any journal-based intervention intended to improve statistical practice is likely to be limited by the knowledge, motivation, and understanding of authors (and reviewers), the improvement of which will require nontrivial changes to reward structures and educational practices deeply rooted in the scientific ecosystem (Goodman 2019).

As noted in the introduction, even well-intentioned statistical guidance offered by journals may be detrimental. This was clear in the case of the flawed statistic $p_{\text{rep}}$ introduced at the journal Psychological Science (Iverson et al. 2009; Killeen 2005) and arguably the case with the ongoing p-value ban at the journal Basic and Applied Social Psychology, depending on one’s statistical philosophy (Fricker et al. 2019; Trafimow and Marks 2015). Mayo (2021) has recently argued that journal editors should enforce proper use of statistical methods but avoid “taking sides” on particularly controversial matters, or advocating for specific statistical philosophies. However, the definition of “proper use” may be in the eye of the beholder and some experimentation with different approaches could be informative, if ethically appropriate and properly monitored with meta-research (Hardwicke et al. 2020). Nevertheless, we found that very few journals in our sample appeared to be outwardly and decisively “taking sides” in the sense of explicitly opposing the use of “hotly debated” statistical tools like p-values and statistical significance; however, the choice of wording often implicitly endorsed or opposed certain practices, which suggests journals may be “taking sides” albeit without clear language.

It was common for journals to provide their own idiosyncratic statistical guidance. However, some publishers had developed guidance that was applied across multiple journals, often in the form of checklists (e.g., the Nature Life Sciences Reporting Checklist). Moreover, many journals (especially in the Health & Life Sciences) referred to the same reporting guidelines, particularly the most established ones available on the EQUATOR website, like CONSORT, ARRIVE, and PRISMA. It is unclear whether authors refer to, and whether editors and reviewers enforce, external reporting guidelines comparably with internal ones. Increasing standardization could improve efficiency and consistency for authors, reviewers, and readers; but some degree of variability and experimentation may also be desirable. It is unclear where the balance lies.

Our study has some important limitations. (a) We focused only on statistical guidance provided on journal websites. We do not know how many authors actually read this information, or the extent to which journals enforce their use through editorial oversight or statistical review (Hardwicke et al. 2019; Hardwicke and Goodman 2020); (b) We deliberately based our sample on 22 broadly defined disciplines to ensure wide coverage across the sciences, but this does not necessarily imply that the findings will straightforwardly generalize to lower-level sub-fields within those disciplines. Moreover, we focused only on journals with the highest Impact Factor because they are likely to have the clearest potential to modify author behavior. Lower-ranked journals may differ in many important respects; for example, they may have fewer resources to dedicate to developing statistical guidance. It is unclear if they might be more or less willing than top-ranked journals to introduce policy changes that could cause controversy, such as p-value bans. (c) As noted, for the topic “preregistration of analyses” we did not count general instructions to register clinical trials unless they specifically referred to the prespecification of statistical analyses. It is possible that some of the examined journals assumed a general instruction to register trials also constituted a specific instruction to prespecify statistical analyses; however, we did not consider that to be sufficiently direct. Similarly, we did not consider a general instruction to authors that they should follow EQUATOR reporting guidelines (without specifically mentioning individual guidelines) to be sufficiently direct and did not count it as statistical guidance.

In conclusion, statistics are often a vital pillar in the edifice of scientific claims, but they are widely misused, misreported, and misinterpreted across several scientific disciplines (Altman 1982; Gigerenzer 2018; Nieuwenhuis, Forstmann, and Wagenmakers 2011; Sedlmeier and Gigerenzer 1989; Strasak et al. 2007; Wasserstein and Lazar 2016). Journals could play a key role in alleviating these statistical ailments before they plague the academic literature. However, our investigation shows that even at top-ranked journals, only a minority provide any statistical guidance. Moreover, there are inconsistencies in the guidance provided across journals. Our investigation provides a layer of empirical evidence upon which to inform ongoing research and debate about the role of journals in improving the statistical practice of scientists.

Supplementary Materials

The supplementary materials contain protocol amendments (supplementary materials A), a STROBE reporting checklist (supplementary materials B), definitions of scientific disciplines (supplementary materials C), procedures for preserving statistical guidance (supplementary materials D), journal Impact Factors stratified by discipline (supplementary materials E), additional data on twenty prespecified statistical topics (supplementary materials F), additional graphs for study one (supplementary materials G), examples of guidance for each of twenty statistical topics (supplementary materials H), the results of a sensitivity analysis on statistical topics without shared publisher-guidance (supplementary materials I) and external sources of statistical guidance (supplementary materials J).

Authors’ Contributions

Conceptualization: T.E.H., J.P.A.I., M.M., and D.S. Data curation: T.E.H. and M.S.H. Formal analysis: T.E.H. and M.S.H. Investigation: T.E.H., M.S.H., M.M., and T.B. Methodology: T.E.H., J.P.A.I., M.M., and D.S. Project administration: T.E.H. Resources: T.E.H. Software: T.E.H. Supervision: T.E.H. and J.P.A.I. Validation: T.E.H. Visualization: T.E.H. Writing – original draft: T.E.H. Writing – review & editing: T.E.H., J.P.A.I., M.S.H., M.M., D.S., and T.B.

Data Availability Statement

All data, materials, and analysis scripts related to this study are publicly available on the Open Science Framework repository (https://dx.doi.org/10.17605/OSF.IO/JRX7D). To facilitate reproducibility this manuscript was written by interleaving regular prose and analysis code using knitr (Xie 2017) and papaja (Aust and Barth 2020), and is available in a Code Ocean container (https://doi.org/10.24433/CO.6131540.v1) which recreates the software environment in which the original analyses were performed.

Disclosure Statement

Mario Malički is a Co-Editor-in-Chief of the journal Research Integrity and Peer Review. All other authors declare no conflicts of interest.

Additional information

Funding

The authors received no specific funding for this work and no funder had any role in the research. Tom E. Hardwicke receives funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 841188. Maia Salholz-Hillel is employed as a researcher under research grants from the German Bundesministerium für Bildung und Forschung (BMBF). The Meta-Research Innovation Center at Stanford (METRICS) is supported by a grant from the Laura and John Arnold Foundation. The Meta-Research Innovation Center Berlin (METRIC-B) is supported by a grant from the Einstein Foundation and Stiftung Charité.