Responding to Confidence and Reproducibility Crises: Registered Reports and Replications in Auditory Perception & Cognition

Publish or perish. For decades, that has been a rule of thumb in academia. Certainly, while systems for the review and advancement of academic employees vary across the globe, published scholarship is generally a major requirement for an individual to obtain employment, tenure where applicable, and/or promotion. Merit raises also may be dependent on the frequency, or total number, of publications. It therefore should be of little surprise that researchers have powerful incentives to see their work in print. Regrettably, the need to publish has resulted in practices that reflect questionable ethics that threaten the field’s knowledge base. These questionable practices include, but are not limited to:

• p-hacking, or the misreporting of true effect sizes resulting from researchers collecting or selecting data or statistical analyses until nonsignificant results become significant, thereby yielding false positives (Head et al., 2015),

• HARKing (“Hypothesizing After the Results are Known”), or presenting a post hoc hypothesis as if it were an a priori hypothesis (Kerr, 1998), and

• QRPs (Questionable Research Practices), such as p-value rounding and excluding data after looking at the impact of doing so on the results, and failing to report all of a study’s dependent measures (John, Loewenstein, & Prelec, 2012).

In light of such practices, the extent to which researchers can depend upon the reliability and validity of relevant findings has recently been questioned. As Simmons, Nelson, and Simonsohn (2011) note, researchers face numerous decisions, such as when to cease data collection, which data/participants should be excluded, and how variables should be combined, modified, and/or defined. These decisions, especially if they are made after data collection has begun, have the capability of influencing outcomes and, thus, possibly whether hypotheses are supported. Pashler and Wagenmakers (2012) discuss a “crisis of confidence” in science resulting from the fabrication and falsification of data, the unwillingness and inability of researchers to share data, and the inability to replicate findings. With regard to the reproducibility of results, a recent endeavor (Open Science Collaboration, 2015; see also Maxwell, Lau, & Howard, 2015) revealed that out of 100 previously published experimental and correlational studies, only approximately one-third yielded significant results, and less than half yielded effect sizes that were comparable to the original study.

Replication is generally acknowledged as a critical component of the scientific method and advancement for any scientific discipline (e.g., Cesario, 2014). Thus, when a study fails to replicate a particularly relevant finding, the new result can be quite informative. Yet, the rate of publication for any replication-based studies, let alone ones that fail to replicate previous results, has remained well below the corresponding rate of publication for novel findings (see Martin & Clarke, 2017). Some of this reduction likely has to do with the fact that researchers and publishers deem replication work to be less likely to draw significant attention from potential readership relative to more novel contributions, and therefore, to be less likely to positively influence a journal’s impact factor. This bias will probably continue until journals routinely reserve publication space for replication work.

Another factor further reduces the chances of publication for studies that fail to replicate previous findings. It has long been established that, for traditional work involving null hypothesis testing, there is a bias to publish work that reports findings that reach statistically significant levels of differences across conditions (e.g., Franco, Malhotra, & Simonovits, 2014; Schooler, 2011; Simmons et al., 2011). In fact, long ago Rosenthal (1979) discussed the “file drawer problem”, which refers to notion that “the file drawers back at the lab are filled with 95% of the studies that show nonsignificant (e.g., p >.05) results” (p. 638). Psychological researchers used to routinely acknowledge this problem when they obtained data that failed to reach statistical levels of significance by joking “Well, there’s another one for the Journal of Null Results”. Later, in response to this bias, and in recognition of the fact that researchers were consistently revisiting the same territory with the same general outcomes, a Journal of Articles in Support of the Null Hypothesis was even eventually established (see https://www.jasnh.com/; Ricker, 2002).

The recognized need for an increased emphasis on replication work, as well as for increases in the accountability of researchers, including transparency about their obtained data and bases for their design and analysis decisions, has spear-headed the recent Open Science movement (e.g., https://www.cos.io). Nowhere are these motivations made more clear than in the stated mission within the strategic plan for the Center for Open Science (or COS), which is “ … to increase openness, integrity, and reproducibility of research” (see https://osf.io/x2w9h/; Last edit: 2 July 2018). The COS provides some fundamental mechanisms to help accomplish their mission. Primary among these mechanisms is the Registered Report (henceforth referred to as RR). RRs refer to empirical studies in which the rationale, methodology (e.g., stimuli and procedure), and proposed analyses are evaluated by reviewers prior to the collection of data (with the possible exception of pilot data). RRs provide an opportunity for investigators to have referees evaluate, improve, and/or approve the theoretical basis and methodological design before data is accumulated. Too often researchers face post hoc criticism of their logic and methodology. RRs provide an a priori evaluation. Problems detected prior to data collection can be remedied, which, in turn, can increase the likelihood a manuscript will be positively evaluated, regardless of the outcome of data analyses. Thus, a priori approval of a research plan within a RR provides a path for publishing a failure to replicate previous findings.

The COS also actively encourages transparency and accountability by having researchers publicly share de-identified data, as well as research and analysis plans, as part of the RR process. This is accomplished through the center providing a hosting service online, the Open Science Framework (or OSF), that permits researchers to store and access all manners of documentation about a research project (see https://osf.io/). Within the OSF platform, researchers share information about research design and idea formation as easily as data sets and analysis information. At the time of writing, well over 300,000 studies are included within the OSF registry.

Many journals have heeded the call for open science. According to the COS (see https://katiedrax.shinyapps.io/cos_registered_reports/), as of the time of writing this editorial, 245 journals offer RRs, most of which are in psychology, particularly in experimental research areas. Several of these are produced by Taylor & Francis, which also publishes AP&C.

The editorial board at AP&C thinks that the recommended steps by the COS have a positive overall impact on science, including within our respective disciplines. We are supportive of any efforts that are pursued in the interest of improving researcher transparency, increasing the number of replication studies, and avoiding common publication biases. As a result, to be responsive to these aforementioned crises, we have updated submission and publication policies at the journal. These changes follow the general spirit of open science, but also reflect a few differences in implementation.

There are two sets of important policy changes: (1) inclusion of RRs, and (2) encouraged submission and publication of replication studies. Fundamental aspects of these policy updates are outlined briefly below. Additionally, for convenience, a more complete copy of the journal’s guidelines for RRs is provided as an appendix to this editorial. Many of these guidelines are identical to the general guidelines provided by the publisher, which can be found online at https://www.tandf.co.uk//journals/authors/registered-report-guidelines.pdf. Further details about these policies also can be found in the journal’s instructions to authors at https://www.tandfonline.com/action/authorSubmission?show=instructions&journalCode=rpac20.

We firmly believe that these adjustments to policy have the capacity to help enhance the knowledge base related to auditory perception and cognition by helping to further strengthen scientific practice. Furthermore, these changes are in keeping with our commitment to provide a publication outlet in AP&C with minimal boundaries to the types of research design, topics, and article formats that will be considered. In this way, auditory researchers should feel even greater encouragement to share their work and submit their manuscripts to AP&C.

Registered Reports

RRs are optional. Manuscripts not initially submitted as RRs will, of course, be considered for publication. RRs are evaluated using a 2 stage process that closely adheres to the guidelines for RRs provided by the Center for Open Science (2020), which can be found at https://osf.io/pukzy/. In Stage 1, prior to the collection of data (with the possible exception of pilot data), the author(s) submit a manuscript limited to the introduction, methods, a statistical power analysis, and an analysis plan. The manuscript will be evaluated by a team of three (3) reviewers, and a decision will be rendered (accept, revise and resubmit, or reject). A Stage 1 decision of accept reflects an in principle acceptance (IPA), meaning that the work will eventually be published as long as the data are collected, analyzed, and interpreted correctly, based on peer-reviewed suggestions and requirements.

Stage 2 submissions consist of the completed manuscript (inclusion of Results and Discussion sections). It is expected authors may wish to conduct analyses not included in the RR. It is not uncommon for interesting and unexpected findings to arise. Supplemental, exploratory analyses are permitted. However, such analyses must be clearly justified in the manuscript and must be reported in a separate section of the Results (“Exploratory analyses”).

Replications

The journal also recognizes the fundamental scientific value of replication work, which might be, but is not required to be, submitted in the form of a RR. As a further indication of this commitment to replication while guarding against reviewer bias toward novel approaches, the journal will regularly reserve space for the publication of replication-based evaluations. As with RRs, publication decisions for replication work are not dependent upon whether author interpretations/conclusions concur with previously published outcomes. Manuscripts submitted as a replication will be evaluated using the method described above (if submitted as a RR) or in accordance with procedures already in existence (manuscripts not submitted as a RR).

Disclosure Statement

No potential conflict of interest was reported by the authors.

Appendix: Guidelines for Submission and Review of Registered Reports

Stage 1

Initial Stage 1 submissions should include the following sections:

Cover Letter

This should include:

• A statement confirming that all necessary support (e.g., funding, facilities) and approvals (e.g., ethics) are in place for the proposed research.

• An anticipated timeline for completing the study if the initial submission is accepted.

• A statement confirming that the authors agree to share their raw data and any digital study materials for all published results. If not possible, an explanation must be provided.

Introduction

• A review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Please note that following a review decision of in principle acceptance, the Introduction section cannot be altered apart from correction of typographical errors.

Method

• Full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g., outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.

• A description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent experiments or any Stage 2 manuscript can be rejected.

• Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only preplanned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results (see below).

• The manuscript must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature. Power analysis must be based on the lowest available or meaningful estimate of the effect size.

Pilot Data (optional)

• Can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Any pilot experiments will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the pre-registered experiment(s).

Authors are reminded that any deviation from the stated experimental procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2.In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g., change of equipment or unanticipated technical error), the authors must consult the editors immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted per editorial discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the experimental procedures more substantially following IPA but still wish to publish their article as a Registered Report, then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. Note that all registered analyses must be undertaken and reported in the Stage 2 manuscript, but additional unregistered analyses may also be included in a final manuscript (see below).

Peer review

Stage 1 submissions that are judged by the editors to be of sufficient quality and scientific importance will be sent for in-depth peer review. In considering papers at the registration stage, reviewers will be asked to assess:

1. The importance of the research question(s).

2. The logic, rationale, and plausibility of the proposed hypotheses.

3. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).

4. Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.

5. Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks.

Following Stage 1 peer review, manuscripts will be rejected outright, offered the opportunity to revise, or accepted. In principle acceptance (IPA) indicates that the article will be published pending completion of the approved methods and analytic procedures, passing of all pre-specified quality checks, and a defensible interpretation of the results. Stage 1 protocols are not published following IPA. Instead they are held in reserve by the journal and integrated into a single completed article following approval of the final Stage 2 manuscript.

Stage 2

Once the study is complete, authors prepare and resubmit their manuscript for full review.

Introduction

Apart from minor stylistic revisions, the Introduction cannot be altered from the approved Stage 1 submission, and the stated hypotheses cannot be amended or appended. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods (e.g., correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of IPA should be covered in the Discussion.

Results & Discussion

• The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis may be subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made, and that the initially proposed analysis was inappropriate.

• It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses.

• Authors reporting null hypothesis significance tests are required to report exact p-values and effect sizes for all inferential analyses.

Peer review

In considering papers at Stage 2, reviewers will be asked to decide:

1. Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls).

2. Whether the Introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required).

3. Whether the authors adhered precisely to the registered experimental procedures.

4. Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative.

5. Whether the authors’ conclusions are justified given the data.

Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty or conclusiveness of the results. Thus, while reviewers are free to enter such comments on the record, they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, authors are not obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.

Manuscript Withdrawal and Withdrawn Registrations

It is possible that authors with IPA may wish to withdraw their manuscript following or during data collection. Possible reasons could include major technical error, an inability to complete the study due to other unforeseen circumstances, or the desire to submit the results to a different journal. In all such cases, manuscripts can of course be withdrawn at the authors’ discretion. However, the journal will publicly record each case in a section called Withdrawn Registrations. This section will include the authors, proposed title, the abstract from the approved Stage 1 submission, and brief reason(s) for the failure to complete the study. Partial withdrawals are not possible; i.e., authors cannot publish part of a registered study by selectively withdrawing one of the planned experiments. Such case must lead to withdrawal of the entire paper. Studies not completed by the agreed Stage 2 submission deadline can be extended in negotiation with the editorial office.