Promoting Transparent Reporting of Conflicts of Interests and Statistical Analyses at The Journal of Sex Research

The publication process for scientific communication is iterative. Researchers first attempt to convince editors and reviewers of claims made about the variables under study; editors and reviewers then evaluate the reasonableness of these claims in light of the strength of the supporting evidence. Editors (with input from reviewers) then either decide that researchers’ claims (1) are acceptable given the evidence, (2) require more evidence to evaluate, or (3) are unacceptable given the evidence. Ideally—and providing other aspects of the manuscript are satisfactory—manuscripts with strong supporting evidence should be published; those with good supporting evidence should be encouraged to improve and then be reconsidered; and those with poor supporting evidence should be rejected. The integrity of this process, however, relies on authors transparently reporting aspects of their research process that ought to influence (for better or for worse) how reviewers and editors would evaluate the evidential value of their article. When important details are omitted from article submissions, reviewers and editors will be less likely to make judicious appraisals of manuscripts. Transparent reporting by authors therefore helps to ensure a healthy, high-quality published literature of scientific findings.

The Journal of Sex Research is updating its submission requirements to promote more transparent reporting in two areas of disclosure: conflicts of interest (COIs) and reporting of statistical analyses. Our intent with this editorial is to outline the nature of these changes and to articulate why we feel they are necessary to implement at this time.

Promoting Transparent Reporting of COIs

Potential conflicts of interest have been defined as “when an investigator’s relationship to an organization affects, or gives the appearance of affecting, his/her objectivity in the conduct of scholarly or scientific research” (Office of Research Integrity [ORI], 2017). While the most obvious examples of conflict of interest are financial in nature—for example, owning stock in pharmaceutical companies—conflicts of interest can occur as a result of many other less-recognized sources, such as personal relationships and academic competition (International Committee of Medical Journal Editors, 2010).

Though a conflict of interest does not guarantee that research findings would have been different than those obtained without such a conflict, the problem is that there is the potential for this to be the case (DeAngelis, 2000). It is also important to note that some conflicts of interest may be unavoidable and may not necessarily be “unethical” (ORI, 2017). Transparency in reporting possible conflicts of interest is nonetheless essential for at least two reasons. First, researchers themselves are not always best placed to recognize whether a conflict of interest exists in relation to their research. Second, it has been argued that disclosure of possible conflicts of interest is essential to maintain public trust in research (Johns, Barnes, & Florencio, 2003).

The advice from professional societies, organizations, and, increasingly, from journals is for authors to disclose any possible conflicts of interest. While at JSR we have asked authors to disclose information about any funding they received for research and have posed a generic question about “conflicts of interest,” to date we have not provided any guidance to authors about what constitutes conflict of interest. We have remedied this now, with additional guidance on conflicts of interest and a request during the online submission process to provide details of any perceived conflicts of interest.

Promoting Transparent Reporting of Statistical Analyses

Sexual science might be thought of as lying, in part, at the intersection of psychological and medical science. And for the past five years or so, both fields have been forced to grapple with the uncomfortable recognition that their research findings are not replicable (Nosek & Errington, 2017; Open Science Collaboration, 2015), populated by unreliable significant effects emerging at a rate that is too good to be true (Fanelli, 2010; Ioannidis, 2005).

There are many reasons why the replicability of a given body of research might leave something to be desired, ranging from systemic to individualized factors (Nosek & Bar-Anan, 2012; Nosek, Spies, & Motyl, 2012) and the use of devious to seemingly innocuous practices. For example, making simple reporting mistakes (Nuijten, Hartgerink, Van Assen, Epskamp, & Wicherts, 2016), running chronically underpowered studies (Maxwell, 2004; Schimmack, 2012), refusing to let others verify one’s findings (Wicherts, Bakker, & Molenaar, 2011; Wicherts, Borsboom, Kats, & Molenaar, 2006), or engaging in flagrant data fabrication (Fanelli, 2009; Stapel, 2014) are all very different behaviors morally, but they share a practical consequence in rendering the scientific record less trustworthy.

But while psychological (Nelson, Simmons, & Simonsohn, 2017; Spellman, 2015) and medical scientists (Baker, 2016; Begley & Ellis, 2012) have debated—often heatedly—the extent of the replicability problem (e.g., Open Science Collaboration, 2015; versus Gilbert, King, Pettigrew, & Wilson, 2016; versus Etz & Vandekerckhove, 2016) and the merits of each solution du jour (e.g., Finkel, Eastwick, & Reis, 2017; versus LeBel, Berger, Campbell, & Loving, 2017; Simons, 2014; versus Crandall & Sherman, 2016), sexual scientists have been notably silent on these matters (see Sakaluk, 2016b). However, for reasons that are not yet apparent, the past year or two seems to have marked the end of this trend. In the past year, sexuality conferences have featured multiple replicability-related talks (e.g., Lalumière, 2016; Sakaluk, 2017); sexuality journals have begun piloting interventions to increase replicability (e.g., the Canadian Journal of Human Sexuality has been evaluating the use of statcheck in the peer-review process; Epskamp & Nuijten, 2016); and sexual scientists have conducted and published direct replications challenging classic findings (e.g., Balzarini, Dobson, Chin, & Campbell, 2017). The most recent example of the increased attention being paid to replicability in sexual science is the new statistical disclosure policies at the journal Sexual Abuse, which encourages authors to disclose practices “that can increase the risk of spurious significant findings.”¹ JSR can now be counted among these recent efforts aspiring to increase transparency and replicability of sexual science findings.

JSR is changing its submission requirements, requiring two new elements of statistical reporting and encouraging two others, for all papers submitted using inferential statistical tests.² The first new requirement—requiring authors to disclose all elements of analytic flexibility—is functionally new and designed to curb increasing rates of Type I (i.e., false-positive) errors of inference. The remaining three submission guidelines (one of which is also required) are practices that many likely already use, and are explicit guidelines of current APA style found in the Publication Manual of the American Psychological Association, Sixth Edition (American Psychological Association, 2009), which the journal already uses; we are now simply formalizing them. We describe each new reporting policy in the sections that follow, including a rationale for why JSR is adopting them.

Required Reporting of Analytic Flexibility

One of the observations made during the earliest days of the so-called replicability crisis is that social scientists regularly (ab)use flexibility when collecting, analyzing, and reporting on their data (John, Loewenstein, & Prelec, 2012). Some examples include collecting more data (58%), or stopping data collection early (22.5%), after researchers have seen whether a key effect is significant, and deciding to exclude cases after looking at the impact on key p values (43.4%), and/or simply reporting only those studies that “worked” (i.e., yielded the desired significant effects; 50%). Critical observers have variously dubbed these practices, and others, questionable research practices (or QRPs; John et al., 2012), p-hacking or researcher degrees of freedom (Simmons, Nelson, & Simonsohn, 2011), and the garden of forking paths (Gelman & Loken, 2013).³

A pivotal simulation paper by Simmons et al. (2011) makes clear the consequences of capitalizing on analytic flexibility: Researchers inadvertently trick themselves into thinking they are conducting null-hypothesis significance tests of their effects at an acceptable Type I error rate (i.e., α = .05), when in fact these rates are dramatically increased (as high as α = .60, in some cases); p values are effectively meaningless when post hoc analytic decisions are made (Dienes, 2008). What this means, in practicalities, is that a significant effect produced via (ab)use of researcher degrees of freedom should not be taken as seriously as a significant effect that was observed using principled data collection, analysis, and reporting practices that were decided a priori.⁴

Though there may be good reasons, a priori, for collecting more data, reporting on some analyses and not others, and/or excluding some participants, once a researcher has seen how these decisions affect p values it is impossible to distinguish between principled decision making and motivated reasoning and mindless data torturing. Researchers should therefore strive to avoid making these kinds of analytic decisions post hoc altogether. Conducting replications (direct and conceptual) and/or making use of preregistration within the same multistudy paper are other recently discussed strategies that researchers can use to reduce the possibility of p-hacking (or the appearance of p-hacking; see Nosek, Ebersole, DeHaven, & Mellor, 2017; Sakaluk, 2016a).

To make it clearer to JSR editors, reviewers, and readers, when sample size, analytic, and reporting decisions have been made a priori versus post hoc, we now require submitting authors to report for all studies, in the Methods section(s), (1) how they determined sample size; (2) all data exclusions (if any); (3) all manipulations (if any); and (4) all measures used (see Simmons et al., 2012). Submitting authors will also confirm that they have reported this information when submitting their papers in ScholarOne, the online submission portal.

Sample-size determination may be new ground for some sexual scientists. Open-source resources for conducting a priori power analyses,⁵ which inform researchers how many participants they need to detect an effect of an expected size in a particular statistical design, are becoming more widely available and accessible (e.g., G*Power, Faul, Erdfelder, Lang, & Buchner, 2007; PANGEA, Westfall, 2016; and the bias-correcting power-analysis apps supplied by Maxwell, Delaney, & Kelley, in press; see also Judd, Westfall, & Kenny, 2017). Researchers struggling to determine an appropriate effect size to use in these calculations might calibrate their power analyses to the average effect size in their area of interest (e.g., r = .21 in social psychology; Richard, Bond, & Stokes-Zoota, 2003) or simply use the smallest effect size that they would deem worthy of their attention, effort, and resources (Lakens, 2014).⁶

Otherwise, meeting our new statistical disclosure requirement at JSR should be relatively straightforward, if you determined your sample size ahead of time, included all data, manipulations, and measures, succinctly say so in the manuscript, and choose the appropriate checklist selection at submission. And if you did not determine your sample size ahead of time, and/or did not include all data, manipulations, and measures, say so, and why. Disclosure of this kind is crucial for readers to be able to assign an appropriate level of confidence (or skepticism) to the effects reported in a paper.

Required Main-Text Reporting of Inferential Tests in Full APA Format

JSR also now requires authors to report the results of inferential statistical tests in the main body of the paper using full APA 6th Edition format (American Psychological Association, 2009, e.g., F (2, 132) = 4.47, p = .01, t (198) = 2.04, p = .04, χ² (3) = 3.35, p = .34, z = 2.77, p = .01). Complete reporting of this sort is necessary in order for readers to ensure that authors have correctly reported test statistic values, degrees of freedom, and p values (these values must “add up”). Studies using human eyeball coding (Bakker & Wicherts, 2011) and automated computer checks (Nuijten et al., 2016) suggest that statistical reporting errors (including those that affect declarations of statistical significance) are disturbingly common. Though JSR is not at this time considering implementing any sort of formal check of accurate statistical reporting (e.g., Sakaluk, Williams, & Biernat, 2014), we think it is important to ensure that the statistical reporting in JSR articles can be verified in this way by interested parties. For the meantime, researchers wishing to verify the accuracy of their own statistical reporting can do so easily using online apps like p-checker (http://shinyapps.org/apps/p-checker/; Schönbrodt, 2015) and statcheck (http://statcheck.io; Epskamp, Nuijten, & Rife, 2016).

Encouraged Reporting of Confidence Intervals and Effect Sizes

JSR strongly encourages researchers to include accompanying confidence intervals and effect sizes for all statistical tests. Confidence intervals are intervals for a point estimate (e.g., a mean, mean-difference, correlation coefficient, regression slope, or odds ratio) that will, in the long run, contain the population parameter for the point estimate a certain percentage of the time (e.g., 95% of 95% confidence intervals, with infinite random sampling of a population, will contain the population parameter).⁷ Effect sizes, alternatively, provide a sense of the magnitude of a given effect (independent of sample size or other influences of estimation precision), often in a standardized metric (e.g., zero-order correlation, standardized mean difference, odds ratio).

Though they convey different information, confidence intervals and effect sizes both help to provide more context and meaning to a statistical test, transcending what is conveyed by p values alone.⁸ A statistical test might be significant, for example, but an accompanying 95% confidence interval might be very broad, nearly encompassing the null value (e.g., 95% CI: .001, 2.45, for a mean-difference on a 5-point rating scale), suggesting that the effect under consideration was not examined with a sufficient level of precision (Cumming, 2014). Or alternatively, an effect might be statistically reliable but small (e.g., d = 0.08); a reader would then need to think critically about whether such a small effect was theoretically or applicably meaningful or trivial (e.g., Hyde, 2005; Prentice & Miller, 1992).

Meeting the encouraged standard of reporting confidence intervals and effect-size measures should be relatively straightforward. Confidence intervals for estimates of effects can often be produced using statistical software with the simple click of an option box, or the inclusion of a handful of characters of code. Producing measures of effect size can sometimes be a bit more onerous. Whereas effect-size measures for associations (e.g., r, R², β, ORs) are typically outputted automatically (or near automatically) by software, effect-size measures for group comparisons are either inconsistently available in statistical software (e.g., Cohen’s d) or when available are less intuitive (e.g., η², and η_p²). Lakens (2013) has published resources designed to help researchers better understand and easily compute effect-size measures for statistical analyses of group comparisons (i.e., t tests and ANOVAs).

Encouraged Reporting of Group Sizes, Means, Standard Deviations, and Zero-Order Correlations

Finally, JSR encourages the reporting of means and standard deviations for all group comparisons (alongside sample sizes per group). In the event that group comparisons were not the focus of a study, JSR encourages the reporting of zero-order correlations for all focal study variables. This reporting standard, also encouraged by the APA (2009), comes in light of the increasing recognition that a singleton study, no matter how skillfully conducted and analyzed, can never yield the sole and final word on a given effect. The generation of knowledge is instead gradual, produced through cumulative science consisting of multiple direct and conceptual replications, as well as extensions and boundary tests, of a given effect. And though there are a number of methods for reviewing related literature (Card, 2012), meta-analysis remains one of the most effective for evaluating the state of knowledge generated by a given area of research (Chan & Arvey, 2012), despite its limitations.⁹

Results from articles published in JSR cannot contribute to meta-analyses (and therefore cumulative science) if meta-analytic researchers cannot easily identify and extract (or calculate, if necessary) effect sizes that fall within the scope of their syntheses. Though the new JSR submission policy of encouraging reports of effect sizes should help toward this aim, a given paper may describe data relevant to an ongoing meta-analytic review that was not the primary focus of the original paper; the needed effect-size measure may therefore not be reported in the body of the main text. It is therefore imperative that the necessary data are available for aspiring meta-analysts, without requiring the ongoing assistance of the original authors who are, for the most part, unresponsive or uncooperative to such requests (Wicherts et al., 2006).

Researchers submitting manuscripts to JSR can report the necessary grouped descriptive statistics or zero-order correlations in a table, when appropriate, in light of the purpose of the manuscript and when space is available. If there are space constraints in the main article, JSR now publishes online supplementary tables that are linked to an article. Researchers can also make use of the free services available for sharing research materials, like the Open Science Framework (http://osf.io), which could host supplemental tables that meta-analysts could easily access. Meeting this encouraged reporting standard will help not only those conducting meta-analyses but the original researchers too; meta-analysts will inevitably cite the papers contributing estimates to their syntheses, which will increase the impact metrics of the original researchers.

Conclusions

Norms for conducting, interpreting, and reporting statistical results are changing in social and medical science fields, in light of widespread—and legitimate—concerns about the trustworthiness of empirical findings. The field of sexual science has largely avoided the debates surrounding these changes, but is starting to recognize and rise to the challenges of fostering a replicable scientific literature. Undisclosed conflicts of interest and capitalizing on flexibility of data analysis are two primary mechanisms through which less trustworthy effects can populate the sexual science literature. We therefore believe that the updated JSR policies promoting transparent disclosure of these features of research will serve as a small but important step toward ensuring the long-term replicability of sexuality research published in JSR.

Notes

¹ We credit Dr. Michael Seto sharing this development on Twitter, prompting discussions that led JSR to change its own submission guidelines.

² These new policies are not relevant for papers based on qualitative research.

³ The first author took an informal survey of attendees during a presentation (Sakaluk, 2017) who reported ever having engaged in any of these practices during their research career; he estimates ≥ 75% of attendees raised their hand (including himself).

⁴ Readers can see for themselves how easy it is to p-hack a truly null effect to significance using the simulator at http://shinyapps.org/apps/p-hacker/.

⁵ Note that post hoc power analyses based on observed effects (i.e., the kind that SPSS provides for you) are not informative for making sample-size decisions because they are determined by your p values.

⁶ Researchers interested in methods of “data peeking” that do not inflate Type I error rates should consult the sequential analysis technique described in Lakens’s (2014) paper.

⁷ Confidence intervals are often misunderstood (see Belia, Fidler, Williams, & Cumming, 2005) as providing an interval estimate that the researcher can be certain, with a certain level of confidence (e.g., 95%), the true population value lies between. A visualization by Magnusson, at http://rpsychologist.com/d3/CI/, helps render the correct interpretation of confidence intervals clearer.

⁸ The APA describes relying on confidence intervals as “in general, the best reporting strategy,” and effect-size measures as “almost always necessary to include” (p. 34).

⁹ Methodologists are beginning to notice that, somewhat ironically, meta-analyses are susceptible to the very same issues of replicability as individual studies (Lakens, Hilgard, & Staaks, 2016), alongside replicability-limiting issues that are unique to meta-analyses, which continue to go unsolved (e.g., accurately correcting for publication bias; see Carter, Schönbrodt, Gervais, & Hilgard, 2017).