The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

ABSTRACT

In response to growing concern about the reliability and reproducibility of published science, researchers have proposed adopting measures of “greater statistical stringency,” including suggestions to require larger sample sizes and to lower the highly criticized “p < 0.05” significance threshold. While pros and cons are vigorously debated, there has been little to no modeling of how adopting these measures might affect what type of science is published. In this article, we develop a novel optimality model that, given current incentives to publish, predicts a researcher’s most rational use of resources in terms of the number of studies to undertake, the statistical power to devote to each study, and the desirable prestudy odds to pursue. We then develop a methodology that allows one to estimate the reliability of published research by considering a distribution of preferred research strategies. Using this approach, we investigate the merits of adopting measures of “greater statistical stringency” with the goal of informing the ongoing debate.

KEYWORDS:

• Meta-research
• Null hypothesis significance Testing
• Publication
• Reliability
• Reproducibility
• Statistical power

1 Introduction

It is to be remarked that the theory here given rests on the supposition that the object of the investigation is the ascertainment of truth. When an investigation is made for the purpose of attaining personal distinction, the economics of the problem are entirely different. But that seems to be well enough understood by those engaged in that sort of investigation.

Note on the Theory of the Economy of Research,

Charles Sanders Peirce, 1879

In a highly cited essay, Ioannidis (2005) uses the Bayes theorem to claim that more than half of published research findings are false. While not all agree with the extent of this conclusion (e.g., Goodman and Greenland 2007; Leek and Jager 2017), recent large-scale efforts to reproduce published results in a number of different fields (economics, Camerer et al. 2016; psychology, OpenScienceCollaboration 2015; oncology, Begley and Ellis 2012), have also raised concerns about the reliability and reproducibility of published science. Unreliable research not only reduces the credibility of science, but is also very costly (Freedman, Cockburn, and Simcoe 2015) and as such, addressing the underlying issues is of “vital importance” (Spiegelhalter 2017). Many researchers have recently proposed adopting measures of “greater statistical stringency,” including suggestions to require larger sample sizes and to lower the highly criticized “p < 0.05” significance threshold. In statistical terms, this represents selecting lower levels for acceptable Type I and Type II error rates. The main argument against “greater statistical stringency” is that these changes would increase the costs of a study and ultimately reduce the number of studies conducted.

Consider the debate about lowering the significance threshold in response to the work of Johnson (2013), who, based on the correspondence between uniformly most powerful Bayesian tests and classical significance tests, recommends lowering significance thresholds by a factor of 10 (e.g., from p < 0.05 to p < 0.005). Gaudart et al. (2014), voicing a common objection, contend that such a reduction in the allowable Type I error rate will result in inevitable increases to the Type II error rate. While larger sample sizes could compensate, this can be costly: “increasing the size of clinical trials will reduce their feasibility and increase their duration” (Gaudart et al. 2014). In the view of Johnson (2014), this may not necessarily be such a bad thing, pointing to the excess of false positives (FPs) and the idea that (in the context of clinical trials) “too many ineffective drugs are subjected to phase III testing […] wast[ing] enormous human and financial resources.”

More recently, a highly publicized call by over seven dozen authors to “redefine statistical significance” has made a similar suggestion: lower the threshold of what is considered “significant” for claims of new discoveries from p < 0.05 to p < 0.005 (Benjamin et al. 2018). This has prompted a familiar response (e.g., Wei and Chen 2018). Amrhein, Korner-Nievergelt, and Roth (2017) review the arguments for and against more stringent thresholds for significance and conclude that: “[v]ery possibly, more stringent thresholds would lead to even more results being left unpublished, enhancing publication bias. […] [W]hile aiming at making our published claims more reliable, requesting more stringent fixed thresholds would achieve quite the opposite.”

There is also substantial disagreement about suggestions to require larger sample sizes. In some fields, showing that a study has a sufficient sample size (i.e., high power) is common practice and an expected requirement for funding and/or publication, while in others it rarely occurs. For example, Charles et al. (2009) found that 95% of randomized controlled trials (RCTs) report sample size calculations. In contrast, only a tiny fraction of articles in some fields –about 3% for psychology, and about 2% for toxicology—justify their sample size (Bosker, Mudge, and Munkittrick 2013; Fritz, Scherndl, and Kühberger 2013). The number is only marginally higher in conservation biology at approximately 8% (Fidler et al. 2006).

One argument is that, once a significant finding is achieved, the size of a study is no longer relevant. Aycaguer and Galbán (2013) explain as follows: “If a study finds important information by blind luck instead of good planning, I still want to know the results.” Another viewpoint is that, while far from ideal, underpowered studies should be published since cumulatively, they can contribute to useful findings (Walker 1995). Others disagree and contend that small sample sizes undermine the reliability of published science (Button et al. 2013a; Dumas-Mallet et al. 2017; Nord et al. 2017). In the context of clinical trials, IntHout, Ioannidis, and Borm (2016) review the many conflicting opinions about whether trials with suboptimal power are justified and conclude that, in circumstances when evidence for efficacy can be effectively combined across a series of trials (e.g., via meta-analysis), small sample sizes might be justified.

Despite the long-running and ongoing debates on significance thresholds and sample size requirements, there has been little to no modeling of how changes to a publication policy might affect what type of studies are pursued, the incentive structures driving research, and ultimately, the reliability of published science. One example is Borm, den Heijer, and Zielhuis (2009) who conclude, based on simulation studies, that the consequences of publication bias do not warrant the exclusion of trials with low power. Another recent example is Higginson and Munafò (2016), who, based on results from an optimality model of the “scientific ecosystem,” conclude that in order to “improve the scientific value of research” peer-reviewed publications should indeed require larger sample sizes, lower the α significance threshold, and give “less weight to strikingly novel findings.” Our work here aims to build upon these modeling efforts to better inform the ongoing discussion on the reproducibility of published science.

The model and methodology we present seeks to add three features absent from the model of Higginson and Munafò (2016). While these authors consider how researchers balance available resources between exploratory and confirmatory studies, this simple dichotomy does not allow for a detailed assessment of the willingness of researchers to pursue high-risk studies (those studies that are, a priori, unlikely to result in a statistically significant finding). Our approach addresses this issue by considering a continuous spectrum of a priori risk, that is, the “pre-study probability.” Second, Higginson and Munafò (2016) define the “total fitness of a researcher” (i.e., the payoff for a given research strategy) with diminishing returns for confirmatory studies, but not for exploratory studies. This choice, however well intended, has problematic repercussions for their optimality model. (Under their framework, the optimal research strategy will depend on T, an arbitrary total budget parameter.) Finally, by failing to incorporate the number of correct studies that go unpublished within their metric for the value of scientific research, many potential downsides of adopting measures to increase statistical stringency are ignored. Other differences between our approach and previous ones will be made evident and include: considering outcomes in terms of distributional differences, and specific modeling of how sample size requirements are implemented.

The remainder of this article is structured as follows. In Section 2, we describe the model and methodology proposed to evaluate different publication policies. We also list a number of metrics of interest and look to recent meta-research analyses to calibrate our model. In Section 3, we use the proposed methodology to evaluate the potential impact of lowering the significance threshold; and in Section 4, the impact of requiring larger sample sizes. Finally, in Section 5, we conclude with suggestions as to how publication policies can be defined to best improve the reliability of published research.

2 Methods

Recently, economic models have been rather useful for evaluating proposed research reforms (Gall, Ioannidis, and Maniadis 2017). However, modeling of how resources ought to be allocated among research projects is not new. See, for example, the work of Greenwald (1975), Dasgupta and Maskin (1987), McLaughlin (2011), and Miller and Ulrich (2016). As noted in the introduction, our framework for modeling the scientific ecosystem is closest in spirit to that of Higginson and Munafò (2016) who formulate a relationship between a researcher’s strategy and their payoff, with the strategy involving a choice of mix between exploratory and confirmatory studies, and a choice of pursuing fewer studies with larger samples or more studies with smaller samples.

The publication process is complex and includes both objective and subjective considerations of efficacy and relevance. The title of this article was chosen specifically to emphasize this point (Gornitzki, Larsson, and Fadeel 2015). A large, complicated human process like that of scientific publication cannot be entirely reduced to metrics and numbers: there are often financial, political and even cultural reasons for an article being accepted or rejected for publication. With this in mind, the model presented here should not be seen as an attempt to precisely map out the peer-review process, but rather, as a useful tool for determining the consequences of implementing different publication policies.

Within our optimality model, many assumptions and simplifications are made. Most importantly, we assume that each researcher must make decisions consisting of only two choices: what statistical power (i.e., sample size) to adopt and what “prestudy probability” to pursue. Before elaborating further, let us briefly discuss these two concepts.

2.1 Statistical Power

Increasing statistical power by conducting studies with larger sample sizes would undeniably result in more published research being true. However, these improvements may only prove modest, given current publication guidelines. When we consider the perspectives of both researchers and journal editors, it is not surprising that statistical power has not improved substantially (Smaldino and McElreath 2016; Lamberink et al. 2018) despite being highlighted as an issue over five decades ago (Cohen 1962).

In practice, multiple factors can influence sample size determination (Lenth 2001; Hazra and Gogtay 2016; Roos 2017). Strictly in terms of publication prospects however, there is little incentive to conduct high-powered studies: basic logic suggests that the likelihood of publication is only minimally affected by power. To illustrate, consider a large number of hypotheses tested, out of which 10% are truly nonnull. Under the assumption that only (and all) positive results are published with $\alpha =0.05$ (which may in fact be realistic in certain fields, Fanelli 2011), simple analytical calculation shows that increasing average power from an “unacceptably low” 55% to a “respectable” 85% (at the cost of more than doubling sample size), results in only a minimal increase in the likelihood of publication: from 10% to 13%. Moreover, the proportion of true findings amongst those published is only increased modestly: from 55% to 64%. Indeed, a main finding of Higginson and Munafò (2016) is that the rational strategy of a researcher is to “carry out lots of underpowered small studies to maximize their number of publications, even though this means around half will be false positives.” This result is in line with the views of many; see, for example, Bakker, van Dijk, and Wicherts (2012), Button et al. (2013b), and Gervais et al. (2015).

From a journal’s perspective, there is also little incentive to require larger sample sizes as a requirement for publication. There are minimal consequences from publishing false claims. Fraley and Vazire (2014) review the publication history of six major journals in social-personality psychology and find that “journals that have the highest impact [factor] also tend to publish studies that have smaller samples.” This finding is in agreement with Szucs and Ioannidis (2016) who conclude that, in the fields of cognitive neuroscience and psychology, journal impact factors are negatively correlated with statistical power; see also Brembs, Button, and Munafò (2013).

2.2 Prestudy Probability

We use the term “prestudy probability” (psp) as shorthand for the a priori probability that a study’s null hypothesis is false. In this sense, highly exploratory research will typically have very low psp, whereas confirmatory studies will have a relatively high psp. Studies with low psp are not problematic per se. To the contrary, there are undeniable benefits to pursuing “long-shot” novel ideas that are very unlikely to work out, see Cohen (2017). While replication studies (i.e., studies with higher psp) may be useful to a certain extent, there is little benefit in confirming a result that is already widely accepted. As Button et al. (2013a) note: “As R [the pre-study odds] increases […] the incremental value of further research decreases.” Most scientific journals no doubt take this into account in deciding what to publish, with more surprising results more likely to be published. This state of affairs persists, despite recent calls for more replication studies (e.g., Moonesinghe, Khoury, and Janssens 2007). Indeed, replication studies are still often rejected on the grounds of “lack of novelty;” see Makel, Plucker, and Hegarty (2012), Yeung (2017), and Martin and Clarke (2017). As such, researchers deciding which hypotheses to pursue toward publication will likely emphasize those with lower psp.

Recognize that the lower the psp, the less likely a “statistically significant” finding is to be true. As such, we are bound to a “seemingly inescapable trade-off” (Fiedler 2017) between the novel and the reliable. Journal editors face a difficult choice. Either publish studies that are surprising and exciting yet most likely false, or publish reliable studies which do little to advance our knowledge. Based on a belief that both ends of this spectrum are equally valuable, Higginson and Munafò (2016) conclude that, in order to increase reliability, current incentive structures should be redesigned, “giving less weight to strikingly novel findings.” This is in agreement with the view of Hagen (2016) who writes: “If we are truly concerned about scientific reproducibility, then we need to reexamine the current emphasis on novelty and its role in the scientific process.”

2.3 Model Framework

We describe our model framework in five simple steps.

1. We assume, for simplicity, that all studies test a null hypothesis of equal population means against a two-sided alternative, with a standard two-sample Student t-test. Each study has an equal number of observations per sample ($n_1=n_2$; $n=n_1+n_2$). Furthermore, let us assume that a study can have one of only two results: (1) positive (p-value $<\alpha$), or (2) negative (p-value $\ge \alpha$). Given the true effect size, ${\mu }_d={\mu }_2-{\mu }_1$ (the difference in population means), ${\sigma }^2$, the common population variance, and the sample size, n, we can easily calculate the probability of a significant result, using the standard formula for power.¹

   Then, for a given true effect size of δ or zero, we have the probability of a TP, FN, FP, and TN, equal to: $\text{Pr}(\text{TP})=\text{Pr}(\text{Positive}|{\mu }_d=\delta ),\text{Pr}(\text{FN})=\text{Pr}(\text{Negative}|{\mu }_d=\delta ),\text{Pr}(\text{FP})=\text{Pr}(\text{Positive}|{\mu }_d=0)$, and $\text{Pr}(\text{TN})=\text{Pr}(\text{Negative}|{\mu }_d=0)$, respectively.

2. Next, we consider a large number of studies, n_S, each with a total sample size of n. Of these n_S studies, only a fraction, psp (where psp is the prestudy probability), have a true effect size of ${\mu }_d=\delta$. For the remaining $(1-\mathit{psp})·n_S$ studies, we have ${\mu }_d=0$. Note that for a given sample size, n, these n_S studies are each “powered” at level $\mathit{pwr}=\mathit{Pr}(\text{TP})$. Throughout this article, we focus on scenarios, where ${\sigma }^2=1$ and $\delta =0.20$ or $\delta =0.43$. These choices reference the analyses of Lamberink et al. (2018) and Richard, Bond Jr, and Stokes-Zoota (2003). Lamberink et al. (2018), based on an empirical analysis of over one hundred-thousand clinical trials conducted between 1975 and 2014, estimate that the median effect size of clinical trials is approximately a Cohen’s d = 0.20. In a similar analysis based on data from over 25,000 social/personality studies, Richard, Bond Jr, and Stokes-Zoota (2003) estimate that the mean effect size in social psychology research is of a Cohen’s d = 0.43.

3. We also label each study as either published (PUB) or unpublished (UN) for a total of 8 distinct categories ($=$ 2 (positive, negative) × 2 (true and false) × 2 (published and unpublished)). One can determine the expected number of studies (out of a total of n_S studies) in each category by simple arithmetic. Table 1 lists the equations for each of the eight categories with A equal to the probability of publication for a positive result, and B equal to the probability of publication for a negative result. Throughout this article, we will always assume that only positive studies are published, hence, we set B = 0. Initially, we will let the probability of publication of a positive study, A, depend on the psp according to the function $A={(1-\mathit{psp})}^m$, where m $\in (0,\infty )$ is a tuning parameter. This simple, decreasing function of psp represents a system in which positive results with lower psp are more likely to be published on the basis of novelty. A higher value of m indicates that higher psp studies (i.e., low-risk hypotheses) are less likely to be published. Consequently, a higher value of m implies a lower overall publication rate. In Section 4, we will define A differently, such that the probability of publication depends on the values of both psp and pwr.

   Table 1 Equations for the expected number of studies (out of a total of n_S studies) for each of the eight categories; with A $=$ prob. of publication for a positive result and B $=$ prob. of publication for a negative result. The number of studies n_S, changes for different values of pwr. We have that: $n_S=k+n{(\mathit{pwr})}^{-1}T$, where n(pwr) is the required sample size to obtain a power of pwr.

   For a fixed resources, T,
   and a given (psp, pwr),	Equation
   the expected number of…	(note: throughout this article, we take B = 0)
   True positives (TPs)
   published	${\text{TP}}_{\text{PUB}}(\mathit{psp},\mathit{pwr})=\mathit{psp}·n_S·A·\text{Pr}(\text{TP})$
   TPs unpublished	$T{P}_{\mathit{UN}}(\mathit{psp},\mathit{pwr})=\mathit{psp}·n_S·(1-A)·\text{Pr}(\text{TP})$
   False negatives (FNs)
   published	$F{N}_{\text{PUB}}(\mathit{psp},\mathit{pwr})=\mathit{psp}·n_S·B·\text{Pr}(\text{FN})$
   FNs unpublished	${\text{FN}}_{\mathit{UN}}(\mathit{psp},\mathit{pwr})=\mathit{psp}·n_S·(1-B)·\text{Pr}(\text{FN})$
   FPs published	${\text{FP}}_{\text{PUB}}(\mathit{psp},\mathit{pwr})=(1-\mathit{psp})·n_S·A·\mathit{Pr}(\mathit{FP})$
   FPs unpublished	${\text{FP}}_{\text{UN}}(\mathit{psp},\mathit{pwr})=(1-\mathit{psp})·n_S·(1-A)·\text{Pr}(\text{FP})$
   True negatives (TNs)
   published	${\text{TN}}_{\text{PUB}}(\mathit{psp},\mathit{pwr})=(1-\mathit{psp})·n_S·B·\text{Pr}(\text{TN})$
   TNs unpublished	${\text{TN}}_{\text{UN}}(\mathit{psp},\mathit{pwr})=(1-\mathit{psp})·n_S·(1-B)·\text{Pr}(\text{TN})$
   Publications	$\text{ENP}(\mathit{psp},\mathit{pwr})=T{P}_{\text{PUB}}+F{N}_{\text{PUB}}+F{P}_{\text{PUB}}+T{N}_{\text{PUB}}$

4. We determine the total number of studies, n_S, based on three parameters: T, the total resources available (in units of observations sampled); k, the fixed cost per study (also expressed in equivalent units of observations sampled); and n, the total sample size per study. Consequently, as in Higginson and Munafò (2016), $n_S={(k+n)}^{-1}T$. Then for any given level of power, pwr, we can easily obtain the necessary sample size per study, n, (using established sample size formula for two-sample t-tests, see equations 1(1) $$\text{Pr}\left(\text{Positive}\right|{\mu }_d)=(1-F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}\left(t_{\alpha /2}^{*}\right))+F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}(-t_{\alpha /2}^{*}),$$(1) and 2(2) $$n={\text{argmin}}_n\left(\right|\mathit{pwr}-(1-F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}(t_{\alpha /2}^{*}\left)\right)+F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}(-t_{\alpha /2}^{*})\left|\right)$$(2) ), and a resulting total number of studies, n_S. Keep in mind that n_S, the number of studies, is a function of pwr. Throughout this article, when necessary, we take T = 100,000. However, note that when comparing the outcomes of different publication policies, this choice is entirely irrelevant. Consider that when k is small, the cost of data, relative to the total cost of a study, is large. Conversely, when k is large, the relative cost of increasing the study sample size is small. For example, suppose k = 20 and $\delta =0.20$, then increasing the power from 0.55 to 0.85, implies doubling the total cost of the study. If instead k = 500 (all else being equal), this increase in power corresponds to only a 50% increase in the total study cost.

5. Finally, let us define a “research strategy” to be a given pair of values for (psp, pwr) within ([0,1] × [α,1]). Then, for a given research strategy we can easily calculate the total expected number of publications (ENP):(3) $$\text{ENP}(\mathit{psp},\mathit{pwr})={\text{TP}}_{\text{PUB}}+{\text{FP}}_{\text{PUB}}+{\text{TN}}_{\text{PUB}}+{\text{FN}}_{\text{PUB}}.$$(3)

Figure 1 illustrates how $\text{ENP}$ is calculated for different values of psp and pwr, with fixed $\alpha =0.05$, k = 100, and m = 3.

Fig. 1 The nine-gridded squares represent nine different research strategies; from left to right, we have $\mathit{psp}=0.2$, 0.5, and 0.8; from bottom to top, we have pwr $=$ 0.2, 0.5, and 0.8. For each pair of values for (psp, pwr), the different colored areas represent the probabilities of a TP, a FP, a TN, and a FN. The Expected Number of Publications ($\text{ENP}$) is calculated by multiplying (1) the probability of a positive finding ($\text{Pr}(\text{Positive})=\mathit{psp}·\text{Pr}(\text{TP})+(1-\mathit{psp})·\text{Pr}(\text{FP})$), (2) the number of studies ($n_S={\{k+n(\mathit{pwr})\}}^{-1}T$), and (3) the probability of publication ($A={(1-\mathit{psp})}^m$). The figure corresponds to an ecosystem defined by $\alpha =0.05$, m = 3, k = 100, B = 0, $\mu =0.20,{\sigma }^2=1$, and T = 100,000.

With this set-up in hand, suppose now a researcher pursues -consciously or unconsciously- strategies that maximize the expected number of publications (Charlton and Andras 2006). This may not be an entirely unreasonable assumption considering the influence of Goodhart’s Law (“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”) on academia, see Fire and Guestrin (2018) and van Dijk, Manor, and Carey (2014). Figure 2 shows the analytical calculations of how $\text{ENP}$ changes over a range of values of psp and pwr and under different fixed values for k and m. (Note how the central panel of Figure 2 (k = 100, m = 3) corresponds to Figure 1.) Depending on k and m, the value of (psp, pwr) that maximizes $\text{ENP}$ can change substantially. With larger k, higher powered strategies will yield a greater $\text{ENP}$; with smaller m, optimal strategies are those with higher psp. It is interesting to observe how the optimal strategy changes under different scenarios. However, it may be more informative to consider a distribution of preferred strategies. This may also be a more realistic approach. While rational researchers may be drawn toward optimal strategies, surely scientists are not willing and/or able to precisely identify these.

Fig. 2 Plots show the number of expected publications, $\text{ENP}$, for different values of $\text{PSP}$ and $\text{PWR}$. The printed numbers correspond to values of ENP at $\mathit{psp}=0.2$, 0.5, and 0.8; and $\mathit{pwr}=0.2$, 0.5, and 0.8. Each panel represents one ecosystem defined by $\alpha =0.05,A={(1-\mathit{psp})}^m,$ and B = 0, with k and m as indicated by column and row labels, respectively. Depending on k and m, the value of ($\text{PSP},\text{PWR}$) that maximizes $\text{ENP}$ can change substantially. With a higher k, higher powered strategies will yield a greater $\text{ENP}$; with smaller m, optimal strategies are those with higher $\text{PSP}$. Assuming that researchers behave based on optimizing the use of their resources, it is interesting to observe how the“optimal strategy” changes under different scenarios.

Taking the incentive to publish as a starting point, we assume scientists are more likely to spend resources on studies whose psp and pwr produce a higher expected number of publications. So we consider a distribution of resource allocation across study characteristics whose density is proportional to the expected number of publications, that is, $f_{\text{RES}}(\mathit{psp},\mathit{pwr})\propto \text{ENP}(\mathit{psp},\mathit{pwr})$. We emphasize that from a funder’s viewpoint, $f_{\text{RES}}()$ describes where dollars are being spent on the (psp, pwr) plane. For example, consider the nine strategies illustrated in Figure 1. Given the relative ENP values, we expect the expenditure on $(\mathit{psp},\mathit{pwr})=(0.2,0.2)$ studies to be 18 times greater than that on $(\mathit{psp},\mathit{pwr})=(0.8,0.5)$ studies.

In the appendix, we give expressions for two distributions that follow on from $f_{\text{RES}}()$. The first is $f_{\text{ATM}}(\mathit{psp},\mathit{pwr})$ which describes where attempted studies (not resources) fall on the (psp, pwr) plane, presuming that resources are allocated according to $f_{\text{RES}}()$. Since higher powered studies are most costly, $f_{\text{RES}}()$ and $f_{\text{ATM}}()$ are not the same. Hence, $f_{\text{ATM}}()$ describes what kinds of studies are attempted more frequently.

Of course not all attempted studies are published. Thus, in the appendix we also give an expression for $f_{\text{PUB}}(\mathit{pwp},\mathit{pwr})$, the density of (psp, pwr) across published studies that is implied when $f_{\text{ATM}}()$ describes the characteristics of attempted studies. Note then that each of $f_{\text{RES}}(),f_{\text{ATM}}()$, and $f_{\text{PUB}}()$ describes relatively favored and disfavored (psp, pwr) combinations. However, the distinction between the three is important. While $f_{\text{RES}}()$ describes resource deployment, $f_{\text{ATM}}()$ describes the resulting constellation of attempted studies, and $f_{\text{PUB}}()$ describes the resulting constellation of published studies.

Armed with $f_{\text{RES}}(),f_{\text{ATM}}()$, and $f_{\text{PUB}}()$, we can investigate the properties of a given scientific ecosystem, and how these properties vary across ecosystems. Specifically, an ecosystem is specified by choices of α, k, m, A, and B.

For any specification, properties of the three distributions are readily computed via two-dimensional numerical integration using a fine (200 by 200) grid of (psp, pwr) values. We do not require any simulation to obtain our results (except for those results relating to the accuracy of published research, see details in the appendix). To illustrate, consider the much coarser 3 by 3 grid plotted in Figure 1 and see how ENP could be easily calculated by summation across the two dimensions.

2.4 Ecosystem Metrics

We will evaluate the merits of different publication policies on the basis of the following six ecosystem metrics of interest. In the appendix, we provide further details on these metrics and introduce some compact notation that will be useful for their calculation from $f_{\text{RES}}()$ and its by-products.

1. The Publication Rate (PR). The ratio of the number of published studies over the number of attempted studies is of evident interest.

2. The Reliability (REL). The proportion of published findings that are correct is a highly relevant metric for the scientific ecosystem. Ideally, we wish to see a literature with as few FPs as possible.

3. The Breakthrough Discoveries (DSCV). The ability of the scientific ecosystem to produce breakthrough findings is an important attribute. Here, a true breakthrough result is defined as a TP and published study that results from a psp value below a threshold of 0.05. Note that we will calculate the total number of true breakthrough discoveries, DSCV, for each ecosystem for T units of resources. We will also note the reliability of breakthrough discoveries ($D_{\text{REL}}$), the proportion of positive and published studies with psp < 0.05 that are true.

4. The Balance Between Exploration and Confirmation ( ${\text{IQpsp}}_{\text{PUB}}$). The interquartile range of the distribution of psp values amongst published studies provides an assessment of the much discussed balance between exploratory and confirmatory research; see Sakaluk (2016) and Kimmelman, Mogil, and Dirnagl (2014).

5. The Median Power of Published Studies ( $\mathit{Mpw}{r}_{\text{PUB}}$). As mentioned earlier, higher powered research leads to fewer yet more reliable publications.

6. The Accuracy of Published Research. We will report the mean absolute estimated effect size in published studies, $|\widehat{{\mu }_d}{|}_{\text{PUB}}$, as well as the proportion of published studies for which the estimated effect size is negative (${\widehat{{\mu }_d}}_{\text{PUB}}^{\text{neg}}$). It is well known that, due to a combination of publication bias and low powered studies, published effect sizes will be inflated, see Ioannidis (2008); Lane and Dunlap (1978). In addition, we will report the exaggeration ratio (the expected Type M error), and the Type S error rate (the proportion of published studies with an error in sign), see Gelman and Carlin (2014).

2.5 Model Calibration

Let us begin by assessing whether or not our basic model framework is well calibrated. Are our modeling choices (specifically the choice of values to consider for parameters k and m) at all consistent with what is known about real conditions? In order to answer this important question, we considered a large number of different scenarios in which α is held fixed at 0.05, and both the fixed-cost parameter, k, and the novelty parameter, m, take various values; see Table 2.

Table 2 For ecosystems defined by fixed $\alpha =0.05,A={(1-\mathit{psp})}^m,$ and B = 0, and varying values of k, m, and δ, the table lists estimates for the metrics of interest.

δ	k	m	$\mathit{PR}$	$\mathit{REL}$	$\mathit{DSCV}$	$D_{\text{REL}}$	$\mathit{Mpw}{r}_{\text{PUB}}$	${\text{IQpsp}}_{\text{PUB}}$	$[q_{25th},q_{75th}]$
0.20	20	1	0.06	0.62	0.17	0.07	0.20	0.35	[0.19, 0.54]
0.20	20	3	0.03	0.36	0.38	0.07	0.12	0.19	[0.07, 0.26]
0.20	20	9	0.03	0.15	0.86	0.06	0.09	0.07	[0.02, 0.09]
0.20	100	1	0.08	0.75	0.09	0.12	0.42	0.33	[0.25, 0.58]
0.20	100	3	0.04	0.52	0.22	0.12	0.32	0.20	[0.09, 0.29]
0.20	100	9	0.03	0.25	0.51	0.10	0.21	0.08	[0.03, 0.10]
0.20	500	1	0.11	0.83	0.04	0.18	0.60	0.31	[0.29, 0.60]
0.20	500	3	0.05	0.65	0.10	0.18	0.55	0.21	[0.12, 0.33]
0.20	500	9	0.03	0.36	0.26	0.16	0.44	0.09	[0.03, 0.12]
0.43	20	1	0.08	0.76	0.42	0.12	0.42	0.33	[0.25, 0.58]
0.43	20	3	0.04	0.53	1.00	0.12	0.32	0.20	[0.09, 0.29]
0.43	20	9	0.03	0.25	2.37	0.11	0.21	0.08	[0.03, 0.10]
0.43	100	1	0.11	0.83	0.21	0.18	0.59	0.31	[0.29, 0.60]
0.43	100	3	0.05	0.65	0.50	0.18	0.54	0.21	[0.12, 0.33]
0.43	100	9	0.03	0.36	1.24	0.16	0.43	0.08	[0.03, 0.11]
0.43	500	1	0.13	0.86	0.07	0.23	0.71	0.31	[0.30, 0.61]
0.43	500	3	0.06	0.71	0.17	0.22	0.68	0.21	[0.13, 0.34]
0.43	500	9	0.03	0.43	0.44	0.20	0.60	0.09	[0.03, 0.12]

NOTES: These include the publication rate (PR), the reliability (REL), the number of true breakthrough discoveries ($\mathit{DSCV}$), the median power of published studies ($\mathit{Mpw}{r}_{\text{PUB}}$), and the interquartile range of the distribution of psp values amongst published studies (${\text{IQpsp}}_{\text{PUB}}$), with corresponding 25th and 75th quartiles. Additionally, the table lists the reliability amongst the subset of published studies for which psp < 0.05 ($D_{\text{REL}}$).

There is a recognized lack of empirical research measuring the relationship between experimental costs and sample sizes (Roos 2017), and it is even more difficult to empirically access appropriate values for m. However, we were able to choose suitable values for m (= 1, 3, and 9) and k (= 20, 100, and 500) based on how our model outputs compared with empirical estimates in the meta-research literature. Specifically, values were chosen to exhibit a sufficiently wide range and so that (1) the reliability, (2) the power, and (3) the publication rate agreed, at least to a certain extent, with what has been observed empirically. Consider the following.

1. Due to the success of several recent large-scale reproducibility projects, we are able to obtain reasonable approximations for the reliability of published research in different scientific fields. For example, Begley and Ioannidis (2015) estimate that the reliability (i.e., TP rate) of social science experiments published in the journals Nature and Science is approximately 67%. Based on the large-scale reproducibility project of OpenScienceCollaboration (2015), Wilson and Wixted (2018) estimate that the reliability of published studies in well-respected social-psychology journals is approximately 49%, and in cognitive psychology journals, approximately 81%. For the chosen values of k and m, our model produces reliability measures ranging from 15% to 83%, for $\delta =0.2$, and from 25% to 86%, for $\delta =0.43$. Note that this wide range of values includes some reliability percentages that seem rather low (e.g., 15% and 25%).

2. There have also been a number of recent attempts to estimate the typical power of published research using meta-analytic effect sizes. For example, Button et al. (2013b) estimate that the median statistical power in published neuroscience is approximately 21% and Dumas-Mallet et al. (2017), using a similar methodology, conclude that the median statistical power in the biomedical sciences ranges from about 9% to 30% depending on the disease under study. Most recently, Lamberink et al. (2018) analyzed the literature of RCTs and determined that the median power is approximately 9% overall (and about 20% in the subset of RCTs included in significant meta-analyses). For the chosen values of k and m, our model produces a simulated published literature with median power ranging from 9% to 60%, for $\delta =0.2$, and from 21% to 71%, for $\delta =0.43$.

3. While the publication rate numbers we obtain may appear rather low, ranging from 3% to 13%, consider that Siler, Lee, and Bero (2015), in a systematic review of manuscripts submitted to three leading medical journals, observed a publication rate of 6.2%. In a review of top psychology journals, Lee and Schunn (2011) found that acceptance rates ranged between 14% and 32%. It is important to recognize that these empirical estimates are calculated based on the subset of studies that are submitted for publication. If researchers are self-selecting their “most publishable” work for submission to journals, then these estimates will substantially overestimate the true publication rate of all studies. Song, Loke, and Hooper (2014) estimate that about 85% of unpublished studies are unpublished for the simple reason that they are never submitted for publication. Our model also assumes that negative studies are never published (B = 0) and this (not entirely realistic) assumption also contributes to the low publication rate numbers we obtain. Senn (2012) provides substantial insights on both these related issues: the level of publication bias, and how researchers may choose to submit research based on the perceived probability of acceptance. Note that while the scenarios with higher publication rates may appear more plausible, these scenarios have very (perhaps unreasonably) high values for reliability and median power.

Regardless of the values chosen for k and m, the overall trends for our metrics of interest will be similar. See Table 2. As k increases, the reliability of published research (REL) will increase, while the number of true breakthrough discoveries ($\mathit{DSCV}$) decreases. As m increases, reliability (REL) decreases while the number of true breakthrough discoveries ($\mathit{DSCV}$) increases. This strikes us as both reasonable and realistic. When a greater emphasis is placed on novelty (i.e., when m is larger), there will be a greater number of smaller, high-risk studies published. While these publications are less reliable, they are more numerous and more likely to produce a breakthrough finding. When data are more affordable relative to overall study cost (i.e., when k is larger), there will be fewer, larger studies and as a result the published literature will be more reliable but less likely to produce a breakthrough finding.

Before moving on, let us also briefly consider how the accuracy of published effect sizes changes with k and m, see Table 3. Overall, we see that the average of the absolute published effect size, $|\widehat{{\mu }_d}{|}_{\text{PUB}}$, is always much larger than δ. Furthermore, we see that a large proportion of published effect sizes are negative. This is to be expected when psp values are small and many published studies are FPs. For the subset of studies that are positive, published, and true, we see that the Type M and the Type S errors are greatest when k is small, m is large, and δ is small. This is the same configuration that produces low-powered publications. Indeed, Type M and Type S errors are known to occur more frequently when power is low (Gelman and Carlin 2014).

Table 3 For ecosystems defined by fixed $\alpha =0.05,A={(1-\mathit{psp})}^m,$ and B = 0, and varying values of k, m, and δ, the table lists estimates for the metrics of interest related to the accuracy of the published effects sizes.

δ	m	k	$\mathit{Mpw}{r}_{\text{PUB}}$	${\widehat{{\mu }_d}}_{\text{PUB}}^{\mathit{neg}}$	$|\widehat{{\mu }_d}{|}_{\text{PUB}}$	Type M err.	Type S err.
0.20	1	20	0.20	0.21	0.60	2.42	0.03
0.20	3	20	0.12	0.31	0.72	2.70	0.04
0.20	9	20	0.09	0.43	0.80	2.77	0.04
0.20	1	100	0.42	0.14	0.37	1.68	0.01
0.20	3	100	0.32	0.23	0.45	1.81	0.01
0.20	9	100	0.21	0.37	0.51	1.92	0.01
0.20	1	500	0.60	0.09	0.29	1.40	0.00
0.20	3	500	0.55	0.17	0.31	1.40	0.00
0.20	9	500	0.44	0.33	0.35	1.48	0.00
0.43	1	20	0.42	0.14	0.75	1.63	0.01
0.43	3	20	0.32	0.21	0.80	1.65	0.00
0.43	9	20	0.21	0.39	0.90	1.74	0.01
0.43	1	100	0.59	0.09	0.60	1.36	0.00
0.43	3	100	0.54	0.18	0.64	1.43	0.00
0.43	9	100	0.43	0.32	0.69	1.47	0.00
0.43	1	500	0.71	0.08	0.54	1.29	0.00
0.43	3	500	0.68	0.17	0.56	1.29	0.00
0.43	9	500	0.60	0.28	0.57	1.30	0.00

NOTES: These include the median power of published studies ($\mathit{Mpw}{r}_{\text{PUB}}$), the mean absolute published effect size ($|\widehat{{\mu }_d}{|}_{P\mathrm{U}B}$), the proportion of published effect sizes that are negative (${\widehat{{\mu }_d}}_{\text{PUB}}^{\text{neg}}$), the “Type S” error, and the “Type M” error.

3 The Effects of Adopting Lower Significance Thresholds

In this section, we investigate the impact of adopting lower significance thresholds. How might the published literature be different if the $\alpha =0.05$ threshold was lowered? For positive studies, we will assume that the sample size of a study does not affect the likelihood of publication (at least not directly) and that studies with lower psp are more likely to be published according to the simple function introduced earlier, $A={(1-\mathit{psp})}^m$. We compute the metrics of interest for 54 different ecosystems. Each ecosystem is uniquely defined with either $\delta =0.2$ or $\delta =0.43$, with one of three possible values for m ($=$ 1, 3, 9), with one of three possible values for k ($=$ 20, 100, 500), and most importantly, with one of three possible values for the α significance threshold ($=$ 0.005, 0.020, 0.050).

3.1 Results

In Table 4, we note how the various metrics change with $\alpha =0.005$ relative to $\alpha =0.05$, by reporting ratios. While we focus our discussion specifically on scenarios for which $\delta =0.20$, the results for $\delta =0.43$ are all very similar. Figures in the supplemental material plot the complete results. Based on our results, we can make the following conclusions on the impact of adopting a lower, more stringent, significance threshold.

Table 4 A comparison between ecosystems with $\alpha =0.05$ and ecosystems with $\alpha =0.005$.

			Change in	Change in	Change in	Change in	Change in	Change in
δ	k	m	PR	REL	DSCV	$D_{\text{REL}}$	$\mathit{Mpw}{r}_{\text{PUB}}$	${\text{IQpsp}}_{\text{PUB}}$
0.20	20	1	1.14	1.58	0.12	7.57	2.73	0.83
0.20	20	3	0.71	2.58	0.15	7.68	4.57	1.05
0.20	20	9	0.32	5.38	0.21	8.01	5.11	1.46
0.20	100	1	0.99	1.30	0.19	5.47	1.37	0.86
0.20	100	3	0.73	1.81	0.23	5.57	1.77	0.95
0.20	100	9	0.39	3.47	0.31	5.89	2.52	1.27
0.20	500	1	0.91	1.18	0.28	4.06	1.07	0.89
0.20	500	3	0.75	1.48	0.33	4.14	1.17	0.95
0.20	500	9	0.46	2.49	0.43	4.40	1.42	1.12
0.43	20	1	0.99	1.30	0.19	5.49	1.36	0.86
0.43	20	3	0.74	1.81	0.22	5.59	1.78	0.95
0.43	20	9	0.39	3.46	0.31	5.91	2.55	1.27
0.43	100	1	0.92	1.18	0.28	4.10	1.08	0.89
0.43	100	3	0.75	1.49	0.32	4.18	1.19	0.95
0.43	100	9	0.46	2.51	0.42	4.44	1.44	1.19
0.43	500	1	0.92	1.14	0.37	3.49	1.02	0.90
0.43	500	3	0.78	1.37	0.42	3.55	1.07	0.95
0.43	500	9	0.51	2.15	0.55	3.78	1.19	1.12

NOTES: For all ecosystems, we have $A={(1-\mathit{psp})}^m$, B = 0, and varying values of k, m, and δ. The table lists estimates for the ratios of the publication rate (PR), the ratio of the reliability (REL), the ratio of the number of true breakthrough discoveries (DSCV), the ratio of the median power of published studies($\mathit{Mpw}{r}_{\text{PUB}}$), and the ratio of the interquartile range of the distribution of psp values amongst published studies (${\text{IQpsp}}_{\text{PUB}}$).

1. Reliability is substantially increased with a lower threshold. Based on our results, comparing $\alpha =0.005$ to $\alpha =0.05$, the impact on REL is greatest when δ is small, k is small, and m is large, see Table 4. This is due to the fact that with a lower significance threshold policy, attempted studies are typically of higher power (particularly so when k is small) and of higher prestudy probability. To understand why there would be more studies with higher power and higher prestudy probabilities, consider that with a lower α threshold, the probability of obtaining a significant result (i.e., p-value $<\alpha$) “by chance” is substantially reduced. To maximize one’s expected number of publications (ENP) when $\alpha =0.005$, it is a much better strategy to pursue higher pwr and higher psp strategies. This is particularly true when the effect size, δ, is small.

2. A disadvantage of the lower threshold is that the number of breakthrough discoveries is substantially lower, see Table 4 and Figure S3. This is due to the fact that with a lower α threshold, fewer “high-risk” (i.e., low psp) studies are attempted. The chance that the p-value will fall below α, when α is lowered to 0.005, is already risky enough. The $D_{\text{REL}}$ numbers we obtain suggest that, while fewer low psp studies will be published, these will be much more reliable.

3. When increasing study power is less costly relative to the total cost of a study (i.e., when k is larger, or when δ is larger), the benefit of lowering the significance threshold (increased $\mathit{REL}$) is somewhat smaller. However, the downside (decreased $\mathit{DSCV}$) is substantially smaller, see left-panels of Figures S1 and S3. This suggests that a policy of lowering the significance threshold would perhaps be best suited in a field of research in which increasing one’s sample size is less burdensome. This nuance recalls the suggestion of Ioannidis, Hozo, and Djulbegovic (2013): “Instead of trying to fit all studies to traditionally acceptable Type I and Type II errors, it may be preferable for investigators to select Type I and Type II error pairs that are optimal for the truly important outcomes and for clinically or biologically meaningful effect sizes.”

4. When novelty is more of a requirement for publication (i.e., when m is larger), the benefit of lowering the significance threshold is larger and the downside smaller. This result is due to the fact that a smaller α will incentivize researchers to allocate resources in the direction toward either higher powered or higher psp studies (i.e., away from the South-West corner of the plots in Figure 2). There is a choice between moving toward higher pwr (North) or toward higher psp (East), and different costs associated with each direction. This suggests that for a lower significance threshold policy to be most effective, editors should also adopt, in conjunction, stricter requirements for research novelty. To illustrate, consider three ecosystems of potential interest with their estimated $\mathit{REL},\mathit{DSCV}$, and PR metrics (all with $\delta =0.2$):

   1. The baseline defined by $\alpha =0.05$, m = 3, and k = 500 with:

   2. $\mathit{REL}=0.650,\mathit{DSCV}=0.105$, and PR = 0.054;

   3. the alternative defined by $\alpha =0.005$, m = 3, and k = 500 with:

   4. $\mathit{REL}=0.963,\mathit{DSCV}=0.034$, and PR = 0.040; and

   5. the suggested defined by $\alpha =0.005$, m = 9, and k = 500 with:

   6. $\mathit{REL}=0.898,\mathit{DSCV}=0.112$, and PR = 0.015.

   7. Note that while the suggested has high REL and relatively high DSCV, the PR is substantially reduced.

5. As mentioned earlier, the balance between exploratory and confirmatory research is an important aspect of a scientific ecosystem. The results show that the interquartile range for psp does not change substantially with α. As such, we could conclude that even with a much lower significance threshold, there will still be a wide range of studies attempted in terms of their psp. However, psp values do tend to be substantially higher with smaller α. As such, we should expect that, with smaller α, research will move toward more confirmatory, and less exploratory studies.

6. With a lower α significance threshold, the published effect sizes are much more accurate. When $\alpha =0.005$, the Type M error is reduced to at most 1.4 (i.e., effect size estimates are inflated by at most 40%) and Type S error is negligible; see Figures 4, 5, S7 and S8. These reductions are greatest when data are relatively expensive and when the true effect size is small (i.e. when k = 20 and $\delta =0.20$).

4 The Effects of Strict a priori Power Calculation Requirements

In this section, we investigate the effects of requiring “sufficient” sample sizes. In practical terms, this means adopting publication policies that require studies to show a priori power calculations indicating that sample sizes are “sufficiently large” to achieve the desired level of statistical power, typically 80%. Whereas before, the chance of publishing a positive study in our framework depended only on psp, we now set the probability of publication, A, to also depend on power. In doing so however, we wish to acknowledge the fact that a priori sample size claims are often “wildly optimistic” (Bland 2009).

The “sample size samba” (Schulz and Grimes 2005)—the practice of “tweaking aspects of sample size calculation” (Hazra and Gogtay 2016) in order to obtain what is affordable—often results in a study with less than 80% power being advertised as having 80% power. Even in ideal circumstances, power can be exaggerated due to “optimism bias” (Djulbegovic et al. 2011), also known as the “illusion of power” (Vasishth and Gelman 2017), which occurs when the anticipated effect size is based on a literature filled with overestimates due to publication bias.

To take into account the problematic nature of a priori power calculations, our model is defined such that possessing only 50% power substantially reduces, but does not eliminate, the chance of publication. For studies which really do have 80% power and higher, there will be no notable reduction in the probability of publication. See the appendix for details on how we define A as a continuous function of both psp and pwr.

We calculated the metrics of interest for the same 54 different ecosystems as in Section 3, with our new definition of A. To contrast these ecosystems with those discussed in the previous section, we refer to these ecosystems as “with SSR” (sample size requirements).

4.1 Results

See Table 5. Based on our results, we can make the following main conclusions on the measurable consequences of adopting a journal policy requiring an a priori sample size justification.

Table 5 A comparison between ecosystems with a sample size requirement and ecosystems without.

			Change in	Change in	Change in	Change in	Change in	Change in
δ	k	m	PR	REL	DSCV	$D_{\text{REL}}$	$\mathit{Mpw}{r}_{\text{PUB}}$	${\text{IQpsp}}_{\text{PUB}}$
0.20	20	1	1.46	1.44	0.32	3.97	3.57	0.87
0.20	20	3	1.04	2.08	0.35	4.01	6.13	1.08
0.20	20	9	0.65	3.33	0.41	4.12	7.61	1.38
0.20	100	1	1.21	1.18	0.53	2.40	1.74	0.92
0.20	100	3	1.03	1.44	0.56	2.42	2.25	1.00
0.20	100	9	0.77	2.03	0.63	2.48	3.33	1.20
0.20	500	1	1.08	1.07	0.75	1.63	1.27	0.95
0.20	500	3	1.00	1.18	0.79	1.64	1.39	1.00
0.20	500	9	0.85	1.44	0.85	1.67	1.68	1.06
0.43	20	1	1.21	1.18	0.53	2.39	1.72	0.92
0.43	20	3	1.03	1.44	0.56	2.41	2.25	1.00
0.43	20	9	0.77	2.02	0.63	2.47	3.33	1.20
0.43	100	1	1.08	1.07	0.74	1.65	1.29	0.95
0.43	100	3	1.00	1.19	0.78	1.66	1.41	1.00
0.43	100	9	0.85	1.45	0.85	1.69	1.72	1.12
0.43	500	1	1.04	1.04	0.89	1.38	1.14	0.97
0.43	500	3	1.00	1.10	0.92	1.38	1.19	1.00
0.43	500	9	0.90	1.25	0.98	1.40	1.32	1.12

NOTE: A is defined by equation A4(A4) $$A=\frac{{(1-\mathit{psp})}^m}{1+\hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-(\hspace{0.17em}\text{log}\hspace{0.17em}19)\frac{\mathit{pwr}-c_{50}}{c_{95}-c_{50}}\right\}}.$$(A4) (with $c_{50}=0.5$ and $c_{95}=0.8$), B = 0, $\alpha =0.05$, and varying values of k and m.

1. With SSR, we observed much higher powered studies, see Figure S5. Reliability is also increased, particularly when novelty is highly prized (m is large) and effect sizes are small. This result is as expected. As soon as having a small sample size jeopardizes the probability of publication, it is in the researcher’s best interest to conduct higher powered studies.

2. Requiring “sufficient sample sizes” for publication can be quite detrimental in terms of the number of breakthrough discoveries. The impact on $\mathit{DSCV}$ is greatest when δ, k, and m are all small; see Table 5. The $D_{\text{REL}}$ numbers show that reliability is particularly increased for low psp publications.

3. In conjunction with requiring larger sample sizes, it may be wise to place greater emphasis on research novelty. As in Section 3, such a combined approach could see an increase in reliability with only a limited decrease in discovery. This trade-off is most beneficial when k is small. Consider three ecosystems of potential interest (all with $\alpha =0.05$, and $\delta =0.2$) with their estimated REL, DSCV, and PR metrics:

   1. The baseline defined by m = 3, k = 100, and without SSR; with REL = 0.524, DSCV = 0.216, and PR = 0.043;

   2. the alternative defined by m = 3, k = 100 and with SSR; with REL = 0.757, DSCV = 0.122, and PR = 0.044; and

   3. the suggested defined by m = 6, k = 100 and with SSR; with REL = 0.502, DSCV = 0.325, and PR = 0.022.

   4. Note that the while the suggested has both higher REL and higher $\mathit{DSCV}$ than the baseline, the PR is reduced.

4. With SSR, the Type M error is at most 1.2 (i.e., effect size estimates are inflated by at most 20%) and Type S error is negligible; see Figures 4, 5, S7 and S8.

4.2 In Tandem: The Effects of Adopting Both a Lower Significance Threshold and a Power Requirement

We are also curious as to whether lowering the significance threshold in addition to requiring larger sample sizes would carry any additional benefits relative to each policy innovation on its own. See Table A.1 for results, and consider two main findings:

1. For ecosystems with SSR, the distribution of published studies does not change dramatically when α is lowered. Figure 3 shows the density describing the characteristics of published studies, $f_{\text{PUB}}(\mathit{psp},\mathit{pwr})$, for the four policies of interest, with k = 500 and m = 3: (1) $\alpha =0.05$, without SSR; (2) $\alpha =0.05$, with SSR; (3) $\alpha =0.005$, without SSR; and (4) $\alpha =0.005$, with SSR. The difference between the densities in (2) and (4) is primarily a matter of a shift in psp.

2. As expected, lowering the significance threshold further increases reliability and the number of breakthrough discoveries is further decreased; see Table A.1, and Figures S1 and S3.

Fig. 3 Heat-maps show the density of published articles, $f_{\text{PUB}}(\mathit{psp},\mathit{pwr})$, for the four policies of interest, with fixed $\delta =0.2$, k = 500 and m = 3.

Fig. 4 Values of ${\widehat{{\mu }_d}}_{\text{PUB}}^{\mathit{neg}}$ for varying values of k, m and α; results for $\delta =0.2$ and $\delta =0.43$ on the left and right panels, respectively. Top panels shows results with no power requirement (i.e., $A={(1-\mathit{psp})}^m$). Bottom panels shows results with power requirement (i.e., A defined as per equation A4(A4) $$A=\frac{{(1-\mathit{psp})}^m}{1+\hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-(\hspace{0.17em}\text{log}\hspace{0.17em}19)\frac{\mathit{pwr}-c_{50}}{c_{95}-c_{50}}\right\}}.$$(A4) (with $c_{50}=0.5$ and $c_{95}=0.8$)).

Fig. 5 Values of $|\widehat{{\mu }_d}{|}_{\text{PUB}}$ for varying values of k, m and α; results for $\delta =0.2$ and $\delta =0.43$ on the left and right panels respectively. Top panels shows results with no power requirement (i.e., $A={(1-\mathit{psp})}^m$). Bottom panels shows results with power requirement (i.e., A defined as per equation A4(A4) $$A=\frac{{(1-\mathit{psp})}^m}{1+\hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-(\hspace{0.17em}\text{log}\hspace{0.17em}19)\frac{\mathit{pwr}-c_{50}}{c_{95}-c_{50}}\right\}}.$$(A4) (with $c_{50}=0.5$ and $c_{95}=0.8$)).

5 Conclusion

There remains substantial disagreement on the merits of requiring greater statistical stringency to address the reproducibility crisis. Yet all should agree that innovative publication policies can be part of the solution. Going forward, it is important to recognize that current norms for Type 1 and Type 2 error levels have been driven (almost) entirely by tradition and inertia rather than careful coherent planning and result-driven decisions (Hubbard and Bayarri 2003). Hence, improvements should be possible.

In response to Amrhein, Korner-Nievergelt, and Roth (2017) who suggest that a more stringent α threshold will lead to published science being less reliable, our results suggest otherwise. However, just as Amrhein, Korner-Nievergelt, and Roth (2017) contend, our results indicate that the publication rate will end up being substantially lower with a smaller α. While going from p < 0.05 to p < 0.005 may be beneficial to published science in terms of reliability, we caution that there may be a large cost in terms of fewer true breakthrough discoveries. One must ask whether a large reduction in novel discoveries is an acceptable price to pay for a large increase in their reliability? Importantly, and somewhat unexpectedly, our results suggest that this can be mitigated (to some degree) by adopting a greater emphasis on research novelty. In practice, implementing stricter requirements for research novelty, could be accomplished by greater editorial emphasis on “surprising results” as a requisite for publication. This approach, however, might be difficult to achieve unless one is willing to accept a much lower publication rate. In summary, publishing less may be the necessary price to pay for obtaining more reliable science.

Recently, some have suggested that researchers choose (and justify) an “optimal” value for α, for each unique study; see Mudge et al. (2012), Ioannidis, Hozo, and Djulbegovic (2013), and Lakens et al. (2018). Each study within a journal would thereby have a different set of criteria. This is a most interesting idea and there are persuasive arguments in favor of such an approach. Still, it is difficult to anticipate how such a policy would play out in practice and how the research incentive structure would change in response. A model, like the one presented here, but with the α threshold set to vary with psp, could provide useful insight.

We are also cautious about greater sample size requirements. In summary, we found that the impacts of adopting a sample size requirement policy are similar to the impacts of lowering the α significance threshold. While improving reliability, requiring studies to show “sufficient power” will severely limit novel discoveries in fields where acquiring data is expensive. Is it beneficial for editors and reviewers to consider whether a study has “sufficient power”? How much should these criteria influence publication decisions? Answers to these questions are not at all obvious. Again, using the methodology introduced, we suggest that adopting a greater emphasis on research novelty may mitigate, to a certain extent, some of the downside of adopting greater sample size requirements at the cost of lowering the overall number of published studies. Given that, as a result of publication bias, it can often be better to discard 90% of published results for meta-analytical purposes (Stanley, Jarrell, and Doucouliagos 2010), this may be an approach worth considering.

Our main recommendation is that, before adopting any (radical) policy changes, we should take a moment to carefully consider, and model how, these proposed changes might impact outcomes. The methodology we present here can be easily extended to do just this. Two scenarios of interest come immediately to mind.

First, it would be interesting to explore the impact of publication bias, i.e., the tendency of journals to reject non-significant results (Sterling, Rosenbaum, and Weinkam 1995). This could be done by allowing B to take different non-zero values. Based on simulation studies, de Winter and Happee (2013) suggest that publication bias can in fact be beneficial for the reliability of science. However, under slightly different assumptions, van Assen et al. (2014) arrive at a very different conclusion. Clearly, a better understanding of how publication bias changes a scientist’s incentives is needed. If statistical significance becomes a much less important screening criterion for publication, how will published science change? Recently, a number of influential researchers have argued that to address low reliability, reviewers should “abandon statistical significance” (McShane et al. 2017). While our framework could be extended to explore this proposal (e.g., by taking A = B), these researchers suggest substituting statistical significance with a “more holistic view of the evidence” to inform publication decisions. Clearly, this does not lend itself to our modeling framework, or similar meta-research paradigms.

Second, it would be worthwhile to investigate the potential impact of requiring study preregistration. Coffman and Niederle (2015) use the accounting of Ioannidis (2005) to evaluate the effect of preregistration on reliability and conclude that pre-registration will have only a modest impact. However, the impact on the publication rate and on the number of breakthrough findings is still not well understood. This is particularly relevant given the current trend to adopt “result-blind peer-review” (Greve, Bröder, and Erdfelder 2013) policies including, most recently, the policy of Registered Reports (Chambers et al. 2015). The conditional equivalence testing publication policy proposed in our earlier work, see Campbell and Gustafson (2018), could also be considered.

Our methodology assumes above all that researchers’ decisions are driven exclusively by the desire to publish. But the situation is more complex. Publication is not necessarily the end goal for a scientific study and requirements with regards to significance and power are not only encountered at the publication stage. In the planning stages, before a study even begins, ethics committees and granting agencies will often have certain minimal requirements; see Ploutz-Snyder, Fiedler, and Feiveson (2014) and Halpern, Karlawish, and Berlin (2002). And after a study is published, regulatory bodies and policy makers will also often subject the results to a different set of norms.

We also assumed a framework of independent Bernoulli trials. Of course, in practice, most studies are not conducted independently. For example, in the context of clinical trials, phase 3 studies are conducted based on the success or failure of earlier phase 2 studies (see Burt et al. (2017) who model the advantages and disadvantages of different clinical development strategies). It is difficult to anticipate the consequences of failing to incorporate such dependencies in our model. A more elaborate model, in which studies are correlated and the psp of subsequent studies is updated appropriately, could prove very informative.

Finally, it is important to acknowledge that no publication policy will be perfect. Science is inherently challenging and we must always be willing to accept that a certain proportion of research is potentially false (Contopoulos-Ioannidis, Ntzani, and Ioannidis 2003; Djulbegovic and Hozo 2007). Each policy will have its advantages and disadvantages. Our modeling exercise makes this all the more evident and forces us to carefully consider different potential trade-offs.

Code

Please note that all code to produce the results, tables, and figures in this article have been posted to a repository on the Open Science Framework, DOI 10.17605/OSF.IO/YQCVA.

Note

(1) $$\text{Pr}\left(\text{Positive}\right|{\mu }_d)=(1-F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}\left(t_{\alpha /2}^{*}\right))+F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}(-t_{\alpha /2}^{*}),$$(1)

where $t_{\alpha /2}^{*}$ is the upper 100$·\frac{\alpha }{2}$-th percentile of the t-distribution with n – 2 degrees of freedom, ${\sigma }^{*}=\sigma \sqrt{(1/n_1+1/n_2)}$, and $F_{\mathit{df},\mathit{ncp}}(x)$ is the cdf of the noncentral t distribution with df degrees of freedom and noncentrality parameter ncp. We calculate the minimum required sample size, n, to obtain a desired power, pwr, as follows:(2) $$n={\text{argmin}}_n\left(\right|\mathit{pwr}-(1-F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}(t_{\alpha /2}^{*}\left)\right)+F_{n-2,\frac{{\mu }_d}{{\sigma }^{*}}}(-t_{\alpha /2}^{*})\left|\right)$$(2)

Acknowledgments

We wish to gratefully acknowledge Prof. Will Welch and Prof. John Petkau for their valuable suggestions and advice. Furthermore, we wish to acknowledge funding from the Natural Sciences and Engineering Research Council of Canada (NSERC grant number RGPIN 183772-13).

Appendix

Here, we introduce some compact notation that will be useful for expressing distributional quantities of interest. Particularly, the probabilities comprising the distribution of a study across the eight categories are expressed as $q_{\mathit{abc}}(\mathit{psp},\mathit{pwr})$, where $a\in \{0,1\}$ indicates the truth (a = 0 for null, a = 1 for alternative), $b\in \{0,1\}$ indicates the statistical finding (b = 0 for negative, b = 1 for positive), and $c\in \{U,P\}$ indicates publication status. As examples, we could write $F{N}_{\text{UN}}=q_{10U}$, or ${\text{TP}}_{\text{PUB}}=q_{11P}$. We also use a plus notation to add over subscripts, so, for instance, $q_{1+P}={\text{TP}}_{\text{PUB}}+F{N}_{\text{PUB}}$.

As motivated above, we consider properties that result from a scientist or group of scientists stochastically allocating T resources (not studies per se) according to a distribution across (psp, pwr). We denote the function n(pwr) as the required sample size to obtain a power of pwr. Presuming the incentive to publish, the density of this distribution is taken proportional to ENP(psp, pwr), which we express as(A1) $$\begin{array}{c}{f}_{\text{RES}}(\mathit{psp},\mathit{pwr})\propto \mathit{ENP}(\mathit{psp},\mathit{pwr})\\ \propto {\{k+n(\mathit{pwr}\left)\right\}}^{-1}{q}_{++P}(\mathit{psp},\mathit{pwr}).\end{array}$$(A1)

Consequently, the distribution of (psp, pwr) across attempted studies has density(A2) $$\begin{array}{c}{f}_{\text{ATM}}(\mathit{psp},\mathit{pwr})\propto {\{k+n(\mathit{pwr}\left)\right\}}^{-1}{f}_{\text{RES}}(\mathit{psp},\mathit{pwr})\\ \propto {\{k+n(\mathit{pwr}\left)\right\}}^{-2}{q}_{++P}(\mathit{psp},\mathit{pwr}).\end{array}$$(A2)

In turn, the distribution of (psp, pwr) across published studies has density(A3) $$\begin{array}{c}{f}_{\text{PUB}}(\mathit{psp},\mathit{pwr})\propto f_{\text{ATM}}(\mathit{psp},\mathit{pwr})q_{++P}(\mathit{psp},\mathit{pwr})\\ \propto {\{k+n(\mathit{pwr}\left)\right\}}^{-2}{\left\{q_{++P}\right(\mathit{psp},\mathit{pwr}\left)\right\}}^2.\end{array}$$(A3)

Note particularly that $f_{\text{PUB}}(\mathit{psp},\mathit{pwr})\propto {\{f_{\text{RES}}(\mathit{psp},\mathit{pwr})\}}^2$. Hence, the distribution of (psp, pwr) across published studies is a concentrated version of the distribution describing how resources are deployed.

A1. Ecosystem Metrics: Further Details

We evaluate each ecosystem of interest on the basis of the following five metrics.

A1.1.Publication Rate and the Number of Studies Attempted/Published

If T units of resources are deployed according to $f_{\text{RES}}()$, then we expect that $N_{\text{ATM}}$ studies will be attempted, where$$T^{-1}{N}_{\text{ATM}}=E_{\text{RES}}\left[{\{k+n(\text{PWR}\left)\right\}}^{-1}\right].$$

Similarly, $N_{\text{PUB}}$ studies will be published, with$$T^{-1}{N}_{\text{PUB}}=E_{\text{RES}}\left[{\{k+n(\text{PWR}\left)\right\}}^{-1}{q}_{++P}(\text{PSP},\text{PWR})\right].$$

The ratio $N_{\text{PUB}}/N_{\text{ATM}}$, which does not depend on T, is of evident interest, as the publication rate (PR) for attempted studies.

A1.1.2. The Reliability (REL)

A highly relevant metric for the scientific ecosystem is the proportion of published findings that are correct. In all the ecosystems, we consider in this article, we make the assumption that only positive results are published (i.e., B = 0). Therefore, we can express reliability (REL) simply as$$\text{REL}=\frac{E_{\text{ATM}}\left\{q_{11P}(\text{PSP},\text{PWR})\right\}}{E_{\text{ATM}}\left\{q_{+1P}(\text{PSP},\text{PWR})\right\}}.$$

More generally, in ecosystems where negative results might be published (i.e., $B(\mathit{psp},\mathit{pwr})\ne 0$), the reliability would equal the proportion of published articles that reach a correct conclusion, i.e.,$$\text{REL}=\frac{E_{\text{ATM}}\left\{q_{00P}(\text{PSP},\text{PWR})+q_{11P}(\text{PSP},\text{PWR})\right\}}{E_{\text{ATM}}\left\{q_{++P}(\text{PSP},\text{PWR})\right\}}.$$

A1.1.3. Breakthrough Discoveries (DSCV)

The ability of the scientific ecosystem to produce breakthrough findings is an important attribute. We quantify this in terms of spending T resource units yielding an expectation of $\mathit{DSCV}$ breakthrough results. Here, a true breakthrough result is defined as a TP and published study that results from a psp value below a threshold, that is, a very surprising positive finding that gets published and also happens to be true. If we set the breakthrough threshold as psp < 0.05, then$$T^{-1}D\mathit{SCV}=E_{\text{RES}}\left\{I_{(0,0.05)}\left(\text{PSP}\right)\frac{q_{11P}(\text{PSP},\text{PWR})}{k+n\left(\text{PWR}\right)}\right\}$$

Note that we can also write this out in terms of the density of attempted studies, $f_{\text{ATM}}$:$$\begin{array}{c}{T}^{-1}\text{DSCV}=\frac{\int h(\text{PSP},\text{PWR})f_{\text{ATM}}(\text{PSP},\text{PWR})(k+n(\text{PWR}\left)\right)}{\int f_{\text{ATM}}(\text{PSP},\text{PWR})(k+n(\text{PWR}\left)\right)}\\ =\frac{E_{\text{ATM}}\left[I_{(0,0.05)}\right(\text{PSP}\left)q_{11P}\right(\text{PSP},\text{PWR}\left)\right]}{E_{\text{ATM}}[k+n(\text{PWR}\left)\right]}.\end{array}$$where we define the function: $h(\text{PSP},\text{PWR})=I_{(0,0.05)}(\text{PSP})\frac{q_{11P}(\text{PSP},\text{PWR})}{k+n(\text{PWR})}$.

We also wish to quantify the reliability of those published studies with psp values below a threshold of 0.05. We have that:$$D_{\text{REL}}=\frac{E_{\text{ATM}}\left\{I_{(0,0.05)}\left(\text{PSP}\right)q_{11P}(\text{PSP},\text{PWR})\right\}}{E_{\text{ATM}}\left\{I_{(0,0.05)}\left(\text{PSP}\right)q_{+1P}(\text{PSP},\text{PWR})\right\}}.$$

A1.1.4. The Median Power of Published Studies ($\mathit{Mpw}{r}_{\text{PUB}}$)

We already mentioned the relevance of the psp marginals of (A2) and (A3). In a similar vein, the marginal distributions of pwr under each of these distributions are readily interpreted metrics of the ecosystem. We define $\mathit{Mpw}{r}_{\text{PUB}}$ as the median of pwr under the $f_{\text{PUB}}()$ distribution.

A1.1.5. The Balance between Exploration and Confirmation (${\text{IQpsp}}_{\text{PUB}}$)

There has been much discussion about the desired balance between researchers looking for a priori unlikely relationships versus confirming suspected relationships put forth by other researchers; see for example, Sakaluk (2016) and Kimmelman, Mogil, and Dirnagl (2014). The marginal distribution of psp arising from $f_{\text{ATM}}(\mathit{psp},\mathit{pwr})$ describes the balance between exploration versus confirmation for attempted studies, while the psp marginal from $f_{\text{PUB}}(\mathit{psp},\mathit{pwr})$ does the same for published studies. More specifically, we report the interquartile ranges of these marginal distributions for a given ecosystem.

A1.1.6. The Accuracy of Published Research

Consider extending $f_{\text{ATM}}(\mathit{psp},\mathit{pwr})$ to $f_{\text{ATM}}(\mathit{psp},\mathit{pwr},d,u)$, where:

$f_{\text{ATM}}(d,u|\mathit{psp},\mathit{pwr})=f_{\text{ATM}}(d|\mathit{psp},\mathit{pwr})f_{\text{ATM}}(u)$, and where

• $f_{\text{ATM}}(u)$ is the density of a standard uniform distribution; and

• $f_{\text{ATM}}(d|\mathit{psp},\mathit{pwr})$ is the density of a two-component mixture distribution such that:

$\mathit{Pr}(d=\delta )=\mathit{psp}$; and $\mathit{Pr}(d=0)=1-\mathit{psp}$.

In order to obtain a sample from the distribution of published effect sizes, we proceed according to the following steps.

Set w = 1, q = 1 and b = 1. While w < 1000:

1. Draw a Monte Carlo realization of $(\mathit{ps}{p}_{[b]},\mathit{pw}{r}_{[b]},d_{[b]},u_{[b]})$ from $f_{\text{ATM}}(\mathit{psp},\mathit{pwr},d,u)$.

2. Simulate a two-sample Normally distributed dataset, $\mathit{dat}{a}_{[b]}$, with a sample size of $n(\mathit{pw}{r}_{[b]})$, a true difference in population means of ${\mu }_d=d_{[b]}$, and a true variance of ${\sigma }^2=1$.

3. Run a standard t-test on $\mathit{dat}{a}_{[b]}$, and obtain the estimated effect size, ${\widehat{\delta }}_{[b]}$, and the two-sided p-value, $\mathit{pva}{l}_{[b]}$.

4. If $u_{[b]}<A(\mathit{ps}{p}_{[b]},\mathit{pw}{r}_{[b]})$ AND $\mathit{pva}{l}_{[b]}<\alpha$, then do:$${\widehat{\delta }}_{+1P}^{\left[q\right]}={\widehat{\delta }}_{\left[b\right]};\hspace{1em}\text{and}\hspace{1em}q=q+1.$$

5. If $u_{[b]}<A(\mathit{ps}{p}_{[b]},\mathit{pw}{r}_{[b]})$ AND $\mathit{pva}{l}_{[b]}<\alpha$, AND $d_{[b]}=\delta$, then do:$${\widehat{\delta }}_{11P}^{\left[w\right]}={\widehat{\delta }}_{\left[b\right]};\hspace{1em}\text{and}\hspace{1em}w=w+1.$$

6. Set $b=b+1$.

Then we have that $\left\{{\widehat{\delta }}_{+1P}\right\}$ is a Monte Carlo sample from the conditional distribution of effect sizes given that studies are both positive and published. We also have that $\left\{{\widehat{\delta }}_{11P}\right\}$ is a Monte Carlo sample from the conditional distribution of effect sizes given that studies are positive, and published, and true.

Using these Monte Carlo samples, we can easily approximate the following metrics of interest:

• Mean Absolute Published Effect Size ($|\widehat{{\mu }_d}{|}_{\text{PUB}}$). Approximated as the mean of $\left|\right\{{\widehat{\delta }}_{+1P}\left\}\right|$.

• Proportion of Published Effect Sizes that are Negative (${\widehat{{\mu }_d}}_{\text{PUB}}^{\mathit{neg}}$). Approximated as the proportion of $\left\{{\widehat{\delta }}_{+1P}\right\}$ that are less than zero.

• The “Type S” error. Approximated as the proportion of $\left\{{\widehat{\delta }}_{11P}\right\}$ less than 0.

• The “Type M” error. Approximated as the mean of the absolute effect size divided by the true effect size: $\left|\right\{{\widehat{\delta }}_{11P}\left\}\right|/\delta$.

A2. The Probability of Publication as a Function of Both psp and pwr

In Section 3, the chance of publishing a positive study in our framework depended only on novelty via$$A={(1-\mathit{psp})}^m.$$

For Section 4, a convenient choice of function to represent a journal policy of requiring an a priori sample size justification would be:(A4) $$A=\frac{{(1-\mathit{psp})}^m}{1+\hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-(\hspace{0.17em}\text{log}\hspace{0.17em}19)\frac{\mathit{pwr}-c_{50}}{c_{95}-c_{50}}\right\}}.$$(A4)

So A is reduced more when power is lower, with the extent of the reduction parameterized by $(c_{50},c_{95})$. Specifically, c₉₅ is the value for pwr at which the multiplicative reduction is near negligible (factor of 0.95), while c₅₀ is the value for pwr at which the multiplicative reduction is a factor of 0.5. We conduct our experimentation using $c_{50},c_{95}=(0.5,0.8)$, with the following rationale. If a journal does require an a priori sample size justification, a claim of 80% power is the typical requirement. Hence, a study which really attains 80% power is not likely to suffer in its quest for publication, motivating $c_{95}=0.8$.

A3. Additional Results

Supplemental materials

The supplementary material includes additional figures which plot various metrics of interest across the entire set of ecosystems considered.

Table A.1 A comparison between ecosystems with both a sample size requirement and $\alpha =0.005$ and ecosystems without a sample size requirement and $\alpha =0.05$.

			Change in	Change in	Change in	Change in	Change in	Change in
δ	k	m	PR	REL	$\mathit{DSCV}$	$D_{\text{REL}}$	$\mathit{Mpw}{r}_{\text{PUB}}$	${\text{IQpsp}}_{\text{PUB}}$
0.20	20	1	1.77	1.61	0.09	11.83	3.77	0.83
0.20	20	3	1.14	2.69	0.10	12.16	6.57	1.03
0.20	20	9	0.53	6.31	0.15	13.23	8.33	1.46
0.20	100	1	1.29	1.31	0.15	6.99	1.81	0.86
0.20	100	3	0.97	1.86	0.17	7.17	2.38	0.95
0.20	100	9	0.52	3.78	0.25	7.77	3.60	1.33
0.20	500	1	1.03	1.19	0.24	4.57	1.29	0.90
0.20	500	3	0.85	1.50	0.28	4.68	1.43	0.93
0.20	500	9	0.52	2.60	0.38	5.03	1.76	1.18
0.43	20	1	1.29	1.31	0.15	6.97	1.79	0.86
0.43	20	3	0.97	1.86	0.17	7.15	2.38	0.95
0.43	20	9	0.52	3.77	0.25	7.74	3.60	1.33
0.43	100	1	1.04	1.19	0.24	4.63	1.31	0.90
0.43	100	3	0.85	1.51	0.28	4.74	1.44	0.93
0.43	100	9	0.52	2.62	0.37	5.09	1.80	1.25
0.43	500	1	0.97	1.15	0.36	3.72	1.15	0.92
0.43	500	3	0.83	1.38	0.41	3.80	1.21	0.95
0.43	500	9	0.54	2.20	0.53	4.07	1.35	1.18

NOTES: For those with a sample size requirement, A is defined by equation A4(A4) $$A=\frac{{(1-\mathit{psp})}^m}{1+\hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-(\hspace{0.17em}\text{log}\hspace{0.17em}19)\frac{\mathit{pwr}-c_{50}}{c_{95}-c_{50}}\right\}}.$$(A4) (with $c_{50}=0.5$ and $c_{95}=0.8$). For those without a sample size requirement, $A={(1-\mathit{psp})}^m$. For all we have B = 0, and varying values of k and m as indicated.