Discussion on “Second-Guessing Clinical Trial Designs” by Jonathan J. Shuster and Myron N. Chang

Abstract

With the option for second-guessing, a group sequential design can be found to match any given data. The key question is, What could we learn from this? Could we claim that the second-guess should have been used instead of the nonsequential design? Could we claim further that the second-guess is superior to other alternative group sequential designs? I propose a criterion and a procedure to assess the credibility of such a claim. I recommend limiting any such claim to those with high credibility. Numerical illustrations are provided.

Keywords:

• Adaptive design
• Group sequential design

Subject Classifications:

• 60J65
• 62L05

1. INTRODUCTION

From training and experience, biostatisticians share the predilection for proposing better and more efficient clinical trial designs. We also share the predicament of sometimes not getting the proposals accepted by investigators or management. With second-guessing, we could now say “gotcha,” and hopefully put pressure on trial sponsors to accept better designs. This is a worthy cause, and the authors, Professors Shuster and Chang, are commended for this interesting and thought-provoking paper.

Learning from past experience is essential to human evolution, and there is no reason why the clinical trial community should not do the same. Analogous to post-hoc data analysis, by which it would be very likely to find a model or an analysis that produces a desired result, second-guessing selects a design that fits the given data. With a second-guess, the key question is, What could we learn from it? Because of the post-hoc nature of second-guessing, it is difficult to separate learning from “witch hunting.” Unless proper guidance is followed, exercising second-guessing could be imminently detrimental to the credibility of statistical science. I propose to evaluate the credibility level of each claim and to limit conclusions to credible claims.

2. A WORKING EXAMPLE

Consider a clinical trial for comparing an experimental treatment to a standard control where the clinical endpoint measures the success or failure of a treatment. Suppose a nonsequential design with balanced randomization were chosen by the trial sponsor. The sample size n = 3, 980 per treatment group was calculated according to the arcsin transformation of the success rates to detect an improvement of the success rates of 2.5% for the experimental treatment, p ₁ = 0.15, from that of the control of p ₀ = 0.125. The one-sided significance level of α = 0.025 and 90% power were used.

Suppose the trial statistician also considered four alternative group sequential designs with four equally spaced analyses. To stop the trial early to reject the null hypothesis H: p ₁ − p ₀ ≤ 0 in favor of the alternative hypothesis A: p ₁ − p ₀ > 0, four significance boundaries with ρ = 1/2, 1/4, 0 and −1/2 from the Wang-Tsiatis family B(α, ρ)(k/4)^ρ−1/2 were used, where k = 1, 2, 3, 4 and α = 0.025. Note that ρ = 1/2 and 0 correspond to the Pocock and O'Brien-Fleming boundaries, respectively. To reflect the practical applications (International Sudden Infarct Study 2 and Zocor trial) considered by Professors Shuster and Chang, where an objective of interest is to establish if the experimental treatment is superior to the placebo control, rather than if the placebo control could be superior to the experimental treatment, we also allow stopping the trial early for futility. A nonbinding β-spending function β(k/4)³ was used for each design, where β = 0.1 and k = 1, 2, 3, 4.

The maximum sample sizes (SS), and expected sample sizes (ESS) for three effect size values, i.e., 0.0514, 0, and −0.0514, of are given in Table 1. We also provide in Table 1 the average sample number (ASN) for different designs under the prior probability model π = (0.1, 0.8,0.1), which was considered to represent past experience for the effect size vector (− 0.0514, 0,0.0514). The trial statistician proposed the group sequential design with the O'Brien-Fleming boundary (ρ = 0) since the average sample number was the smallest among the four designs.

Table 1. Sample size: and expected sample size for group sequential designs (GSD)

	d 1/2	d 1/4	d 0	d −1/2	NSD
SS	4,818.00	4,335.00	4,193.00	4,124.00	3,980
ESS δ = 0.0514	2,739.76	2,851.21	3,060.21	3,401.48	3,980
ESS δ = 0	2,757.56	2,643.67	2,599.86	2,580.90	3,980
ESS δ = −0.0514	1,533.63	1,452.71	1,428.22	1,415.92	3,980
ASN π = (0.1, 0.8, 0.1)	2,665.39	2,545.33	2,528.46	2,546.92	3,980

3. SECOND GUESSING

For a given X(1) from the nonsequential design, I first calculate the expected sample size of each design, i.e., d _1/2, d _1/4, d ₀, d _−1/2, or non sequential design, following the simulation procedure of Professors Shuster and Chang (Section 2). With each replication, the sample size is the information (number of patients or events) at which a boundary is crossed. The expected sample size of a design for a given X(1) is simply the average sample sizes over a large number of replications. A second-guess, for a given X(1), is the design from the competing designs with the smallest expected sample size.

To evaluate probabilities of choosing a second-guess and the competing designs, including nonsequential designs, using the second-guessing procedure, X(1) is randomly generated according to the following mixture distribution: (1) Δ is randomly selected to follow the prior probabilities P{Δ = −0.0514} = 0.1, P{Δ = 0} = 0.8, and P{Δ = 0.0514} = 0.1, and (2) given Δ, X(1) is selected from the normal distribution with mean n ^1/2Δ and variance 1. This process is repeated with a large number of simulation runs. The credibility level for a claim is the probability, which is approximated by simulations, that the second-guess is chosen. We recommend limiting conclusions to claims with high credibility levels.

Now suppose that at the end of the nonsequential trial, the test statistic is 9. By the second-guessing procedure, the Pocock design is chosen. The statistician wants to evaluate the following two claims: (l) in comparison to the nonsequential design, the Pocock design should be used, and (2) none of the other four alternative designs should be used.

For the first claim, there are two competing designs: the Pocock design and the nonsequential design. The probabilities for the two designs are 0.93 and 0.07. The credibility level for the Pocock design is 0.93, and therefore, in retrospect, the statistician can conclude that the Pocock design should be used. For the second claim, there are five competing designs, i.e., the Pocock design (d _1/2), three other group sequential designs (d _1/4, d ₀, d _−1/2), and the nonsequential design. The probabilities for the five designs are 0.0, 0.041, 0.039, 0.909, and 0.011, respectively. The credibility of the Pocock design of the second claim is 0, and therefore, the second claim cannot be made.

4. CONCLUDING THOUGHTS

In evaluating the second claim, I am surprised to find that the probability for choosing the O'Brien-Fleming design, which is optimal according to the prior probabilities π = (0.1,0.8,0.1), is only 0.039. This may be indicative of an underlying methodological issue with the second-guessing method proposed by Professors Shuster and Chang. In light of this issue, it is sensible to recommend the traditional approach of evaluating only a proposed group sequential design that was not chosen by the investigators or trial sponsors. For most applications, this should be adequate for achieving the goals of learning from past experience. There are, however, many applications where group sequential designs were not adequate, and the lessons learned point to a different approach: adaptive designs.

Notes

d _1/2 = GSD with the Pocock boundary {}d _1/4 = GSD with ϕ = 1/4 d ₀ = GSD with the O'Brien-Fleming boundary d _−1/2 = GSD with ρ = −1/2 NSD=Nonsequential design that has no interim analysis

Recommended by N. Mukhopadhyay