Discussion on “Second-Guessing Clinical Trial Designs” by Jonathan J. Shuster and Myron N. Chang

Abstract

An argument is put forward to include fully sequential tests as useful methods in clinical trials. Generalizations of the Pocock (1977) and O'Brien and Fleming (1979) group sequential tests to fully sequential strategies are available now for several data structures. Looking at the whole statistical process gives a lot of additional information, and this makes flexible decision making possible.

Keywords:

• Clinical trial
• Group sequential tests
• Sequential tests

Subject Classifications:

• 60J65
• 62L05

1. INTRODUCTION

This contribution raises the important issue of fixed sample size versus different group sequential trial designs. We might go one step further and add fully sequential methods to the pool of available arsenal.

When data accrues over a lengthy period of time, it is an important consideration how often the data should be looked at to be able to come to a timely and correct decision. If in a study only the end summary is published (fixed-sample-size design), then Shuster and Chang suggest that one can estimate by interpolation what knowledge would have been available at some earlier times. This allows readers to decide if a more frequent evaluation scheme would have been more efficient in saving time and/or money, which is naturally of great importance for ethical and financial reasons.

Currently, fixed-sample-size and group sequential methods are in use. Fully sequential trials are not considered as an option simply because they are not in the pool of the approved methodologies. As the nature and purpose of medical trials is widely varied, in many cases the full statistical sequence could add a lot of insight into the phenomenon under study. Its use and presentation could circumvent later criticisms for failing to be open or to adopt a strategy for early decision making. Second-guessing as discussed in this paper would then be unnecessary.

Before putting forth arguments for adding fully sequential monitoring to the accepted arsenal of sequential tests, let us have a brief look at why group sequential trials are the methods of choice when interim analyses are called for.

The need for interim analyses has already been well established, and there is no longer any pressing debate about it now. The vast literature on group sequential methods and their popularity are also well known. It is their technical feasibility that made group sequential methods so widely used since the 1980s. Yet the first sequential methods in statistics were fully sequential in nature: for example, control charts (dated to the period 1920–1930) and the sequential likelihood ratio tests of Wald (around 1940).

Clinical trials have more complicated data structures, so monitoring the mean of independent, identically distributed observations or choosing between two simple hypotheses are not sufficient for their need. Attempts to extend the first sequential testing methods above to more complex situations, such as when nuisance parameters are present, were not successful (see, e.g., Remark 1 of Gombay, 2002a, and Gombay, 2002b, and references therein). In many cases the distribution of the full sequence, Stat(k), k = 1, 2,…, n, that is, the values of the statistic calculated after each new observation was deemed intractable (e.g., Basseville and Nikiforov, 1993).

On the other hand, fixed-sample-size statistics were well understood and accepted for many such situations. The work of Pocock (1977) and O'Brien and Fleming (1979) pioneered the adjustment of critical values for the fixed-sample-size tests when interim analyses are called for. Their implementation called for numerical calculations involving multivariate distributions.

2. NEW TESTS

With the development of strong approximations in statistics, however, often the full statistics sequence can now be used for testing hypotheses and theoretical critical values are available that have been shown to work well. For example, the generalized likelihood ratio (Gombay, 1996), the score vector, and the Wald statistics sequence (Gombay, 2002a) have now strong approximations that allow sequential tests. There is evidence in published (Gombay and Hussein, 2006) and unpublished simulations that the sequential versions of the Pocock and O'Brien-Fleming tests are not losing power or stop later than the group sequential version, allowing only a limited number of looks at the data. Furthermore, the full statistics sequence can reveal important characteristics of the treatments. See Example 1 of Gombay (2005), where the statistical process is significant at the early stages, but later no difference in treatment effect was present. This revealed that Treatment A had a fatal effect on many people with a heart condition soon after the treatment began, but for those who survived it was not effective.

From this example it is obvious that reconstructing earlier situations from end results, as advocated by the paper under discussion, is only useful if the process is homogeneous along the time variable. It cannot reveal hidden variable effects that may present themselves only in the analysis of the full sequence.

So, if the clinical trial's results have to be open to scrutiny, fully accessible to all (or to a select group), then only the whole statistics process can answer the “what if” questions. Otherwise, we have a case of need for second-guessing, which may or may not be correct. There is a trend now to call for more transparency (e.g., Lilford et al., 2001; Reidpath, 2001; and in discussions in meetings), and if this will be a successful effort for the protection of all involved, then fully sequential methods will have to be an integral part of the available accepted methods.

Having said that, it is obvious that not all trials need fully sequential methodology. However, one argument used against it, though valid 20 years ago, is no longer so. This is the argument of the technical difficulty arising from the meeting times of the data and safety monitoring committees. We now have the technology of vast databases with data entry from all over the world and instantaneous updating. Using this, or indeed a much smaller-scale, technology for multicenter medical trials makes this problem obsolete. Also, a statistics process can be presented effectively by a graph, together with the critical values, and this would not violate any privacy requirements.

In conclusion, I welcome this contribution for raising many important, though somewhat dormant issues, and for arguing that medical trial designs may have to be more flexible and open than they are allowed to be by the current methodologies of choice.

Notes

Recommended by N. Mukhopadhyay