Accelerate vaccine development using seamless phase 2/3 trial designs

ABSTRACT

Background

Traditional vaccine development, often a lengthy and costly process of three separated phases. However, the swift development of COVID-19 vaccines highlighted the critical importance of accelerating the approval of vaccines. This article showcases a seamless phase 2/3 trial design to expedite the development process, particularly for multi-valent vaccines.

Research Design and Methods

This study utilizes simulation to compare the performance of seamless phase 2/3 design with that of conventional trial design, specifically by re-envisioning a 9-valent HPV vaccine trial. Across three cases, several key performance metrics are evaluated: overall power, type I error rate, average sample size, trial duration, the percentage of early stop, and the accuracy of dose selection.

Results

On average, when the experimental vaccine was assumed to be effective, the seamless design that performed interim analyses based solely on efficacy saved 555.73 subjects, shortened trials by 10.29 months, and increased power by 3.70%. When the experimental vaccine was less effective than control, it saved an average of 887.73 subjects while maintaining the type I error rate below 0.025.

Conclusion

The seamless design proves to be a compelling strategy for vaccine development, given its versatility in early stopping, re-estimating sample sizes, and shortening trial durations.

KEYWORDS:

• Seamless phase 2/3 trial
• co-primary endpoints
• conditional power
• group sequential design
• interim analysis

1. Introduction

Vaccination is one of the most effective ways to reduce mortality and morbidity from infectious diseases. The recent success in developing COVID-19 vaccines underscores the importance, as well as the feasibility, of fast development and approval of safe and effective vaccines [1–3]. The objective of this article is to introduce and develop the master-protocol-based, seamless phase II-III trial design strategy that could be used to accelerate the development of vaccine, in particular multi-valent vaccine.

Conventionally, vaccine development is a long, complex and expensive process, consisted of three separated phases. Phase 1 primarily focuses on safety and possibly immunogenicity, usually involves a small group of subjects (20–80). Phase 2 consists of more subjects (hundreds of people) to explore safety, immunogenicity, proposed doses, schedule of immunizations, and method of delivery. Phase 3 trials are randomized and double-blind confirmatory studies that continue to assess vaccine safety and immunogenicity on a much larger scale (thousands to tens of thousands of people) and explore rare adverse effects that may not be found in previous trials with small groups of subjects. With unprecedented progress of biomedical research, this conventional paradigm has become increasingly cumbersome and inefficient [4]. First, the conventional approach requires the separate development of a research protocol, regulatory compliance, funding, and related infrastructure for each phase of the development, resulting in excessive ‘time-to-clinical use,’ overhead and overlap of infrastructure. Second, the ‘white’ space between phases slows down the process of testing candidate agents. Third, the data from each phase are analyzed separately (e.g. phase 3 trials do not directly use the data from phase 2 trials), which wastes information and results in larger overall sample size.

Novel adaptive designs, including seamless designs and master protocol, provide a useful way to improve the efficiency and success rate of clinical trials, as noted by US Food and Drug Administration (FDA) guidance [5,6]. These novel approaches are widely accepted and increasingly used in cancer treatment development but are less known and under-utilized in vaccine trials. The objective of this article is to introduce and extend seamless phase 2/3 trial design strategy with application to multi-valent vaccine development.

There is a rich body of literature on phase 2/3 adaptive trial designs, many of which focus on oncology. Ellenberg and Eisenberger proposed a two-stage phase 2/3 design for binary outcomes, using futility early stopping rule after stage 1 [7]. Thall et al. developed a two-stage phase 2/3 design that optimizes the expected overall sample size, controlling over type I error rate and power [8]. Schaid et al. proposed a 2-stage phase 2/3 design to screen multiple experimental treatments in stage 1, following by randomized comparison to the control in phase 3 [9]. Bauer and Kieser proposed an adaptive two-stage design for the situation of multiple treatments to be compared with a control [10]. Stallard and Todd presented a group sequential approach incorporating treatment selection [11]. Liu and Pledger developed a two-stage phase 2/3 trial design with multiple doses [12]. Schmidli et al. provided several examples of phase 2/3 trial designs [13]. Jiang and Yuan proposed a seamless phase 2/3 strategy to reduce the sample size for dose optimization [4].

The majority of these designs evaluate the effectiveness of the treatment based on a single primary endpoint. Multi-valent vaccine trials, however, aim to demonstrate the efficacy of the vaccine for each pathogen and thus require the use of co-primary endpoints (CPEs). Such trials are defined as success if and only if all CPEs meet the efficacy requirement simultaneously [14,15]. For example, 9-valent human papillomavirus (HPV) vaccine trial requires that the geometric mean titer ratios of each of 9 HPV types (i.e. 9 CPEs) reach statistical significance [16]. In this paper, we elucidate a seamless phase 2/3 approach for multi-valent vaccine trials. Compared to the conventional vaccine trial design paradigm, the proposed design has several advantages: (1) requires only one master protocol to encompass both phase 2 and phase 3, simplifying the trial logistics, infrastructure requirements and regulatory approval; (2) is efficient as the design use the data from both phases to compare the vaccine with the control, thus reducing the total sample size and associated cost; (3) allows to re-estimate the sample size of phase 3 in light of phase 2 data, if the effect size assumed for the sample size determination at the trial design phase deviates from the truth, making the trial more adaptive; (4) allows early stopping for superiority if the trial shows better treatment effect than expected, shortening the vaccine development time.

The remaining of this article is organized as follows. Section 2 first describes the adaptive seamless phase 2/3 design for multi-valent vaccine trials in detail, then provides the simulation scheme for evaluating the operating characteristics of the proposed design and compares it with the conventional approach. Section 3 presents the results. Summary and discussion are provided in Section 4.

2. Methods

2.1. Trial design

In alignment with vaccine development objectives, distinct trial scenarios have been designed to validate the efficacy of K-valent vaccines. These include: (1) a seamless phase 2/3 design focusing on efficacy for interim analysis (denoted as Case 1), (2) a seamless phase 2/3 design considering both safety and efficacy for interim analysis (denoted as Case 2), and (3) a two-stage trial assessing interim analysis under both balanced and unbalanced sample allocation ratios (denoted as Case 3). Post-vaccination immunogenicity responses, presumed to follow a K-variate normal distribution, serve as CPEs for K = 1, … , K. For Cases 1 and 2, initially, a Phase 2 stage is conducted to evaluate M (m = 1, … , M) doses of an experimental vaccine against a positive control vaccine, aiming to select an optimal dose, S. Phase 3 stage is then conducted to compare dose S against the control. For Case 3, an experimental vaccine is compared with a positive control vaccine under r_T : r_C sample allocation ratios. Figure 1 illustrates the workflow for these scenarios. Detailed methodological insights are available in the Appendix.

Figure 1. Design flow chart.

*The conditional power is calculated based on information from the phase 2 stage.

^△The test statistic is calculated based on information from phase 2 and phase 3A stages.

2.1.1. Case 1: seamless phase 2/3 design based on efficacy for interim analysis

This proposed design consists of a phase 2 stage and a phase 3 stage that are seamlessly connected. The trial starts by equally randomized n₁ patients into the M doses of the vaccine and the control group. At the end of phase 2, an interim analysis is conducted to compare the M doses of the experimental vaccine against the control to identify the optimal dose with superior efficacy. The go/no-go decision leans on conditional power (CP), gauging the probability of obtaining statistically significant results if the trial continued and completed phase 3 stage with an additional n₂ participants. Should the CP for the optimal dose fall below a pre-specified boundary η (e.g. η = 0.5), indicating a slim probability of positive outcomes, the trial is halted; otherwise, it proceeds to the phase 3 stage. Of note, only the optimal dose proceeds to phase 3 stage, with suboptimal doses ceasing after phase 2 stage.

The interim data from Phase 2 stage offer an opportunity to adaptively refine the original trial design, enhancing alignment with trial objectives. Adaptations include sample size re-estimation and performing interim analysis in phase 3 stage for early efficacy determinations. The re-estimation of sample size is facilitated through CP, incorporating the anticipated overall power of the study, thereby updating the originally planned sample size n₂. The decision for early stopping is implemented using a group sequential approach based on the O’Brian-Fleming alpha spending function [17–19]. Specifically, after a certain proportion of information is collected during Phase 3 stage, the newly gathered information is combined with that from Phase 2 stage for interim analysis. The immunogenicity response results of the vaccine group and the control group are compared. If the statistic exceeds the pre-specified c_A boundary, the trial will be stopped early, and a conclusion of vaccine efficacy will be drawn; otherwise, the trial continues, and ultimately, an analysis based on the combined information from Phases 2 and 2 stages is conducted. The combined statistic is compared with the pre-specified c_B boundary. To control the inflation of Type I error due to sample size re-estimation, the inverse normal combination test is used to aggregate information from different stages when making decisions about efficacy.

2.1.2. Case 2: seamless phase 2/3 design based on safety and efficacy for interim analysis

Augmenting Case 1, this scenario introduces an additional safety stopping rule, enabling researchers to consider both efficacy and safety outcomes at the end of Phase 2 stage for dose selection and early stopping decisions. Specifically, the rate of adverse events (AEs) of grade 3 or higher occurring in the control group, denoted as p_AE,C, is used as the upper limit acceptable for researchers. At the end of Phase 2 stage, the rate of grade 3 or higher AEs, p_AE,m (m = 1, … , M), is calculated for each dose group of the experimental vaccine and compared with p_AE,C. If the p_AE,m for any dose exceeds that of the control group, that dose is excluded from the study. If all doses are excluded, the study is prematurely stopped and declared as a failure due to toxicity; Otherwise, the trial continues, and the remaining dose groups are subjected to futility stopping assessments. From the doses that survive both toxicity and futility evaluations, the one showing the best efficacy is selected to proceed to Phase 3 stage.

2.1.3. Case 3: two-stage design with interim analysis under even and uneven sample allocation ratios

Efficacy trials may benefit from control-favored randomization, whereas novel platforms or adjuvants might prioritize vaccine allocation for accelerated safety data accrual. Therefore, this scenario primarily focuses on how to conduct two-stage phase 3 design under conditions of unbalanced sample allocation ratios. Assuming the sample allocation ratio between the experimental group and the control group is r_T : r_C. Before the trial begins, the required sample size for each group under each endpoint is first estimated based on predetermined significance level, power, and estimated effect sizes. The largest sample size among all endpoints is chosen as the sample size for the entire study. If this sample size is below the minimum required by regulatory authorities, n_min, then n_min is chosen as the final sample size. After a certain proportion of information is collected during the trial, an interim analysis based on efficacy results is conducted, comparing the smallest statistic across K endpoints, z₁, against the futility stopping boundary, c_ful, and the efficacy stopping boundary, c_eff. If z₁ is less than c_ful, the trial is prematurely stopped and declared a failure; if z₁ is greater than c_eff, the trial is prematurely stopped and declared as a success; otherwise, enrollment continues until the anticipated sample size is reached. If the trial proceeds, data from both stages are combined, and the smallest statistic across K endpoints, z₂, is compared against the final analysis boundary, c_final. If z₂ exceeds c_final, the trial is declared a success. This scenario employs a group sequential method design, where the initial sample sizes for each group, the efficacy stopping boundary, the futility stopping boundary, and the final analysis boundary are calculated using an O’Brian-Fleming type alpha spending function.

2.2. Simulation study

Monte Carlo experiment was conducted to probe the performance of these cases. The simulation study is based on a non-inferior 9-valent HPV vaccine trial that evaluated immunogenicity and tolerability in heterosexual men (HM) and women 16–26 years old [16], full details in Supplementary Table S1. Briefly, the objective was to demonstrate non-inferior on month 7 geometric mean titers (GMTs). The trial was claimed successful when the lower bound of the two-sided 95% confidence interval (CI) of GMT ratio (HM/young women) is greater than the pre-specified statistical criterion (0.67) for each HPV type. In addition to studies with 9 CPEs, some HPV vaccines have 2 or 4 CPEs. Simulations for trials with 2 or 4 CPEs were also performed for sensitivity analysis purposes.

In the simulation study, we assume that the same endpoint (Geometric mean ratio, GMR) is used for the stop for futility/efficacy decision and the final analysis at the end of phase 3 stage. To demonstrate the non-inferiority of the experimental vaccine with respect to a control vaccine, we wish to test the null hypothesis H₀: $\mathit{GM}{R}_{\mathit{mk}}=\mathit{\delta }$ for at least one k, against the one-sided alternatives H₁: $\mathit{GM}{R}_{\mathit{mk}}>\mathit{\delta }$ for all k, where δ represents the pre-specified non-inferiority margin. In testing the preceding hypotheses, the null hypothesis H₀ is rejected if and only if all of the null hypotheses associated with each of the K CPE are rejected at a one-sided significance level of α.

We assume that higher dose result in higher immunogenicity response and higher toxicity. Thus, for the middle dose experimental group, the log_e transformed post-vaccination immunogenicity responses of each valent in is denoted as 0.2 lower than the high dose, with a toxicity rate 0.6% lower. Similarly, the log_e transformed post-vaccination immunogenicity responses for the low dose is marked as 0.4 lower than the high dose, with a toxicity rate 1.2% lower. To evaluate type I error rate, it is set that $\mathit{GM}{R}_{\mathit{mk}}=\mathit{\delta }$ for all K endpoints (m = 1, … , M; k = 1, … , K). The correlation among endpoints is denoted by ${\rho }_{\mathit{kk}{\mathit{ }}^{\mathrm{\prime }}}$ = (0, 0.3, 0.6) (k = 1, … , K, k′ = 1, … , K, k $\ne$ k′), representing three scenarios: endpoints being independent, moderate correlated, and highly correlated. On the contrary, to access type II error rate, a series of scenarios were generated that meet the requirement of $\mathit{GM}{R}_{\mathit{mk}}>\mathit{\delta }$ for all k, based on effect sizes observed in actual clinical studies, with the scaling factor r represented as r = Exp(μ_sim)/Exp(μ_real), indicating the ratio of simulated immunogenic response values for the high-dose group to the true values derived from the 9-valent HPV trial. When r < 1, the effect of the experimental vaccine is not as good as that of the real study; when r = 1, true responses are used for parameter setting; and r> 1 indicates that the effect of the experimental vaccine is better than that of the real study. The immunogenicity response of the control vaccine is fixed. Besides, we assume that the white space between phases 2 and 3 was six months. It took ten months from the first patient-in to the end of the phase 2 stage, and the duration of that of phase 3 stage was 12 months. Under each scenario 10,000 simulated trials were conducted. For more details of the simulation study, see the Appendix.

2.3. Performance metrics

For each of the scenarios, we calculated the following metrics, where indicators (a)–(d) apply to all Cases, and indicator (e) applies only to Cases 1 and 2.

(a) Power and type I error rate

Power is defined as the percentage of simulated trials that rejected the null hypothesis and concluded that the vaccine is effective when the vaccine actually is effective. Type I error rate is defined as the percentage of simulated trials that rejected the null hypothesis and concluded that the vaccine is effective when the vaccine actually is ineffective.

(b) Average sample size

The average sample size over 10,000 simulation trials.

(c) Trial duration

The average trial duration over 10,000 simulation trials.

(d) Percentage of early stop

The percentage of simulated trials be stopped early for toxicity, futility or efficacy. The first one is denoted as percentage of toxicity stop (TFS), the second is denoted as percentage of futility stop (PFS) and the last one is denoted as percentage of efficacy stop (PES).

(e) Percentage of correct dose selection

The percentage of correct dose selection is defined as the percentage of simulated trials where the optimal dose is correctly selected at the end of phase 2 and evaluated in phase 3.

3. Results

This study evaluated several key indicators. When it was assumed that the null hypothesis H₀ held, the focus was on the Type I error rate, average sample size, duration of the trial, and the proportion of early stops, detailed in Tables 1–3. Conversely, when the null hypothesis H₀ was not assumed to hold, assessments included the overall power, average sample size, trial duration, the proportion of early stops, and the frequency of correct dose selection, which were illustrated in Figures 2–6.

Figure 2. Performance of seamless design compared to conventional design for case 1 in terms of (a) Overall power, (b) Average sample size and (c) Trial duration when the null hypothesis is false.

r, the ratio of simulated post-vaccination immunogenicity response values of the high dose group to the real values derived from the 9-valent HPV trial, r = Exp(μ_sim)/Exp(μ_real).

Figure 3. Performance of seamless design for Case 1 in terms of (a) Early stop percentages, (b) Correct dose selection when the null hypothesis is false.

r, the ratio of simulated post-vaccination immunogenicity response values of the high dose group to the real values derived from the 9-valent HPV trial, r = Exp(μ_sim)/Exp(μ_real).

Figure 4. Performance of seamless design compared to conventional design for Case 2 in terms of (a) Overall power, (b) Average sample size and (c) trial duration when the null hypothesis is false.

r, the ratio of simulated post-vaccination immunogenicity response values of the high dose group to the real values derived from the 9-valent HPV trial, r = Exp(μ_sim)/Exp(μ_real).

Figure 5. Performance of seamless design for Case 2 in terms of (a) Early stop percentages, (b) Correct dose selection when the null hypothesis is false.

r, the ratio of simulated post-vaccination immunogenicity response values of the high dose group to the real values derived from the 9-valent HPV trial, r = Exp(μ_sim)/Exp(μ_real).

Figure 6. Performance of seamless design compared to conventional design for case 3 in terms of (a) Overall power, (b) Average sample size, (c) Trial duration and (d) Early stop percentages when the null hypothesis is false.

r, the ratio of simulated post-vaccination immunogenicity response values of the high dose group to the real values derived from the 9-valent HPV trial, r = Exp(μ_sim)/Exp(μ_real).

Table 1. Performance of seamless and conventional designs for Case 1 when the null hypothesis is true.

Abbreviations: K, number of endpoints; $\mathit{\rho }$, common correlation coefficient among endpoints; PES, percentage of efficacy stop; PFS, percentage of futility stop.

^aAverage sample size = 4n₁ +2n₂, the light green, green, and dark green regions indicate 100, 500, and 1000 subjects fewer for seamless design than for conventional design respectively.

^bThe light green, green, and dark green regions indicate 5, 10, and 15 months fewer for seamless design than for conventional design respectively.

^cThe light green, green, and dark green regions indicate [30, 60), [60, 90) and [90, 100] of PFS for seamless design.

Table 2. Performance of seamless and conventional designs for Case 2 when the null hypothesis is true.

Abbreviations: K, number of endpoints; PTS, percentage of toxicity stop; PES, percentage of efficacy stop; PFS, percentage of futility stop.

^aAverage sample size = 4n₁ +2n₂, the green region indicate 500 subjects fewer for seamless design than for conventional design.

^bThe light green, green, and dark green regions indicate 5, 10, and 15 months fewer for seamless design than for conventional design respectively.

^cThe light green and green regions indicate [30, 60) and [60, 90) of PFS for seamless design.

Table 3. Performance of two-stage and conventional designs for Case 3 when the null hypothesis is true.

Abbreviations: K, number of endpoints; PES, percentage of efficacy stop; PFS, percentage of futility stop.

^aAverage sample size = $n_{\mathit{T}1}+n_{\mathit{C}1}+n_{\mathit{T}2}+n_{\mathit{C}2}$, denote the whole sample size used for the experimental and control groups, the light green region indicate 100 subjects fewer for two-stage design than for conventional design.

^bThe light green region indicate 5 months fewer for two-stage design than for conventional design.

^cThe light green and green regions indicate [30, 60) and [60, 90) of PFS for two-stage design.

3.1. Comparative performance of seamless and conventional designs in case 1

Table 1 presented the performance of seamless and conventional designs under various indicators when the null hypothesis H₀ was true. It was observed that both designs managed to keep the Type I error rate below the 2.5% level across different scenarios. When there are two CPEs, the seamless design, despite using a larger average sample size than the conventional design across various correlation scenarios, consistently achieved an early stop for futility rate of 23.88% or higher and reduced the trial duration by at least 8.88 months. When the number of endpoints increased to four, the seamless design further decreased the sample size required, increased the rate of early stops, and shortened the trial duration in comparison with the conventional design. As the endpoints rose to nine, taking the scenario with zero correlation as an example, the seamless design demonstrated a 94.85% probability of correctly stopping an ineffective trial early, saving an average of 1232.5 samples and reducing the trial duration by 17.38 months compared to the conventional design. Comparing across different correlation scenarios, the seamless design performed best when the endpoints were independent, particularly in comparison to the scenario with an endpoint correlation of 0.6.

Figures 2 and 3 presented the performance of seamless design with respect to the conventional design under various scenarios when the GMT responses in the high dose group were non-inferior to those in the control group. It is worth noting that the seamless design performs better in terms of overall power compared with conventional design when r, the ratio between simulated GMT and real GMT values derived from the 9-valent HPV trial, was less than 0.85 (2 CPEs), 0.85 (4 CPEs) or 0.76 (9 CPEs). This trend was most pronounced when the endpoints were independent of each other (ρ = 0). For example, when r was equal to 0.83 (2 CPEs), 0.84 (4 CPEs) and 0.75 (9 CPEs), the overall power of the conventional design was 64.67%, 70.74% and 73.10%, respectively, whereas the seamless design achieved over 80% overall power. When the experimental vaccine had a good efficacy (r $\ge$ 0.9), both designs achieved over 90% overall power. Regarding the impact of correlation on overall power, Figure 2(a) revealed a gap in overall power between $\mathit{\rho }$ = 0 and $\mathit{\rho }$ = 0.6, specifically, the difference in overall power between scenarios with $\mathit{\rho }$ = 0.6 and $\mathit{\rho }$ = 0 for seamless trials with 2, 4, and 9 CPEs did not exceed 7.47% (r = 0.73), 12.19% (r = 0.73) and 16.03% (r = 0.70), respectively. When r was greater than 0.76 ($\mathit{\rho }$ = 0), 0.78 ($\mathit{\rho }$ = 0.3) and 0.74 ($\mathit{\rho }$ = 0.6), the difference in overall power was less than 5%, indicating that the impact of correlation on overall power was negligible.

In all scenarios, the average sample size (ASS) of the conventional design was fixed at 600 (2 CPEs), 600 (4 CPEs), and 1920 (9 CPEs). In contrast, the ASS for the seamless design was dynamically adjusted based on the results of the interim analysis. Figure 2(b) clearly demonstrated that as the scaling factor r increased, the ASS for the seamless design initially rose before decreasing. This pattern was similar across scenarios with an equal number of endpoints, regardless of the correlation between the endpoints. For instance, in scenarios where the endpoints were independent, the maximum ASS consumed by the seamless design were 695.72 for 2 CPEs (r = 0.76), 722.30 for 4 CPEs (r = 0.80) and 1945.29 for 9 CPEs (r = 0.74), with corresponding overall powers of 43.82% (2 CPEs), 67.55% (4 CPEs) and 75.02% (9 CPEs). Although these results did not meet our expectations, they were still higher than the 23.90% (2 CPEs), 35.24% (4 CPEs) and 58.72% (9 CPEs) achieved by the conventional design. Futhermore, when r exceeded 0.85 (2 CPEs), 0.90 (4 CPEs) and 0.76 (9 CPEs), trials employing the seamless design were able to save more samples while maintaining comparable overall power. For example, when the actual GMT levels (r = 1) were used in the simulation study, on average, saved 1004.99 subjects compared to conventional design.

Figure 2(c) presented the expected trial duration across different scenarios when the experimental vaccine was effective. In the conventional design, the durations of phase 2, phase 3, and the interim period between these phases were fixed, cumulating to a total trial duration of 28 months. However, the seamless design demonstrated a distinct advantage in reducing trial duration. Particularly when r ≥ 1.0, the seamless design trials achieved a reduction in duration of at least 11.16 (2 CPEs), 10.88 (4 CPEs), or 11.99 (9 CPEs) months compared to conventionally designed trials.

Figure 3(a) depicted the performance of the seamless design in terms of the percentage of stops at each scenario when the experimental vaccine was effective. Notably, the PFS dropped below 5% when the scaling factor r reached 0.84 for 2 CPEs, 0.90 for 4 CPEs, and 0.78 for 9 CPEs. Additionally, the PES exceeded 50% when r attained values of 0.90 for both 2 and 4 CPEs, and 0.79 for 9 CPEs.

Figure 3(b) presented the percentage of CDS for each scenario when the null hypothesis is false. The CDS percentage reached 83.31% (2 CPEs), 81.23%(4 CPEs) and 96.63% (9 CPEs) or higher when r was greater than or equal to 0.8.

3.2. Comparative performance of seamless and conventional designs in case 2

Table 2 illustrated the performance metrics under scenarios where the null hypothesis H₀ was true and the common correlation of 0.3 among endpoints was considered. Both the seamless design with added safety stopping rules and the conventional design managed to keep the Type I error rate below 2.5%. Trends observed under this Case mirrored those in Case 1, showing that when the TSP was at or below 0.04%, the seamless design typically achieved a PFS of at least 30.83% and reduced the trial duration by at least 9.71 months. The advantage of using the seamless design was most pronounced in scenarios with 9 endpoints compared to those with only 2 endpoints.

Figures 4 and 5 depicted the performance metrics under the assumption that the alternative hypothesis H₁ was true, again considering only a common correlation of 0.3, and including the additional safety stopping rules. With 2 and 4 endpoints, the probabilities of the low, medium, and high dose groups being excluded due to toxicity were 1.81%, 8.80%, and 19.24%, respectively, with a 0.04% chance of all doses being excluded. With 9 endpoints, these probabilities were 0.47%, 6.24%, and 22.24%, respectively, with a 0.01% chance of all doses being excluded. The additional safety stopping rule resulted in the high-dose group, theoretically the most effective and toxic, having a probability of being excluded of not less than 19.24%, leading to a further loss in overall power. From Figure 4(a), it is evident that although the seamless design could still ensure an overall power above 90% when r was greater than or equal to 1, it had similar or lower overall power compared to the conventional design, with the maximum difference reaching 19.22% (r = 0.9, 2 CPEs), 18.32% (r = 0.9, 4 CPEs), and 23.99% (r = 0.8, 9 CPEs). Figure 4(b–c) concluded that in terms of ASS and trial duration, the seamless design exhibited the flexibility to adjust sample size based on information collected at interim analysis, thus shortening the trial duration and maintaining superiority over the conventional design. This advantage was marked in trials with 9 CPEs, where, when the actual GMT levels (r = 1) were used in the simulation study, the seamless design saved 917.97 subjects on average compared to the conventional design. Even at r = 0.74, the seamless design still saved a minimum of 283.22 subjects relative to the conventional approach.

Figure 5(a) showed the performance of the seamless design in terms of the percentage of early stops when the alternative hypothesis H₁ was true, considering only a common correlation of 0.3, with the added safety stopping rule. Compared to Case 1 without the safety stopping rule, the seamless design used in Case 2 had a higher probability of incorrectly stopping trials early for futility and a lower probability of correctly stopping early for efficacy. The PFS in the seamless design dropped below 5% when r reached or exceeded 0.95 (for 2 CPEs), 0.95 (for 4 CPEs), and 0.9 (for 9 CPEs). The PES surpassed 50% when r reached 0.95 (for 2 CPEs), 1 (for 4 CPEs), and 0.82 (for 9 CPEs).

Figure 5(b) displayed the CDS percentage for each scenario. Similarly, affected by the safety stopping rule, the proportion of CDS in Case 2 was lower than that in Case 1. Specifically, at r = 0.8, the CDS percentage was only 67.75% (for 2 CPEs), 65.92% (for 4 CPEs), and 75.42% (for 9 CPEs).

3.3. Comparative performance of two-stage and conventional designs in case 3

Table 3 presented the performance of a two-stage phase 3 design compared to a conventional design across various metrics, under the scenario where the trial vaccine was ineffective and only $\mathit{\rho }$ = 0.3 was considered. Both approaches maintained the Type I error rate below 2.5% across all scenarios. It was observed that the sample size ratio between the experimental and control groups had a minimal impact on the trial duration, PES, and PFS. The ASS was quite similar under the two opposite allocation ratios of 2:1 and 1:2. Compared to the conventional design, the two-stage design had probabilities of correctly stopping ineffective trials at interim analysis not less than 26.71% (2 CPEs), 41.56% (4 CPEs), and 62.17% (9 CPEs); it saved at least 39.02 (2 CPEs), 58.95 (4 CPEs), and 261.11 (9 CPEs) subjects; and reduced the trial duration by 2.24 (2 CPEs), 3.49 (4 CPEs), and 5.22 (9 CPEs) months.

The difference in overall power between the conventional design and the two-stage design across all scenarios did not exceed ± 2%. To prevent the overlapping of lines due to highly similar results, Figure 6 only showed the performance of the two-stage design when the experimental vaccine was effective, considering the scenario with $\mathit{\rho }$ = 0.3. Figure 6 revealed that the performance was similar for both a 2:1 and a 1:2 sample size ratio between the experimental and control groups concerning overall power, ASS, trial duration, and the probability of early stopping. Further observation showed that, compared to a 1:1 sample size ratio, the unbalanced allocation ratios corresponded to higher overall power (Figure 6(a)), higher ASS used (Figure 6(b)), and shorter trial duration when the efficacy of the experimental vaccine was better than expected (r > 1) (Figure 6(c)). The difference brought about by the unbalanced allocation of sample size was most pronounced in trials with 9 CPEs as compared to those with 2 or 4 CPEs.

For the conventional design, when the number of endpoints was 2 and 4, the total sample size consumed was fixed at 300, 200, and 300 for sample size ratios of 2:1, 1:1, and 1:2, respectively; when the number of endpoints was 9, the total sample size consumed was fixed at 900, 600, and 900 for the respective ratios. The sample size consumed by the two-stage design was slightly lower than that of the conventional design across all scenarios.

Regarding trial duration for the conventional design, it was fixed at 12 months. The average trial duration of the two-stage design was consistently lower than that of the traditional design across all scenarios. When there were 9 CPEs and the anticipated efficacy of the trial vaccine was better (r = 1.2), the two-stage design could reduce the trial duration by up to 5.53 months (sample size ratio of 2:1) compared to the conventional design.

4. Discussion

Seamless phase 2/3 design has been developed as an alternative to conventional design by combining both the dose selection and confirmation phases into one trial. By using seamless designs, the trial duration could be reduced by skipping the long pause between phase 2 and phase 3 and sample size could be saved by early stopping. There has been considerable interest in seamless design that focus on trials with a single endpoint [20,21], or trials with 2 or 3 CPE [22–24]. However, the performance of seamless designs in multi-valent vaccine trials with multiple CPE can be influenced by many factors, such as the variation of effect size among endpoints, number of endpoints, and correlation among endpoints. Whether seamless design can offer substantial benefits for the development of vaccine with multiple CPEs remains a question. To address this, we introduced a seamless phase 2/3 design suitable for trials with CPEs (Case 1), which allows for early futility stopping decisions and sample size re-estimation at the end of phase 2, as well as early efficacy stopping decisions during phase 3. Furthermore, we explored how to carry out a seamless design considering both safety and efficacy (Case 2). Lastly, the investigation centered on the phase 3 phase, examining the performance of group sequential in interim analysis with even and uneven sample size allocations (Case 3).

A series of simulations were conducted referencing an actual 9-valent HPV vaccine trial. The results from Case 1 suggest that seamless designs are particularly appealing in trials with multiple CPEs, as they allow for early stopping due to futility or efficacy, with a timely re-estimation of sample sizes based on phase 2 data, and reduce the overall duration while controlling the type I error rates. The appeal of seamless designs increases with the number of endpoints and decreases with the correlation among endpoints. The results from Case 2 indicate that when the safety endpoint is considered alongside efficacy, some dose groups may be eliminated due to excessive toxicity. Compared to a seamless design that did not take safety into account, the design used in Case 2 maintains good performance in terms of early futility stopping and trial duration when the experimental vaccine is ineffective. Conversely, when the vaccine is effective, this design offers the flexibility by increasing or decreasing the sample size based on the effect size of subsequent doses. While Case 2 simulations indicate higher overall power for the conventional design versus the seamless design, this should not suggest the conventional design’s superiority. This conclusion assumes the conventional design’s phase 2 relies exclusively on efficacy for dose selection, disregarding safety. However, if the conventional design were to integrate safety considerations, toxic doses might be eliminated, potentially diminishing its overall power. Lastly, the findings from Case 3 reveal that multi-stage phase 3 study employing a group sequential approach perform well under both balanced and unbalanced sample size allocations. Based on these findings, we advocate for the use of seamless designs in phase 2 and phase 3 trials for multivalent vaccines.

Concerns may exist regarding type II error rate and the resultant sample size inflation for trials with multiple CPEs, as highlighted in literatures [25–27]. In our study, the conditional power that the non-inferiority hypothesis will be met if the study continues is calculated for each endpoint at the end of phase 2 part. Then, the minimum CP is selected for the calculation of sample size required for the next stage. Our results show that the loss of overall power due to CPEs is acceptable. One possible explanation is that the endpoints are widely differed in effect size and variability in the real world. The family-wise type II error rate for the whole study will be controlled if the type II error rate for the one endpoint that is least likely to success were controlled. Our approach is consistent with the guidance for the evaluation of combination vaccines for preventable diseases published by the Center for Drug Evaluation and Research [28], which recommends that the sample size required for each endpoint should be calculated separately and the largest sample size should be selected as the sample size for the overall trial. It should also be highlighted that for innovative vaccines or platforms, regulatory agencies typically require a minimum number of 3,000 subjects for the experimental group [29]. In such contexts, the application of seamless design may be somewhat constrained. When the experimental vaccine is ineffective, seamless design can still expedite the saving of research time and resources through early futility stopping. However, when the experimental vaccine is effective, in order to meet the sample size requirements of regulatory authorities, the seamless design may have to compromise some of its flexibility in sample size adjustment and early stop for efficacy.

Another major concern of the regulatory agencies in developing seamless phase 2/3 design is whether type I error rate can be controlled [30]. Theoretically, both the re-estimation of sample size and interim decisions on efficacy could lead to inflation of type I error. To prevent such inflation, two steps were taken in our study. The first step was to use combination test, whereby accumulated information was combined using a weighted inverse normal method at each look. The second step was to calculate the efficacy stopping boundaries using the O’Brien-Fleming type alpha spending function, which set a high standard for statistical significance early in the trial, providing a conservative testing strategy while maintaining the type I error rate. Additionally, according to the findings of Chang’s research [31], the stopping boundaries of group sequential design for a trial with CPEs are the same for a trial with a single endpoint, since this design is alpha-exhaustive.

One of the primary appeals of seamless phase 2/3 design is its ability to skip the white space between phase 2 and phase 3 stages, thereby reducing the overall trial duration. Our findings in Case 1 for trials with 9 CPEs show that when the experimental group is effective (r = 1), the total duration of the seamless design can be reduced to 42.82% of that of the conventional design. In contrast, when the vaccine is ineffective, the seamless design, on average, shortens the trial’s overall duration to 53.25% of the conventional design. However, as the correlation between endpoints increases, the duration required for the seamless design extends, albeit still shorter than the conventional design. Asakura et al. use the group sequential futility boundary adjusted by the correlations to evaluate a joint effect on two CPEs [32]. In trials with multiple CPEs, further work is required to refine seamless design methods for futility stopping decisions based on conditional power. The study’s conclusions, while based on certain assumptions, simplify the intricacies of real-world trials whose durations are influenced by various factors. For example, multicenter trials across various countries might require less time. Sensitivity analysis, using Case 1 as a reference, was conducted to determine how such conditions affect seamless design. Detailed in the appendix, the results indicate that seamless design consistently shortens trial duration by 42.79% when effective and 56.71% when not, relative to conventional methods. This underscores the time efficiency of seamless designs even when conventional designs are expedited. Naturally, this conclusion is predicated on the seamless execution of the seamless design, while, as suggested by Maca et al. [33], there may not be much time savings for seamless design compared to conventional design unless the phase 3 stage is initiated quickly after the phase 2 part.

This study focuses on the performance of seamless phase 2/3 designs that utilize multiple continuous outcomes as CPEs. The methodologies employed in this research are adaptable to a broader range of scenarios. Building on this foundation to include phase 1 clinical trial could significantly enhance the rapid development of vaccines, particularly in the context of a pandemic. Reflecting on lessons from the COVID-19 pandemic, the Coalition for Epidemic Preparedness Innovations has advocated for a ‘100-day mission’ [34], aiming to develop and initiate production of a viable vaccine within 100 days following the identification of a potentially pandemic pathogen. The insights gleaned from this study are instrumental in advancing this objective. Future research will focus on how to effectively incorporate phase I clinical trials into this accelerated framework, aiming to streamline and optimize the vaccine development process further.

5. Conclusion

In summary, the use of seamless design in trials with multiple CPEs is very attractive because of its flexibility and effectiveness in stopping the trial early for efficacy or futility, re-estimation the sample size based on information accumulated at interim analysis and shortening overall duration while maintaining a relatively small type I error rate.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contribution statement

All authors have made substantial contributions to the conception and design of this article. All authors contributed to the writing of the manuscript and have critically revised the work for important intellectual content. All authors have approved the final version of the manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/14760584.2024.2348612.

Additional information

Funding

This manuscript was funded by Jiangsu Province “The 14th Five-year Plan” Key Discipline - Public Health and Preventive Medicine.