Generalized ROC methods for immunogenicity data analysis of vaccine phase I studies in a seropositive population

Figures & data

Figure 1. Youden index (YI) threshold method. (A) Probability distribution curves of overall baseline levels and postvaccination levels from RSV sF 20, 50, or 80 μg ± 2.5 µg GLA-SE plotted on the log-scale. The dashed line represents the YI threshold value for each biomarker. Threshold values and (baseline percentile) for MN IC₅₀ titer = 739.2 (76%); anti-F IgG = 221.6 units/mL (91%); PCA = 20.7 µg/mL (96%), and ELISPOT = 42.2 SFC/10⁶ PBMC (74%); (B) Percentage of subjects with postvaccination values greater than or equal to the threshold value for each assay or simultaneously for multiple assays. The values for all doses combined are used as a benchmark (white background), and percentages from individual doses are presented in red (lower than), white (equivalent to) and blue (higher than) all doses combined. Red and blue gradients represent the strength of the difference, with the darkest color representing the greatest difference. Ab, antibody; ELISPOT, enzyme-linked immunospot; GLA-SE, glucopyranosyl lipid adjuvant-stable emulsion; IgG, immunoglobulin G; MN, microneutralizing antibody; PBMC, peripheral blood mononuclear cells; PCA, palivizumab-competitive antibody; RSV, respiratory syncytial virus; sF, soluble fusion protein; SFC, spot-forming cells.

Figure 2. ROC-B method. (A) ROC curves for individual assays. The x-axis represents baseline percentiles and the y-axis is the probability that the postvaccination response was greater than or equal to the baseline percentile threshold. The ROC curve is shown for each formulation and combined arms of 20, 50, or 80 μg of RSV F protein ± GLA-SE; (B) Three-dimensional ROC surface plot for the combination of results from the MN and ELISPOT assays compared to baseline values from all arms combined. The x-axis and y-axis are baseline percentiles for ELISPOT and MN data, respectively, and the z-axis is the percentage of postvaccination values greater than or equal to both the MN and ELISPOT baseline percentile thresholds. The ROC surface is shown for each formulation and for the combination of all (20, 50, and 80 μg) dosage levels of sF combined ± 2.5 µg GLA-SE. The ROC surface for the all doses combined is shown in green and used as a benchmark, and the ROC surface for each individual formulation is shown in red, green and blue representing values on the z-axis lower than, equivalent to and higher than the benchmark ROC surface. ELISPOT, enzyme-linked immunospot; GLA-SE, glucopyranosyl lipid adjuvant-stable emulsion; IgG, immunoglobulin G; MN, microneutralizing antibody; PCA, palivizumab-competitive antibody; ROC, receiver-operating characteristic; ROC-B, receiver-operating characteristic relative to baseline; RSV, respiratory syncytial virus; sF, soluble fusion protein.

Figure 3. ROC-P method. (A) ROC curves for individual assays. The x-axis represents percentiles of the combined postdose values, and the y-axis is the percentage of postvaccination levels greater than or equal to the corresponding percentile threshold. The ROC curve is shown for each dose arm alone and for the combination of all vaccine formulations; (B) Three-dimensional ROC surface plot for the combination of results from the MN and ELISPOT assays compared to postdose values from all arms combined. The x-axis and y-axis are percentiles of the combined postdose distribution for ELISPOT and MN values, respectively, and the z-axis is the percentage of postvaccination responses greater to or equal than both the MN and ELISPOT thresholds. The ROC surface is shown for each formulation and for all formulations combined. The ROC surface for the all doses combined is shown in green and used as benchmark, and the ROC surface for each individual formulation is shown in red, green and blue representing values on the z-axis lower than, equivalent to and higher than the benchmark ROC surface. ELISPOT, enzyme-linked immunospot; GLA-SE, glucopyranosyl lipid adjuvant-stable emulsion; IgG, immunoglobulin G; MN, microneutralizing antibody; PCA, palivizumab-competitive antibody; ROC, receiver-operating characteristic; ROC-P, receiver-operating characteristic of postdose levels; VUS, volume under the surface.

Table 1. Outline of the 3 methods.

Method 1: Youden Index (YI) Threshold Method • Requires overlap of distributions of baseline and postvaccination values • May not be able to efficiently differentiate between vaccine doses if baseline levels are low and/or have small overlap with postvaccination levels • Apply a single YI threshold value for each assay that best discriminates between overall baseline and postvaccination biomarker levels • For each dose formulation and all doses combined, calculate percentage of subjects with postvaccination values greater than or equal to the YI threshold for each assay individually or simultaneously for assay combinations Method 2: ROC Relative to Baseline (ROC-B) Method • Uses the distribution of all baseline values as a reference • Requires overlap of distributions of baseline and postvaccination values • May not be able to efficiently differentiate between vaccine doses if baseline levels are low and/or have small overlap with postvaccination levels • For each dose formulation and all doses combined, construct ROC curve and surface using threshold values from the full range of baseline levels • Calculate AUC of ROC curve for individual assay and VUS of ROC surface for assay combinations • AUC and VUS values measure degree of separation of postvaccination values of individual formulations from the distribution of overall baseline values Method 3: ROC of Postdose (ROC-P) Level Method • The ROC-P method uses postvaccination data only and can be applied to situations when the YI and ROC-B methods are not suitable • Applies to both seropositive and seronegative populations • Uses the pooled postvaccination values as a reference distribution • For each dose formulation and all doses combined, construct ROC curve and surface using threshold values from the full range of postdose levels • Calculate AUC of ROC curve for individual assay and VUS of ROC surface for assay combinations • AUC and VUS values measure degree of separation of postdose values of individual formulations from the distribution of pooled postvaccination values Dose selection approach: Each of the 3 methods evaluates and visualizes the immunogenicity of different formulations by comparing each formulation to the overall dose effect of all formulations combined and selecting the formulation with the highest and most robust immune responses.

AUC, area under the curve; ROC, receiver-operating characteristic; VUS, volume under the surface.