Qualification of a 21-valent pneumococcal urine antigen detection assay and development of clinical positivity cutoffs

Abstract

Aim: Serotype-specific assays detecting pneumococcal polysaccharides in bodily fluids are needed to understand the pneumococcal serotype distribution in non-bacteremic pneumonia. Methods: We developed a urine antigen detection assay and using urine samples from adult outpatients without pneumonia developed positivity cutoffs for both a previously published 15-valent and the new 21-valent assay. Clinical sensitivity was confirmed with samples from patients with invasive pneumococcal disease. Results: Total assay precision ranged from 7.6 to 17.8% coefficient of variation while accuracy ranged between 80 and 150% recovery, except for three serotypes where recoveries ranged from 32 to 60%. Clinical sensitivity was 86.4% and specificity was 96.5% across all 30 serotypes. Conclusion: The assay could potentially assess serotype-distribution in non-infected and infected participants with pneumococcal disease.

Article highlights

Assay development

• A 21-valent serotype-specific urine antigen detection assay was developed to measure capsular polysaccharides and fragments thereof from Streptococcus pneumoniae (Pneumococcus)

• Pneumococcal polysaccharides are captured from urine samples using a multiplexed, sandwich Luminex immunoassay with each bead representing a distinct serotype.

Assay qualification

• Assay qualification determined assay performance: total assay precision ranged from 7.6 to 17.8% coefficient of variation while accuracy ranged between 80 and 150% recovery, except for three serotypes where recoveries ranged from 32 to 60%.

• Low spike recoveries of 15A, 23A and 35B were mostly due to potential urine interference.

• Assay specificity was assessed against bacterial lysates from 61 pneumococcal serotypes and 11 non-pneumococcal bacteria.

A pre-specified strategy to determine the clinical positivity cutoffs

• A novel method was developed to determine clinical positivity cut-offs based on nonparametric tolerance intervals combined with assay LLOQs and further enhancement based on serotype prevalence.

• The new cutoffs were then determined using a cutoff strategy, primarily based on a statistical method called nonparametric tolerance interval (NTI).

• The extended strategy considered using both the NTI method and the values of the low LOQs of the assays to jointly determine the cutoffs.

• The cutoffs were then refined by utilizing negative control urine samples from adult outpatients without pneumonia (same data previously used) and urine samples from patients with community acquired pneumonia with invasive pneumococcal diseases (positive blood cultures obtained from a sterile site).

• Clinical sensitivity was 86.4% and specificity was 96.5% across 30 pneumococcal serotypes measured by a previously published 15-valent and this 21-valent assay.

Keywords: :

• clinical cutoff
• pneumococcus
• SSUAD
• urine antigen detection

1. Background

Streptococcus pneumoniae (S. pneumoniae or pneumococcus) is one of the leading bacterial causes of pneumonia responsible for substantial morbidity and mortality globally [1–3]. Pneumococcus is a gram-positive bacterium with an outer polysaccharide capsule that has been the target of current vaccines. More than 100 serotypes have been identified that are characterized by their unique polysaccharide capsule [4,5]. Although the polysaccharide capsule is used for serotyping of clinical isolates, the pneumococcal phenotype is much more varied during its in vivo lifecycle [6]. Polysaccharide expression can be down-regulated temporarily to allow transition to an invasive stage, and lengths can vary depending on the environment between the nasopharyngeal space, inner ear space and blood circulation [7]. During an infection, capsular polysaccharides can be isolated from sputum, blood and urine as well as middle ear fluid in case of otitis media. The path from a bronchial pneumococcal infection to the appearance of associated polysaccharides in urine during a noninvasive disease is not well understood [8], and polysaccharides are likely fragmented to different degrees.

While vaccines have greatly reduced the frequency of pneumococcal disease [9–12], the burden of noninvasive community-acquired pneumonia, and specifically, noninvasive pneuymococcal pneumonia, still requires more data to better understand the residual disease burden through pneumococcal serotype distributions. Several publications investigate the distribution of pneumococcal serotypes associated with invasive pneumococcal disease (IPD) [13–19]. However, without a sensitive and specific in vitro assay, serotype-specific assessment of noninvasive pneumococcal disease can be challenging. IPD can be serotyped through latex agglutination and Quellung reactions of clinical isolates with immune-sera [20–23]. This is not a path for noninvasive disease where there are rarely bacterial cultures available.

Pneumococcal capsule polysaccharides including fragments thereof in urine samples allow serotype-specific assessment of pneumococcal infection. Several pneumococcal urine antigen detection assays have been developed covering different serotype compilations and applying different approaches to cutoff determination [8,24–30]. The cut off is defined as the response threshold above which a sample is considered positive, and below which a sample is considered negative. For urine antigen detection assays, cutoff determinations are often hampered by the lack of an orthogonal assay or a “gold standard” for serotype-specific, noninvasive pneumococcal disease. As mentioned above, the utility of microbial cultures from sterile sites is limited, and approved point-of-care urinary diagnostic tests do not distinguish serotypes. Interestingly, available data with currently available urine assays, both serotype-specific and non-serotype-specific, do not entirely correlate indicating the impact of multiple factors on the presence of pneumococcal urine antigens [8,25,26,29]. Non-serotype specific assays can be positive for serotypes that are not covered by the specific assays, resulting in an expected discrepancy between the assays. However, there are also cases where a serotype-specific positive sample does not have a matching positive result in a non-serotype specific assay. Different analytical assay sensitivities may be the simplest possible explanation. However, as detailed knowledge of the pathways leading to the presence of pneumococcal polysaccharides in urine is limited, other more speculative explanations cannot be ruled out. For example, polysaccharides may be fragmented in individual samples in a way that certain epitopes are preserved and others. This might occur randomly and on an individual basis [31–33]. Another possibility could be differential clearance by the immune system could cause specific polysaccharides of be cleared slower than polysaccharides with commonly shared epitopes to which an immunologic memory may exist from prior infections [34,35].

Thus, cutoffs for pneumococcal urine antigen detection assays are developed based on polysaccharide distributions in urine samples from patients with no known active pneumococcal disease as ‘true negatives’ and with urine samples from patients with invasive disease as ‘true positives’. However, both populations also contain false determinations. For example, non-symptomatic pneumococcal carriers or recent vaccine recipients would be labeled as ‘true negatives’ when they could have pneumococcus antigens present in the urine. Similarly, not all invasive disease samples contain measurable urine pneumococcal polysaccharides which depend on timing between blood and urine samples, impact of concurrent antibiotics, the nature of polysaccharide expression and the polysaccharide pathways and degradation on its way to the urine sample [36].

We developed a 21-valent multiplexed pneumococcal serotype-specific urine antigen detection assay (SSUAD) against serotypes that based on recent epidemiological data are expected to be common among the USA adult population with community-acquired pneumonia. Assay performance was assessed in qualification studies. In addition, we developed clinical cutoffs and assessed clinical specificity and sensitivity across two SSUAD assays, a 15-valent assay published previously [29] and the new 21-valent assay to cover a total of 30 serotypes (six serotypes are shared between both assays).

Our assay will allow further evaluation of the disease burden.

2. Methods

2.1. Pneumococcal serotype-specific urine antigen detection assay

Pneumococcal capsular polysaccharide-specific monoclonal antibodies (mAbs) used to capture and detect polysaccharides were developed by Merck & Co., Inc., West Point, PA, USA. Antibodies were generated in rabbits that were immunized with pneumococcal polysaccharides (PnPS) conjugated to CRM197 as used in the pneumoconjugate vaccines. Antibodies were selected for specificity against serotypes present in the 15- and the 21-valent vaccines. Specificity was then confirmed in bioanalytical samples against non-vaccine pneumococcal serotypes and several non-pneumococcal bacteria as described in the Specificity section below. PnPs used as standard and to generate qualification samples were manufactured by Merck & Co., Inc., PA, USA, as well. Goat anti-rabbit-IgG- (Fc fragment specific) conjugated to R-phycoerythrin (PE) was purchased from Jackson ImmunoResearch (PA, USA, cat. #111-117-008). Casein blocker was purchased from Thermo Scientific (cat. #37528) and the xMAP Antibody Coupling Kit and Luminex Magplex microspheres were purchased from Diasorin-Luminex (cat. #40-50016 and MC100XX-04, respectively).

Iron-core microspheres (Luminex Magplex) were conjugated with anti-pneumococcal capsular polysaccharide specific mAbs as described previously [29]. Antibody concentrations are provided in Supplementary Table S1. Microspheres with different spectral regions and coupled to different anti-polysaccharide antibodies were mixed to create a 21-valent cocktail of 5 × 10⁴ capture beads/ml, in other words, approximately 2500 beads/region and assay well. Twofold diluted urine samples or controls were mixed 1:1 with assay buffer (0.05% Casein, 0.05% Tween-20 in phosphate-buffered saline) in a total volume of 50 μl in a 96-well microtiter plate. A twofold serially diluted, 11-point standard was prepared separately in assay diluent. Then 50 μl of the bead cocktail were added to the plate and incubated for 15–18 h at 23 ± 2°C. After washing the plates three times with phosphate-buffered saline + 0.05% Tween-20, a mix of 21 rabbit anti-pneumococcal polysaccharide-specific mAbs was added and incubated for 60–75 min. After another wash, the phycoerythrin-labeled goat anti-rabbit Ab was added and incubated for 40–50 min. followed by a wash and resuspension of microspheres in Dulbecco's phosphate-buffered saline (DPBS) for read in a Luminex FlexMap3D instrument with median fluorescence intensity as read-out.$y_i=f(x_i;\theta )+{\varepsilon }_i$ $f(x_i;\theta )=D+\frac{A-D}{1+{\left(x_i/C\right)}^{-B}},\hspace{0.17em}{y}_i$

The standard curve was modeled using a four-parameter logistic function with a weighting factor of 1.6. The function has the form, , where is the average of the replicate median fluorescence intensity (MFI) readings, is the concentration of the standard and are the asymptotic maximum and minimum parameters, respectively, is the slope parameter and is the concentration corresponding to the median response (EC₅₀).

2.2. Assay qualification experiments & statistical methods

2.2.1. Accuracy & limit of detection

The accuracy evaluation was performed in two assay runs consisting of four plates per run. Each of two analysts performed one assay run. Urine samples from eight individual subjects were pre-screened in the assay to have low or no measurable PnPs concentrations for most serotypes. These urines were then spiked with a known concentration of each of the 21 PnPs. 13 aliquots of each individual urine sample were spiked with varying levels of the pooled PnPs reference standard across the range of the assays. Successive spikes were twofold less than the previous spike concentration. The 14th aliquot of each urine sample was no spike (NS) urine sample run in triplicate on the plate.

PnPs recovery for each spiked sample was calculated as Percent Relative Accuracy = 100% × (Observed Concentration /Expected Concentration), where the observed concentration is the interpolated PnPs concentration, and the expected concentration is the PnPs concentration spiked into the sample. For each test sample, the observed concentration was corrected for the concentration in the NS sample matrix where measurable. Samples where NS samples were < LLOQ did not require concentration correction. Overall relative accuracy (Geometric Mean of %Recovery) was determined for each of the 13 spike concentration levels by combining the 16 estimates for each concentration level.

The MFI signals of spiked and unspiked samples in accuracy evaluation were used to determine LOD. The differences in MFI values between the spiked samples and the unspiked samples were determined separately within each run for each serotype and then averaged across runs for each spike level. The lowest spike concentration level for which the one-sided 95% lower prediction bound on the individual differences is >0 is considered to be the limit of detection (LOD), where significance is based on a t- distribution at the 5% significance level.

2.2.2. Precision & ruggedness

The individual sample precision and the total or intermediate assay precision were assessed across eight runs consisting of one or two plate(s) per run. Each of two analysts performed four runs across four different days and across 2 weeks. To assess assay precision and ruggedness across the range, 16 normal individual human urine samples were each spiked with one of 16 alternative concentrations of the full-length PnPs chosen to span the range of PnPs concentrations generated by the reference standard. Spike levels were progressively decreased by 1.5-fold. The urine samples were also tested within each plate without PnPs added to assess the background response in the urine matrix and were also used as low concentration samples if quantifiable.

The total variability of each sample was obtained by fitting a mixed model to the natural log transformed concentrations with a random term for Run. The assay variability estimates across samples were obtained on the natural log transformed concentrations using a mixed model containing a fixed effect for Sample and a random effect for Run. The same dataset used to assess the assay precision was also used to assess the ruggedness of the assay to analyst.

2.2.3. Dilutional linearity

The dilutional linearity of the assay, which is often referred to as parallelism or dilutability, was evaluated in two assay runs consisting of two plates per run. Each of two analysts performed one assay run. A known concentration of PnPs was spiked into each of 8 individual human urine samples and diluted twofold for 8 points (neat, 1:2, 1:4, 1:8, 1:16, 1:32, 1:64 and 1:128) to create 64 samples before testing.

Dilution linearity was assessed across the range of dilutions evaluated based on regressing the log10 transformed dilution corrected PnPs concentrations against the log10 transformed dilutions. The dilution linearity was estimated by fitting a mixed model that included fixed terms for Sample and for the average dilution effect (slope) and a random term representing the variability in the dilution effect across the individual test samples. Dilution fold-bias per tenfold dilution was calculated as 100% × (10^b) where b represents the estimate of the average dilution effect (slope) from the mixed model.

2.2.4. Limit of quantitation

The assay limit of quantitation (LOQ) was determined by evaluating the dilution-response profile of the reference standard of all experiments, the precision profile for individual test samples and the accuracy of the assay. The LOQ was the quantifiable concentration range with acceptable performance of the assay, i.e., ≥ 80% of the test samples with concentrations within the LOQs must have variability estimates of ≤ 25% CV and overall percent relative accuracy between 80% and 150%. In several cases, a more conservative approach was taken to further limit the ranges for practical reasons beyond assay precision and accuracy performance to the second lowest and second highest standard curve points (Supplementary Table S3).

2.2.5. Selectivity

The selectivity evaluation for the potential of urine interference in the measurement of PnPs was performed in two assay runs consisting of two plates per run. Each of two analysts performed one assay run. Eight different urine samples were tested, and each individual urine sample was used neat and diluted twofold (up to seven levels) in assay buffer to create a subset of eight samples (neat, 1:2, 1:4, 1:8, 1:16, 1:32, 1:64 and 1:128), creating a total of 64 samples. Then, a known concentration of PnPs was spiked into all the samples so that each sample contained the same concentration of spiked PnPs with increased diluted urine in assay buffer.

For a given sample, the Percent Recovery was calculated as 100% × (observed Concentration / Expected Concentration), where the observed concentration is the interpolated PnPs concentration, and the expected concentration is the PnPs concentration spiked into the sample. The average percent recoveries were the geometric mean recovery estimates across 8 samples at each pre-dilution level of urine.

The selectivity expectations were recovery ranges between 80% to 125%.

2.2.6. Specificity

Homologous specificity against vaccine antigens:

The specificity evaluation was performed in two assay runs consisting of one plate per run. Each analyst performed one assay run containing 25 samples. Samples 1 – 21 were comprised of pools of 20 PnPs, with each sample pool missing a different PnPs serotype. All sample pools had the same concentration of PnPs spiked into pooled normal human urine pre-screened to have low or no PnPs concentrations. Samples 22 and 23 were the positive urine with all 21 PnPs tested twice, and samples 24 and 25 were the unspiked normal human urine pool tested twice.

The ratio of the measured concentration to the spike concentration was determined for each measured value, and the resulting geometric mean ratios were estimated across two runs. Any measurable PnPs concentration for the unspiked sample for a given serotype was subtracted from the spiked sample measurement for that serotype. Specificity was determined as a measured concentration of <10% of the assay range in samples missing the serotype of interest.

Similarly, during development, cross-reactivity was assessed against antigens from the 15-valent assay and against cross-reactive serotypes 6C and 15B.

Heterologous specificity against non-vaccine antigens:

Additional specificity assessments were made during assay development against non-vaccine pneumococcal strains and other bacteria. Lysates of non-pneumococcal bacterial isolates, Acinetobacter baumannii, Escherichia coli K12, Klebsiella pneumoniae, Klebsiella quasipneumoniae, Pseudomonas aeruginosa, Enterococcus faecalis, Neisseria gonorrhoeae, Clostridium difficile, Bacteroides fragilis, Staphylococcus aureus SACO1, Staphylococcus aureus, RN4220 were generated as described [37] and the lysates were tested with 15-plex and 21-plex SSUAD.

2.3. Clinical sample population

Data are derived from patients attending routine outpatient follow ups or healthy participants part of the PNEUMO study at Emory University funded by Merck Sharp &Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA and approved by Emory IRB (IRB00105020). Written patient consent was obtained for the study.. Participants did not have evidence of pneumonia nor pneumococcal vaccination for at least 3 months prior to enrollment. A total of 990 samples were used in the 21-valent assay and 940 samples in the 15-valent assay. Note that many patients had multiple conditions and were therefore accounted for more than once across the different categories (Table 1). The age ranged from 19 – 94 years with a median of 60 years and an average of 57 years.

Table 1. Clinical urine samples without known pneumococcal disease used for clinical specificity evaluation.

Co-morbidity	Patient number
Pulmonary disease	206
Cardiovascular disease	326
Diabetes	283
Healthy	269

Pulmonary disease included COPD, asthma, emphysema and chronic bronchitis. Cardiovascular disease included hypertension, congestive heart failure. Renal disease included chronic kidney disease. Note: Some patients also had multiple underlying diseases and were accounted for in each applicable category.

Clinical sensitivity using these cutoffs was assessed with urine samples from patients with diagnosed invasive pneumococcal disease from US and European centers participating in the PNEUMO study (Table 2).

Table 2. IPD samples used for clinical sensitivity evaluation with serotyping results determined by blood culture isolates.

Assay serotype alignment with vaccine types	Number of samples
21-valent assay unique serotypes	32
15- and 21-valent assays shared serotypes	21
15-valent assay unique serotypes	6
15- and 21-valent assays 30 serotypes	59^†
Non-vaccine serotypes	11
Total	70

† Some samples were applicable to multiple sub-categories so that the sum of the category numbers exceeded up to the total number of 70 samples.

3. Results

3.1. Assay development

The 21-valent SSUAD is a sandwich immunoassay using serotype-specific mAbs to capture and detect pneumococcal polysaccharide fragments in human urine samples. The assay follows similar principles as a 15-valent SSUAD published previously (Figure 1A) [29]. Individual antibody concentrations, incubation times and the minimal required sample dilution were optimized to maximize the assay range and the signal-background ratios (data not shown).

Figure 1. Development of the 21-valent SSUAD. (A) Schematic representation of the assay workflow, not drawn to scale. (B) Representative standard curves from four plates from selectivity testing. Vertical lines indicate the assay range.

3.2. Assay qualification

The assay performance was assessed for precision, ruggedness, accuracy, dilutional linearity, selectivity and specificity in the 21-valent assay qualification and the qualification results were summarized in Table 3 and the consolidated results were listed in Table 4. Details of assay performance at the range limits are presented in Supplementary Table S3. Representative standard curves are shown in Figure 1B.

Table 3. Assay qualification performance in 21-valent assay.

Performance characteristic	Result
Precision	Total assay precision ranged between 7.6 and 17.8% CV across the 21 serotypes
Ruggedness	PnPs concentrations differences were within 19% between analysts
Accuracy	Relative accuracy was estimated between 80% and 150% across the quantifiable range for 18 serotypes. The recoveries were around 60% for 15A, 52% for 23A and 32% for 35B, respectively, due to potential urine interference
Dilution linearity	The dilution fold-bias per tenfold dilution was <2-fold for all 21 serotypes
Selectivity	The average recovery was between 80% and 125% for each pre-dilution of urine with limited exceptions. At the lower pre-dilutions of urine, recoveries less than 80% were observed for serotypes 15A (53–73%), 23A (50–77%) and 35B (33–76%). Within these three serotypes, the low recovery results improved when urines were diluted more, which suggested a potential of urine interference with the measurement of PnPs
Specificity	Based on development and qualification data, no interference of measurements between 30 serotypes from the 21-valent and the previously published 15-valent assay [29] was observed The resulting measured concentration for the samples absent the target PnPs generated a result that was below its determined lower limit of quantitation (LLOQ) for all serotypes except for serotype 23B with geometric mean concentration of 0.053 ng/ml, just above the LLOQ 0.045 ng/ml Though the recoveries for each spiked sample were borderline of 50% for serotype 15A and 23A and were relatively low (30%) for serotype 35B, potentially due to urine interference, the data provided no evidence that any one serotype either potentiates or interferes with the measurement of another serotype No interference was observed against non-assay pneumococcal polysaccharides 6D, 7C, 13, 21, 34, 35F, 38 No interference was observed against non-pneumococcal bacterial lysates of Acinetobacter baumannii, Escherichia coli K12, Klebsiella pneumoniae, Klebsiella quasipneumoniae, Pseudomonas aeruginosa, Enterococcus faecalis, Neisseria gonorrhoeae, Clostridium difficile, Bacteroides fragilis, Staphylococcus aureus SACO1, Staphylococcus aureus, RN4220

Table 4. Consolidated table for parameters from serotype-specific urine antigen detection assay 21-valent assay qualification.

Serotype	LLOQ (ng/ml)	ULOQ (ng/ml)	Total precision (%CV)	Assay ruggedness to analyst (%diff. in concentration)	Accuracy range (ng/ml PnPs conc. range with 80% to ~150% recovery^†)	On target specificity (%recovery across absent serotypes)	Off target specificity (ng/ml)	Selectivity (%recovery in neat urine)	Dilutional linearity (dilution fold-bias per tenfold dilution)
3	0.0229	3.125	9.0%	6.30%	[0.012, 3.125]	0.92	<LLOQ	86%	1.55
6A	0.0898	12.5	7.8%	3.93%	[0.024, 12.5]	1.16	<LLOQ	113%	1.60
7F	0.7597	100	9.3%	5.80%	[0.195, 100]	1.05	<LLOQ	112%	1.60
8	0.1976	25	7.7%	1.43%	[0.098, 25]	1.34	<LLOQ	132%	1.64
9N	0.0898	12.5	8.4%	3.03%	[0.049, 12.5]	0.99	<LLOQ	96%	1.52
10A	0.7376	100	8.0%	5.39%	[0.195, 100]	1.23	<LLOQ	120%	1.58
11A	0.0899	12.5	11.2%	7.20%	[0.049, 12.5]	1.12	<LLOQ	106%	1.44
12F	0.0919	12.5	8.2%	8.58%	[0.049, 12.5]	1.04	<LLOQ	99%	1.48
15A	0.1953	50	14.0%	4.81%	[0.098, 50]	0.52	<LLOQ	53%	1.58
15C	0.0873	12.5	17.8%	18.69%	[0.049, 12.5]	1.09	<LLOQ	111%	1.43
16F	0.0998	12.5	8.0%	6.05%	[0.024, 25]	1.33	<LLOQ	108%	1.52
17F	0.0455	6.25	8.2%	7.15%	[0.024, 6.25]	1.13	<LLOQ	94%	1.53
19A	0.0122	3.125	7.6%	9.03%	[0.006, 6.25]	1.07	<LLOQ	89%	1.55
20A	0.0903	12.5	10.7%	7.05%	[0.049, 12.5]	1.14	<LLOQ	108%	1.39
22F	0.0453	6.25	8.1%	6.78%	[0.012, 6.25]	1.14	<LLOQ	111%	1.56
23A	0.0488	12.5	8.0%	4.84%	[0.024, 25]	0.45	<LLOQ	50%	1.46
23B	0.0447	6.25	7.6%	5.95%	[0.012, 6.25]	1.24	0.053	105%	1.57
24F	0.0864	12.5	9.3%	5.93%	[0.049, 12.5]	0.98	<LLOQ	87%	1.52
31	0.0454	6.25	8.2%	7.43%	[0.024, 6.25]	1.20	<LLOQ	102%	1.47
33F	0.0879	12.5	8.6%	6.56%	[0.049, 12.5]	0.99	<LLOQ	87%	1.53
35B	0.0682	6.25	9.5%	4.75%	[0.024, 12.5]	0.29	<LLOQ	33%	1.22

† Upper end of the accuracy range expanded from 125% to around 150%.

CV: Coefficient of variation; LLOQ: Lower limit of quantitation; ULOQ: Upper limit of quantitation.

The recoveries of spiked polysaccharides from several samples used to assess accuracy, selectivity and specificity were lower for serotypes 15A, 23A and 35B than for the other 18 serotypes tested. Specificity assessment did not exhibit cross-reactivity across 30 pneumococcal vaccine polysaccharides in either the 15- or the 21-valent assay (Table 4), across 31 non-vaccine pneumococcal bacterial lysates in the 21-valent assay (Supplementary Table S2), or across 11 non-pneumococcal bacterial lysates (Table 4).

The dilution linearity was assessed using samples with concentrations diluted into and across the assay range. The overall dilution fold-bias estimates per tenfold dilution were between 1.22-fold and 1.64-fold across all 21 serotypes in the 21-valent assay. Thus, the assay met the expectation of a less than twofold fold-bias per tenfold dilution.

Assay LOQs are listed in Table 5 for shared serotypes between the 15- and the 21-valent assay and are applicable to the 15-valent assay which was used to measure clinical samples. Assay LOQs of the 21-valent assay were determined separately in Table 4 but not applied to clinical sample testing for share serotypes.

Table 5. Clinical positivity cut-offs and assay LOQs for 30 serotypes.

Assay	Serotype	Cutoff (ng/ml)	LLOQ (ng/ml)	ULOQ (ng/ml)
15-valent assay	1	0.5245	0.0625	1
	3^†	0.0117	0.0078	1
	4	0.1656	0.0313	4
	5	0.1020	0.0313	4
	6A^†	0.2343	0.0781	5
	6B	0.2517	0.0156	1
	7F^†	3.0459	0.3125	20
	9V	0.0620	0.0156	2
	14	0.0939	0.0313	4
	18C	0.7500	0.2500	4
	19A^†	0.0234	0.0078	1
	19F	1.5000	1.0000	20
	22F^†	0.3750	0.2500	4
	23F	3.3576	0.3125	40
	33F^†	0.1875	0.1250	4
21-valent assay	8	0.2964	0.1976	25
	9N	0.1344	0.0898	12.5
	10A	2.2128	0.7376	100
	11A	0.1350	0.0899	12.5
	12F	0.1380	0.0919	12.5
	15A	1.0965	0.1953	50
	15C	0.2616	0.0873	12.5
	16F	0.1500	0.0998	12.5
	17F	0.1582	0.0455	6.25
	20	0.1356	0.0903	12.5
	23A	0.2445	0.0488	12.5
	23B	0.0672	0.0447	6.25
	24F	0.2592	0.0864	12.5
	31	0.0547	0.0454	6.25
	35B	0.1519	0.0682	6.25

† The cut-offs and LOQs from six overlapping serotypes between the 15-valent and 21-valent assays are derived from the 15-valent assay as clinical sample data were determined with the 15-valent assay.

LOQ: Limit of quantitation; LLOQ: Lower limit of quantitation; ULOQ: Upper limit of quantitation.

3.3. A pre-specified strategy to determine the clinical positivity cutoffs

In the published 15-valent assay, a positive result in the SSUAD assay was defined as a concentration ≥twofold higher than the low limit of quantitation (LLOQ) of the assay. This cut-off was based on a set of healthy patient urines and urine form patients with confirmed pneumococcal disease as diagnosed by sputum samples [29]. However, when more clinical data became available, certain serotypes, which were known to have very low prevalence in circulation, had unexpectedly high positivity rates using the 2xLLOQ cutoffs. This indicated that the 2xLLOQ cutoffs were not suitable for these serotypes. Consequently, new cutoffs were developed for the 15-valent assay by optimizing the balance between sensitivity and specificity at the participant level. The same cutoff methodology was subsequently applied to the 21-valent assay after its qualification.

Prior to establishing the new cutoffs, the 30 serotypes included in the 15-valent or 21-valent assays were classified as either high or low prevalence. This classification was based on the average incidence of invasive pneumococcal disease (IPD) cases reported by the Centers for Disease Control and Prevention (CDC) between 2015 and 2019. Serotypes with an average IPD incidence case rate of less than 0.5 per 100,000 population were designated as low prevalence, while those with higher case rates were classified as high prevalence. Out of the 30 serotypes, 16 were thus classified as high prevalence and 14 as low prevalence.

The new cutoffs were then determined using a cutoff strategy, primarily based on a statistical method called nonparametric tolerance interval (NTI). Our approach is data-driven and utilizes only negative control samples (similar to [25,26]), enabling us to predefine it prior to accessing any pneumonia patient data. This procedure made our subsequent sensitive evaluation of the cutoffs more credible. For high-prevalence serotypes, the cutoffs were determined by the NTI method, as the upper bounds of the 99%-confidence intervals of the estimate that at least 98% of negative urine samples fell within the predicted range. Cutoffs derived in this manner have a mathematically guaranteed minimal serotype-level assay specificity of 97%. For low-prevalence serotypes, the thresholds were also set using NTI, but with parameters ensuring a mathematically guaranteed minimal serotype-level assay specificity of 98.5%. Fully utilizing public data as our prior knowledge enabled us to set the thresholds to achieve more aggressive specificity for those low-prevalence serotypes, thereby achieving a better balance between specificity and sensitivity.

Negative control urine specimens were obtained from non-hospitalized non-pneumonia adult patients with and without co-morbidities. This dataset served as the negative control samples in the previously mentioned cutoff strategy. However, due to the extremely high specificity of our assays in some of the 15-valent and 21-valent serotypes, these assays yielded few quantifiable concentration data in the negative controls. Since the NTI method requires a sufficient number of quantifiable concentration data points, reasonable initial cutoffs could not be estimated for these serotypes using only the NTI method. To address this, our cutoff strategy was further extended beyond relying only on the NTI method. The extended strategy considered using both the NTI method and the values of the low LOQs of the assays to jointly determine the cutoffs. For serotypes with high prevalence, the cutoffs were set as the larger value between (1) the NTI upper bounds with a minimum serotype-level specificity of 97% and (2) 1.5 × LLOQ, prioritizing sensitivity. For serotypes with low prevalence, the initial cutoffs were set as the larger value between (1) the NTI upper bounds with a minimum serotype-level specificity of 98.5% and (2) 3 × LLOQ, prioritizing specificity.

3.4. Refinement & evaluation of the clinical positive cutoffs

The cutoffs were set in this way prior to the availability of patient data and were called the initial cutoffs since they are designed to be latter refined by utilizing negative control urine samples from adult outpatients without pneumonia (same data previously used) and urine samples from patients with community acquired pneumonia with IPDs (positive blood cultures obtained from a sterile site). This refinement aimed to improve the specificity using negative controls from adult outpatients without pneumonia and sensitivity using IPDs at the patient level, as well as to align the positivity rates and distributions for serotypes with their known prevalence in circulation and published results. Out of the thirty serotypes, the initial cutoffs of only five were refined, while the remaining twenty-five serotypes in the 15-valent or 21-valent assays maintained the initial cutoffs obtained using our initial pre-specified cutoff strategy. The 21-valent assay demonstrated an estimated sensitivity (IPD) of 85.7% and specificity (negative controls from adult outpatients without pneumonia) of 97.2% (Figure 2). Similarly, the 15-valent assay exhibits a sensitivity of 91.7% and specificity of 98.6%. The combined 15- and 21-valent assays show a sensitivity of 85.7% and specificity of 97.2%. Additionally, the positivity rates and distribution data align with CDC distribution data and published results, which will be presented in a separate paper [38]. The new cutoffs in the 15-valent or 21-valent assays using extended NTI method will be further validated and refined to better suit real-world data.

Figure 2. Clinical specificity and sensitivity to detect serotype-specific pneumococcal infection based on the presence of pneumococcal polysaccharides in patient urines. Clinical specificity and sensitivity were assessed for each assay individually and combined as indicated.

4. Discussion

Overall, the bioanalytical assay performance was found to be suitable for the assay application to detect serotype-specific pneumococcal capsule polysaccharides in urine samples.

Although serotypes 15A, 23A and 35B exhibited lower recoveries for spiked polysaccharides in accuracy testing, one out of eight urine samples used in accuracy and selectivity, however, did exhibit recoveries well within the common 80% to 125% window indicating a potential matrix effect for the other seven samples (data not shown). This urine sample also was clearer with less of a yellow color compared with the other urine samples, potentially indicating a more dilute matrix. Thus, the low spike recoveries of 15A, 23A and 35B were mostly due to potential urine interference. Since dilution linearity was demonstrated beyond the assay range, samples exceeding the assay range could be diluted into the range as needed to obtain a quantitative result.

As clinical cutoff values were developed based on relative determinations of pneumococcus-negative samples (for specificity) to samples with invasive disease (for sensitivity) that were both tested in the same assay, the under-recoveries of serotypes 15A, 23A and 35B were not considered to impact the application of the assay in clinical samples (Table 5). Cut-off values must be viewed in the context of this assay and the standard curve, which is based on full-length, purified polysaccharides. In contrast, the make-up of urine polysaccharides such as the length of the fragments is unknown and impact the assigned polysaccharide concentration. Thus, cut-off values are not absolute or transferable to other assays.

Clinical positivity cut-offs balanced real-world data from patient samples, assay performance, with refinement based on serotype prevalence. This approach allowed reducing false positive and negative rates in a real-world environment. While a purely statistical approach might theoretically have been preferred, we found this to result in lower clinical accuracy than the combined approach.

5. Conclusion

The 21-valent SSUAD assay expanded the ability to evaluate pneumococcal serotype distribution in noninvasive pneumococcal disease. Assay qualification demonstrated fit-for-purpose performance and met expectations for precision, robustness, accuracy, dilutional linearity, selectivity and specificity.

Three serotypes, 15A, 23A and 35B exhibited matrix interference in some samples with unknown cause. While sample dilution could reduce the impact to some extent, it also would come at the cost of reduced assay sensitivity. Since clinical positivity cutoffs were designed based on the measured PnPs distribution of a negative control population from adult outpatients without pneumonia in the same assay, the impact of the matrix effect on clinical conclusions is likely limited. Selection of different antibodies could potentially result in improved assay performance [30] but will require time with unpredictable possibility of success.

Using the qualified assay, clinical positivity cutoffs were developed based on a diverse set of negative control samples to determine clinical specificity and sensitivity. The cut-offs were then confirmed with known positive samples from invasive disease. The novel approach to cutoff setting combined measured values with adjustments based on estimated serotype prevalence to maximize both specificity and sensitivity.

Specificity and sensitivity could potentially be evaluated either for each serotype or on a patient-level across all serotypes. While the serotype-distribution is important from an epidemiological perspective, only positivity/negativity for pneumococcal disease is critical from a medical perspective. We chose to evaluate clinical specificity and sensitivity on a patient level, in other words, answering the question whether a given patient has pneumococcal disease or not. Then the SSUAD could be used as a secondary assessment to identify the associated serotype. Using a representative negative control population beyond healthy patient samples was critical to develop clinical cutoffs resulting in updated cut-offs for the previously published 15-valent SSUAD. Application of the assay to different patient populations, particularly pediatric populations, will likely require further cut-off optimization.

Going forward, the multiplexed 21-valent SSUAD or both the 15- and the 21-valent assays could be used to understand noninvasive pneumococcal disease and serotype distribution in adult populations.

Author contributions

Experimental execution: A Choudhury, R Chamcha, A Akgul, N Stachura, T Follmer, W Newhard, G Jackson, J Herbert, C Nguyen, H Pham, S Nasrin, K McCabe. Patient sample collection & manuscript review: N Rouphael, SY Cruz, KD Johnson. Statistical analysis: Y Zhang, J Ma, R Li. Data analysis, scientific guidance, assay development and manuscript preparation: A Choudhury, T Verch, G Rajam.

Financial disclosure

Authors as indicated were employed by Merck & Co., Inc. The authors have no other financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Writing disclosure

No writing assistance was utilized in the production of this manuscript.

Acknowledgments

Antibody clones were developed at Merck & Co., Inc., West Point, PA, USA by the Vaccines Analytical Research and Development department. Patient samples were collected with informed consent and under IRB approval by staff at the Hope Clinic and Emory Healthcare.

Supplemental material

Supplemental data for this article can be accessed at https://doi.org/10.1080/17576180.2024.2357030

Competing interests disclosure

Authors as indicated were employed by Merck & Co., Inc. which develops and manufactures pneumococcal vaccines. The authors have no other competing interests or relevant affiliations with any organization or entity with the subject matter or materials discussed in the manuscript apart from those disclosed.