Comparative analysis of eleven SARS-CoV-2 immunoassays and neutralisation data: time to enhance standardisation and correlation of protection

Abstract

Background

To infer a reliable SARS-CoV-2 antibody protection level from a serological test, an appropriate quantitative threshold and solid equivalence across serological tests are needed. Additionally, tests should show a solid correlation with neutralising assays and with the protection observed in large population cohorts even against emerging variants.

Objectives

We studied convalescent and vaccinated populations using 11 commercial antibody assays. Results were compared to evaluate discrepancies across tests. Neutralisation capacity was measured in a subset of the samples with a lentiviral-based assay.

Methods

Serum from convalescent (n = 121) and vaccinated individuals (n = 471, 260 with Comirnaty, 110 with Spikevax, and 96 with Vaxzevria) was assessed using 11 different assays, including two from Abbott, Euroimmun, Liaison, Roche, and Vircell, and one from Siemens. A spike protein-lentiviral vector with a fluorescent reporter was used for neutralisation assay of serum from convalescent (n = 26) and vaccinated (n = 39) individuals.

Results

Positivity ranged between 81.3 and 94.3% after infection and 99.4 and 99.7% after vaccination, depending on the assay. Both cohorts showed a high level of qualitative agreement across tests (Fleiss’ kappa = 0.598 and 0.719 for convalescent and vaccinated respectively). Spikevax vaccine recipients showed the highest level of antibodies in all tests. Effectiveness of each test predicting SARS-CoV-2 neutralising capacity depended on assay type and target, with CLIA and anti-S being more effective than ELISA and anti-N assays, respectively.

Conclusions

High-throughput immunoassays are good predictors of neutralising capacity. Updated targets and better standardisation would be required to find an effective correlate of protection, especially to account for antibodies against new variants.

Keywords:

• SARS-CoV-2
• antibodies
• immunoassays
• neutralisation assay
• correlate of protection

Background

Humoral immunity against COVID-19, obtained either after natural infection or vaccination, can be measured in vitro by neutralisation assays. These assays have shown to be a good predictor of protection since the beginning of the SARS-CoV-2 pandemic [1]. However, they have important limitations: the biosafety, the high cost, they are time consuming and have high expertise requirements. All of these complications restrict their use on a large scale [2,3].

Currently, SARS-CoV-2 commercially-available immunoassays are available as a high-throughput cost-effective method to estimate immunity as an alternative to neutralising antibody assays [4]. Antibody assays mostly focus on the spike protein (S), highly immunogenic and the target of the most available vaccines [5,6], or the nucleocapsid protein (N). As non-infected vaccinated populations only generate antibodies against the spike protein, anti-N antibodies can be used as a diagnostic test for past infection [7] and can help estimate the true prevalence of SARS-CoV-2 infection in the population [8].

The goal of obtaining a correlate of protection, that is, a quantifiable immunity value sufficient to prevent infection (for example, in the form of an antibody count readout) [9–13] is to provide healthcare systems a tool to recognise future immune escapes in different population groups. Using effective correlates of protection could also reduce the need of large trials for the approval of new vaccines by using just the antibody value to estimate the level of protection (immuno-bridging) [10]. In order to obtain a reliable correlate of protection from a serological test, it is required to obtain homogeneous results across serological tests as well as a solid correlation with the serum neutralising capacity and with the protection observed in large population cohorts. Some models have shown a remarkable level of accuracy when compared with breakthrough-infection data [13], but the models require a very high amount of participating individuals, and the neutralisation data still needs to be extrapolated to the results of the various commercially available immunoassays used across the world.

There are several studies available comparing results of various immunoassays [14–22]. However, most of the comparisons between immunoassays focus on convalescent sera, so more data are necessary to understand their reliability after immunisation across different vaccines. Additionally, to understand effectiveness of protection the results must be accurately correlated with neutralisation assays. This is of capital importance especially when SARS-CoV-2 has been shifting away from the variants targeted by the first-generation vaccines and into new variants showing more antibody escape capacity such as Delta and Omicron. Importantly, it has been suggested that current immunoassays may have reduced sensitivity for antibodies elicited by Omicron infection [23–26]. Neutralisation models could help vigilance systems extrapolate immunoassay data results even across different variants.

Serological test results may also vary depending on the target protein of the assay or even the epitope targeted within the same protein [15]. Different assays also differentiate between IgG and total antibody detection, although no significant difference in agreement was found in a previous study [16]. Other variables such as age, days after infection/vaccination or severity of infection must be accounted for in order to transform these data into information usable for public health. For instance, a study found no correlations between immune markers and infection in the case of asymptomatic patients [27].

In this paper we studied convalescent and vaccinated populations and compared the results of eleven antibody assays with a neutralisation antibody assay using a spike protein-carrying lentiviral vector.

Methods

Patient selection

This study focuses on two population cohorts. Cohort 1 was a retrospective transversal cohort with 121 patients from the Complexo Hospitalario Universitario de Vigo (CHUVI), Vigo, Spain. Sera samples were taken from March to June, 2020, between 14 and 85 days after SARS-CoV-2 infection diagnosis by PCR as part of the care routine. Due to the strain on the healthcare systems during this period, only symptomatic individuals were tested and therefore, only symptomatic individuals were selected.

Cohort 2 was composed of 471 healthcare workers of CHUVI, 466 of whom received two doses of Pfizer-BioNTech Comirnaty (260), Moderna Spikevax (110) or AstraZeneca Vaxzevria (96). Five participants received a combined dose of Vaxzevria + Comirnaty. Sera samples were taken from June to October, 2021, between 13 and 245 days after the second dose (Figure 1).

Figure 1. Graphical summary. Sera samples were extracted from a cohort of convalescent individuals (post-infection) or from a cohort of vaccinated individuals, after the second dose of vaccination (post-vaccination). Immunity levels were measured with commercial antibody assays (ELISA and CLIA) as well as a neutralising antibody assay with pseudovirus particles. Figure includes additions created with biorender.com.

The study protocols POST_inf (Cohort 1) and POST-VAC (Cohort 2) were approved by the Galician network of committees of research ethics (2020/391 and 2021/219, respectively). For cohort 1, our study received approval from the ethics committee with the exception of informed consent. The serum samples used were obtained as part of the standard diagnostic procedure for measuring immunity to SARS-CoV-2, conducted by the responsible medical professional. No determinations other than the original were performed. For cohort 2, informed consent was obtained from all individuals.

Inclusion criteria

For cohort 1, individuals with at least one SARS-CoV-2 detection by RT-PCR in respiratory samples and one quantification of SARS-CoV-2 antibodies in serum were selected. Sera samples were required to have enough volume to perform the tests and for the sera archive.

For cohort 2, only CHUVI workers vaccinated with the vaccines included in the study were included after signing an informed consent form.

Exclusion criteria

In cohort 2, patients with a previous infection confirmed by a positive anti-nucleocapsid antibody test were excluded from further analysis.

Antibody tests used

Eleven antibody commercial tests were conducted in this study following manufacturer’s instructions:

Abbott SARS-CoV-2 IgG II Quant (Abbott diagnostics, Chicago, IL, USA) (#1), Abbott SARS-CoV-2 IgG AntiN (Abbott diagnostics, Chicago, IL, USA) (#2), Atellica IM SARS-CoV-2 Total (Siemens Healthineers, Munich, Germany) (#3), Anti-SARS-CoV-2 ELISA IgG (EuroImmun, Lübeck, Germany) (#4), Anti-SARS-CoV-2 QuantiVac ELISA IgG (EuroImmun, Lübeck, Germany) (#5), Elecsys Anti-SARS-CoV-2 S (Roche Diagnostics, Basel, Switzerland) (#6), Elecsys Anti-SARS-CoV-2 N (Roche Diagnostics, Basel, Switzerland) (#7), LIAISON SARS-CoV-2 S1/S2 IgG (DiaSorin, Saluggia, Italy) (#8), LIAISON SARS-CoV-2 TrimericS IgG assay (DiaSorin, Saluggia, Italy) (#9), COVID-19 VIRCLIA IgG Monotest (Vircell Microbiologists, Granada, Spain) (#10), COVID-19 Spike Quantitative VIRCLIA IgG Monotest (Vircell Microbiologists, Granada, Spain) (#11).

Cohort 1 samples were analysed using immunoassays from #2 to #10. Cohort 2 samples were tested with immunoassays #1, #3, tests from #6 to #9 and assay #11. The targets of the techniques are shown in Table 1. Results were presented in BAU/ml (Binding Antibody Units), adjusted to the WHO Standard Unitage, if possible.

Table 1. List of assays used in this study.

#	Assay	Target		Cohort 1	Cohort 2	BAU factor
1	Abbott IgG (Abbot)	IgG	Spike (RBD)		X	0.142
2	Abbott Nucleocapsid (AbbottNP)	IgG	Nucleocapsid	X		NA
3	Atellica Total (Atellica)	IgG + IgM	Spike (RBD)	X	X	6.26
4	Euroimmun ELISA (Euroimmun)	IgG	Spike (S1)	X		NA
5	Euroimmun QuantiVac ELISA (QuantiVac)	IgG	Spike (S1)	X		NA
6	Roche Elecsys S (Roche)	IgG + IgA + IgM	Spike (RDB)	X	X	1.03
7	Roche Elecsys N (RocheNP)	IgG + IgA + IgM	Nucleocapsid	X	X	NA
8	LIAISON S1/S2 IgG (S1S2)	IgG	Spike (S1 & S2)	X	X	NA
9	LIAISON TrimericS IgG (TrimericS)	IgG	Trimeric spike	X	X	2.6
10	Virclia IgG Monotest (Virclia)	IgG	Spike (RBD)	X		NA
11	Quantitative Virclia Monotest (VircliaQ)	IgG	Spike (RBD)		X	1

In brackets, short identifier name used across the text. RBD: receptor binding domain. For all cases tests in which it is available, conversion factor to BAU (binding antibody units), is shown.

ELISA tests Euroimmun and QuantiVac were not used for the second cohort due to their lower performance in the cohort one. Newly available quantitative tests Abbott and Virclia Quantitative (Virclia Q) were added to the comparison for cohort 2. For Cohort 1, Atellica Total was used exclusively to obtain qualitative results because 85 out of 121 samples produced results that exceeded the maximum linear factor of 10 provided for this test at the time of analysis. We were concerned about diluting a large number of samples and not having sufficient sample volume for the rest of the study. However, by the time of Cohort 2, Atellica Total had already increased its maximum linear factor to 75, and sample volume wasn’t an issue.

Neutralising antibodies assay

A total of 26 samples of Cohort 1 and 39 of Cohort 2 (13 per vaccine), were tested with a neutralising assay. Neutralisation assays were performed based on the protocol by Nie et al. [28] using commercially available lentiviral vectors transfected with Green-Fluorescent Protein gene and presenting SARS-CoV-2 spike protein variant B.1 (VectorBuilder, Neu-Isenburg, Germany). Sera from convalescent or vaccinated individuals was heat-inactivated (30 min 56°) and serially diluted in duplicates five times. The lentiviral vectors were added to the sera in a volume to achieve a MOI of 50. A positive control (no serum, viral control) and a blank (no virus, cellular control) were added on each plate in triplicate.

Incubation in 96 TC-treated well-plates was performed for 1 h at 37 °C. Then, HEK293T-ACE2 cells (25,000 cells/well) were added and incubated for 72 h at 37 °C, 5% CO₂ in a FLUOstar Omega Fluorescence plate reader (BMG LABTECH, Offenburg, Germany). After 72 h, fluorescence was measured and half-maximal inhibitory dilution (ID50) was calculated (Reed-Muench method).

Statistical analysis

All statistical data analysis was done using R (version 4.1.1, https://cran.r-project.org/). Data visualisation was performed with the R program ggplot2 [29]. Shapiro–Wilk normality test was performed to check for normality and Levene test to assess for equal variances. T-test, unpaired Wilcoxon rank-sum test, Welch’s ANOVA and Games-Howell test were used when indicated. When multiple pairwise tests were performed simultaneously, p-value was adjusted using the Bonferroni correction. Result agreement between techniques was calculated using Fleiss’ kappa. Spearman’s rank correlation coefficient (ρ) was calculated for all tests in pairs for both cohorts.

Results

A majority of patients had a detectable level of antibodies after infection by all tests

A total of 121 patients from the convalescent cohort were checked for both anti-spike (7 tests) and anti-nucleocapsid (2 tests) antibodies. Test results for all samples were considered positive or negative, excluding some tests that allow a possible uncertain interpretation (borderline/undetermined/grayzone) (from here on uncertain).

All samples were therefore classified as positive or negative according to the result obtained in a majority of tests. All samples with at least a single uncertain result fell into the ‘majority positive’ group. An equal number of positive and negative results was not found in any of the samples studied.

Percentage of positive results varied between 94% (S1S2) and 81% (Euroimmun), although this variability might be partially explained by the uncertain result given by some of the kits (Figure 2(A)). Seven samples were considered negative (antibodies not detected by a majority of tests). None of the samples with a negative result belonged to immunosuppressed individuals. Twenty samples had a conflicting result in at least one test (17%). An extra seven samples showed an uncertain result in at least one test (6%). Therefore, 94 samples got the same qualitative result in all assays (78%), showing a moderate inter-agreement level (Fleiss’ kappa = 0.598). The test with the most discrepant samples was QuantiVac (7.5%) while S1S2 and Abbott Nucleocapsid assay had no discrepancies at all (Table 2).

Figure 2. Technique result agreement in sera of convalescent individuals. (A) The ratio of individuals with positive detection of antibodies after infection ranged from 94% (Virclia) to 81% (Euroimmun), with 78% of the samples giving the same qualitative result in all tests. (B) Age was not related with antibody detection (68.4 ± 18.9 vs. 81.2 ± 13.6; p = 0.051, Wilcox rank-sum test) (C) Day of sera sampling had a non-significant impact in the seroconversion results (47.2 ± 16.5 vs. 46.2 ± 5.6; p = 0.71, t-test). In (B and C), in case of discrepancies, negative and positive were defined as the result given by a majority of tests.

Table 2. Samples with discrepant results across tests.

Technique	Uncertain result	Discrepant (negative)	Discrepant (positive)	Total discrepant
AbbottNP	5	0	0	0 (0%)
Atellica Total	0	1	0	1 (0.8%)
EuroImmun	7	8	1	9 (7.3%)
QuantiVac	1	5	1	6 (7.5%)
Roche	1	0	1	1 (0.8%)
RocheNP	0	1	0	1 (0.8%)
S1S2	0	0	0	0 (0%)
TrimericS	0	7	0	7 (5.7%)
Virclia	0	0	1	1 (0.8%)
Total	14	22	4	26

Individual result compared to the result given by the majority of tests. Percentages may not correlate with the absolute number of tests in all cases due to not all samples being measured by all tests.

Samples were divided in positive and negative according to the result given by a majority of tests. Average age for seropositive individuals was 68.4 ± 18.9 years and 81.2 ± 13.6 years for seronegative individuals. The difference found was narrowly non-significant (p = 0.051, Wilcoxon rank-sum test), although there was a small number of negative results (n = 7) (Figure 2(B)). The days between diagnosis and extraction of serum were not associated with the seroconversion (47.2 ± 16.5 vs. 46.2 ± 5.6; p = 0.71) (Figure 2(C)). When analysing these results by tests individually, differences in the results in relation to age and extraction day were also non-significant (Supplementary Figures 1 and 2).

Most vaccinated participants showed seroconversion of anti-spike by all tests

A total of 471 sera samples from vaccinated individuals were measured on seven different tests, six with anti-S antibodies as target and one with anti-N antibodies as target (RocheNP). Twelve samples were discarded from quantitative analysis as they showed detectable levels of anti-N antibodies, suggesting previous infection (Supplementary Figure 3).

A total of 260 sera samples were taken from Comirnaty (BioNTech/Pfizer) vaccine recipients, 110 from Spikevax (Moderna) and 96 from Vaxzevria (AstraZeneca). Five extra vaccine recipients received a combined dose of Vaxzevria and Comirnaty.

The results showed substantial agreement for all techniques on detection of anti-spike IgG (Fleiss’ kappa = 0.719). From all samples, 468 showed a positive result, one showed a negative result, and two showed discrepancies throughout tests: one sample with one negative test, and a second one with three positive and three negative tests. Notably, the individual with a negative result was under immunosuppressive treatment due to a solid organ transfer. Overall, this showed high consistency throughout tests, as well as high positivity rate on antibodies from vaccine recipients.

Spikevax vaccine recipients showed the highest antibody value detected in all antibody tests

Values of antibodies were converted to BAU/ml according to the international standard for all tests in which a conversion rate is provided by the corresponding manufacturer.

Spikevax showed the highest average antibody count, albeit with differences across assays. In all the commercial kits used, the convalescent cohort sera showed the lowest average antibody count followed by Vaxzevria (Figure 3). Significant differences were found for all tests and cohorts except between Comirnaty and Spikevax (3 tests non-significant) and between Comirnaty and Vaxzevria (1 test non-significant) (Supplementary File 1).

Figure 3. Boxplot of antibody values all for the tests used targeting anti-spike antibodies in the vaccine cohort. The kits showed discrepancies in the antibody levels depending on the vaccine and test used. S1S2 results were not added to the graph as the S1S2 test does not provide a standardisation value to BAU/ml.

All techniques showed direct correlation between their results

In order to explore the correlation between the results of various techniques, Spearman’s rank correlation coefficient (ρ) was calculated for all tests in pairs for both cohorts. Correlation between results was found across all tests. For the cohort of infected subjects, Euroimmun and Roche Nucleocapsid showed the lowest correlation (ρ = 0.28; p = 0.002). The highest correlation between techniques was observed between S1S2 and TrimericS (ρ = 0.91; p < 10⁻⁴⁶) (Figure 4(A)). In the case of the vaccinated Cohort, the highest correlation was observed between Atellica and Roche (ρ = 0.91; p < 10⁻¹⁶³), and the lowest between S1S2 and VircliaQ (ρ = 0.71; p < 10⁻⁷⁰) (Figure 4(B)).

Figure 4. Correlation matrix across techniques expressed as spearman’s correlation coefficient (ρ). (A) Pairwise correlation coefficients for the convalescent cohort. (B) Pairwise correlation coefficients for the vaccine cohort.

Immunoassays show more agreement in the range covered by the WHO standard

The optimal working range of the WHO standard is 0–1000 BAU/ml, as the standard serum is calibrated to 1000 BAU/ml [30]. In cohort 1, values above 1000 BAU/ml were uncommon (n = 12/121; 9.9%). However, in cohort 2, 200/459 samples fell outside the working range (43.6%). For all pairwise correlations but one, agreement was higher in the working range of the WHO standard (<1000 BAU/ml). The only exception was Atellica and VircliaQ, where Spearman’s correlation was the same in both groups (ρ = 0.60) (Supplementary Figure 4).

In vitro neutralising antibody assays suggested differences in the sera antibody results for convalescent and vaccinated samples

A neutralising antibody assay was performed for samples in both cohorts. Serum samples were cultured with a lentiviral vector presenting SARS-CoV-2 spike protein and used to infect a cell line of HEK293t-ACE2 cells. The lentiviral vectors were transfected with the GFP gene as a reporter to track infection through time (Figure 5(A)).

Figure 5. Representative data from the results of the neutralising antibodies assay. (A) GFP fluorescence was measured for 72h, with a cell control with no virus (blank) and a viral control with no antibodies (0% inhibition). (B) Boxplot of the 50% inhibitory dilution for all cohorts. All pairwise comparisons were non-significant (t-test). (C–M) Scatterplot of all test results (antibodies in serum) compared to the ID50 values from the neutralising antibody assay. The grey area shows 95% confidence of the linear approximation.

Statistical comparisons of all groups showed non-significant differences across vaccines and convalescent samples (p = 0.773, Welch’s ANOVA), albeit data showed a much higher variability in the neutralisation results. 50% inhibitory dilution (ID50) median ± IQR was 207 ± 288.5; 152 ± 213.5; 175 ± 229.5 and 250.5 ± 2148.5 for Comirnaty, Spikevax, Vaxzevria and Convalescent, respectively (Figure 5(B)).

Regarding the accuracy of the different assays, Roche, TrimericS and S1S2 all showed similar correlation with the neutralisation results (ρ = 0.81; p < 10⁻⁶) in the convalescent cohort. This was the best correlation amongst all tests. Roche Nucleocapsid assay presented the worst accuracy, which showed no significant correlation (ρ = 0.11; p = 0.60), albeit this test is used to measure anti-N antibodies. Amongst the tests measuring anti-S antibodies, QuantiVac had the worst correlation (ρ = 0.39; p = 0.017). For the vaccine cohort, the technique that correlated best with the neutralising antibody assay was Abbott (ρ = 0.79; p < 10⁻⁸). The worst correlation was in S1S2 (ρ = 0.46; p = 0.003) (Figure 5(C–M)).

Discussion

Patients included in this study had a high rate of seropositivity, over 80% after infection and over 99% after vaccination. Our study showed, from a qualitative sense, a high level of agreement across techniques in the vaccine cohort, over 99%. In the case of the convalescent cohort, variability was higher, with only 78% of the samples giving a full agreement amongst all tests, which is a similar level of agreement observed by another publication using a similar number of immunoassays [15].

Quantitatively, agreement between tests was affected by the target of each kit. In the convalescent cohort, as expected, the level of correlation between different anti-spike tests was higher than between anti-spike and anti-nucleocapsid tests. Differences between results in anti-nucleocapsid and anti-spike tests could be due to actual technical limitations or to real biological differences in the humoral response [31,32]. A similar study by Leuzinger et al. showed that when compared to other anti-spike tests such as TrimericS, Roche-S has a higher correlation agreement than Roche-NP, in agreement with our findings [15]. In the vaccine cohort, anti-nucleocapsid antibodies could be used to detect past infection [7,33].

Comparison of antibody values between assays is challenging, even when adjusted to the WHO standard serum, due to different methodologies. A recent publication found that, even when comparing different immunoassays to the same neutralising antibody test, BAU cut-off values for positivity were different (48–559 BAU/ml), showing the complexity of standardisation [34]. Additionally, WHO standard is calibrated to 1000 BAU/ml [30], and therefore immunoassay reads significantly above this value fall outside the WHO standardisation curve. A previous publication showed that alignment between tests disperses in values above 1000 BAU/ml [35]. In our data, a BAU mean value across tests above 1000 BAU/ml is uncommon in the convalescent cohort. This is consistent with data from Kroidl et al. [36]. In our vaccine cohort however, a high number of samples had a mean value above 1000 BAU/ml (n = 200/459; 43.6%). For all tests in this cohort, correlation between assays was higher in the group of samples <1000 BAU/ml compared to the >1000 group, with the exception of Atellica and Virclia, where Spearman’s correlation was the same in both groups (ρ = 0.60). The lower agreement of the tests in high-value sera samples suggests caution should be taken when measuring high-value antibody samples, useful data in order to select sera for passive immunisation or treatment of hospitalised patients with high titre serum.

While in-house normalisation using a standard reference has been suggested to improve cross-laboratory normalisation [37], variations in study targets and objectives still complicate comparisons. Some publications focus on specific populations, such as those with one vaccine dose [18,38], convalescent patients [16,17], or long-term antibody dynamics in a small cohort [22]. Sampling dates also exhibit significant variability in publications. For example, one study observed peak antibody levels in Comirnaty vaccine recipients at eight days post second dose [14]. However, this is difficult to extrapolate to other vaccines, as eight days post second dose time corresponds to roughly four weeks post dose one for Comirnaty, five weeks for Spikevax, and 9–13 weeks for Vaxzevria [39,40]. In our study, one limitation is the variability in the time after vaccination between vaccine groups. In the convalescent cohort, time since positive PCR was not different between individuals who seroconverted and those who did not (Figure 2(C)), suggesting minimal impact on our results due to the post-diagnosis sampling time. In the vaccine cohort, the mean time between the second dose and serum extraction was 141, 180, and 105 days for Comirnaty, Spikevax, and Vaxzevria, respectively. While this hinders direct comparisons between vaccine groups, it is noteworthy that Spikevax exhibited the highest binding antibody units among the three groups. While sampling time was variable in our study, concordance of antibody levels between techniques, which is not affected by the date of serum collection, supports the reliability of the test measurements.

Previous studies have highlighted the role of assay targets in levels of agreement between techniques [15,35]. Our data encompasses anti-nucleocapsid and anti-spike antibody testing. Among the anti-spike tests, there are tests available against the Receptor-Binding Domain (RBD) (tests #1, #3, #6, #10, #11), S1 domain (tests #4, #5), S1 and S2 domains (test #8), or the native spike trimer (test #9). Differences in measurements may be attributed to different antibody responses. Vaccine antibodies predominantly target the RBD, while convalescent antibodies show a broader response [41–43]. Our data showed that TrimericS correlated more strongly with other anti-spike tests in the convalescent group than in the vaccine group, possibly due to differences in the antibodies generated. As the vaccines used in this study elicit antibodies against the RBD, RBD-focused kits could better assess vaccine response, as supported by previous research [14].

Neutralising antibodies are crucial predictors of protection [1] and their correlation with immunoassay results evaluates immunoassay efficacy. Our neutralising antibody data showed some generalities regarding the protection predictive value of immunoassays. First, greater correlation was found when immunoassays exclusively target anti-RBD antibodies in the vaccine cohort. Second, CLIA assays outperform ELISA assays in predicting neutralising capacity in the convalescent cohort, which is in line with the greater diagnostic accuracy found for CLIA assays [44]. Finally, as expected, anti-nucleocapsid antibody quantification tests were less reliable in predicting neutralising capacity compared to anti-spike antibody tests. This could be explained, at least partially, by our lentiviral pseudovirus carrying only the spike protein from SARS-CoV-2. Comparing the measurements of anti-NP antibodies with our neutralisation assay can provide insights of the ability of anti-NP measurements to predict an anti-spike response. Our results highlight that anti-nucleocapsid antibodies might not be appropriate to predict anti-spike responses. It is important not to overstate the conclusions of the neutralisation results, as sample size was smaller than in the in vitro assays due to limitations in costs of the reagents used. Nonetheless, these results are in line with other previously published data. A similar conclusion was obtained by Takei et al. using a live virus neutralisation assay, who found better correlation with neutralisation using an anti-Spike immunoassay than an anti-Nucleocapsid immunoassay [45]. Overall, correlation between neutralisation and immunoassays varied across tests. In the convalescent cohort for example, Spearman’s correlation coefficient ranked from over 0.8 in Roche, S1S2 and TrimericS to below 0.5 in QuantiVac or Roche Nucleocapsid, showing differences in the prediction ability of each test. A recent publication by Fajfr et al. compared eight immunoassays against a Plaque Reduction Neutralisation Test (PNRT) in a convalescent cohort, also showing variability across tests [46]. The study also found higher correlation in spike assays than in nucleocapsid assays, as well as a higher predictive value in AbbotNP than in Roche NP (0.73 vs. 0.16 Fajfr et al.; 0.69 vs. 0.10 this study). In spike assays, similar correlation values were found for TrimericS (0.88; 0.81) and Euroimmune (0.70; 0.59). The main discrepancy was found in the Roche assay (0.28; 0.82). Overall, the data suggests variability in predictive capacity between different tests, which should be taken into account in future large-cohort immunity studies.

It is worth noting that we observed different correlations between immunoassays and neutralising antibody responses in the two cohorts. Specifically, the linear regression model’s slope differs between the cohorts (Figure 5(H,J,K)), with a higher slope for convalescent sera. This difference in predicting the neutralising capacity by automatic assays may be attributed to the distinct antibody fractions generated by both cohorts. Convalescent antibody levels might be underestimated due to greater epitope variability, as mentioned previously [41–43]. This difficulty in comparing antibody values across cohorts using automated assays poses a significant challenge, particularly as many publications normalise vaccine neutralisation efficacy values as fold-of-convalescent [9,11].

Immunity to SARS-CoV-2 faces new challenges with the global emergence of Omicron subvariants. These subvariants significantly reduce the neutralising capacity in vaccinated individuals (8–127-fold reduction compared to ancestral variants in various studies) [47]. Recent research indicates reduced effectiveness of commercial immunoassays in detecting Omicron-induced antibodies [23,25,26]. On a recent study, anti-nucleocapsid assays were found to be more sensitive than anti-spike assays for Omicron-induced antibodies, likely due to Omicron’s high mutation rate, especially in the RBD [25], making them a valuable option in convalescent cohorts, at least until new reagents become available.

It has also been described that in vaccinated patients, after infection or vaccination with Omicron, the memory B cell response is directed towards shared or cross-reactive epitopes of previous variants that possess variable neutralising activity against Omicron and could show significant differences in the neutralisation capacity among Omicron sublineages [48–50]. Properly characterising B cell response against different variants after immunity boosting is one of the main challenges in order to estimate population protection.

Our patient recruitment period occurred during the peak years of the COVID pandemic social restrictions. This caused limitations in the inclusion of samples as the access to sera from infected individuals was reduced. However, having a cohort of convalescent individuals pre-vaccine rollout allowed us to measure humoral immune response caused exclusively by infection, without the added benefit for the immune system of vaccine boosting. Patient recruitment is also limited by the amount of new cases in the area and the restrictions included in the informed consent. In the case of the vaccination cohort, vaccine rollout in Spain for healthcare workers limited our recruitment, with most samples coming from Comirnaty-vaccine recipients as a majority of the population in the period received this vaccine.

Due to our time of sampling, antibodies from both cohorts were generated against the original variants pre-Variants of Concern (VOCs). Additionally, the neutralisation assay was performed against the same spike proteins, therefore not having neutralisation data against currently circulating spike variants, which would give better insight on the correlation between immunoassay result and protection to Omicron subvariants. Nonetheless, we believe that these data on antibody measurement performance is still relevant as differences between post-infection and post-vaccination are still a reality in the current situation in the pandemic. More importantly, the data shown indicates how there is a need to a) update immunological tests with epitopes detecting new variants and b) take the in-vitro assay results cautiously as there are differences between methodology and origin of antibodies (vaccine or natural immunisation). The results of serological tests have clinical relevance as it has been used in clinical settings, for instance, on its use to determine high-titre convalescent plasma for its use as treatment [51]. Our data, alongside with data obtained in similar studies, points out to the need of caution in interpreting serological test data, as variations between vaccines and convalescent sera must be taken into account. Not only this but accuracy of antibody values is an issue that healthcare systems will encounter in future pandemics, highlighting the need for more data in the performance of these tests in predicting the true protection level of a community.

As a final remark, both immunoassays and neutralising antibody assays have their own advantages and limitations. The ELISA and CLIA assays are automatable and high-throughput, but they detect all binding target antibodies regardless of efficacy. Also, these assays need to be adapted to detect epitopes from the currently circulating variants to better predict protection. The neutralising antibodies assay is regarded as accurately correlating with protection efficacy, but it is a complex and expensive technique to operate, especially for a high number of samples. Comparison between studies is an additional difficulty due to differences in methodology and therefore, more data is required to accurately use automated immunoassays to measure protection, which is considerably tough in an environment with new emerging SARS-CoV-2 variants with a high rate of spike mutations. Finally, the evidence generated from the experiences during the COVID-19 pandemic is crucial for enhancing our preparedness for future pandemics. These lessons will help develop more robust and equitable response systems, better equipping us to face any new pathogen that may emerge.

Acknowledgements

We would like to thank all companies of the immunoassays used for kindly providing reagents for the assays. We would also like to thank the staff from the Complexo Hospitalario Universitario de Vigo (CHUVI), for their work and dedication, as well as for donating samples for the study. Finally, we would like to thank the Galicia Sur Research Health Institute for their contribution to this research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Axencia Galega para a Xestión do Coñecemento en Saúde under the Grant Traslaciona COVID-19 program [Grant (CT850A-7)]; The Galician Innovation Agency under [Grant IN607B-2022/19]; and the Consellería de Sanidade, Galicia, Spain.