Will droplet digital PCR become the test of choice for detecting and quantifying ocular Chlamydia trachomatis infection? Maybe

Response to: Schachter J. Will droplet digital PCR become the test of choice for detecting and quantifying ocular Chlamydia trachomatis infection? Maybe not. Expert Rev. Mol. Diagn. 13(8), 789–792 (2013).

Julius Schachter offers a provocative title in his expert review of the diagnostic Chlamydia trachomatis droplet digital PCR (CT ddPCR) assay developed by Roberts et al. [1,2]. He put forward four main objections to the utility of this test in the face of other next-generation commercial nucleic acid amplification test platforms. These objections may be summarized as follows:

• The CT ddPCR assay is insufficiently sensitive for use in mass drug administration (MDA) treatment programs for trachoma control;

• The CT ddPCR assay was evaluated against an inappropriate commercial reference test (Roche Amplicor CT/NG test);

• No gold standard was used to evaluate the CT ddPCR assay;

• The original report failed to document the processes and procedures that are required to perform a test on a droplet digital PCR platform.

We would like to respond to the comments as discussed below.

Sensitivity

The sensitivity required of a test for infection for use by trachoma control programs is currently unknown. At a recent (September 2013) meeting at the London School of Hygiene and Tropical Medicine, leading trachoma experts were asked by the Bill and Melinda Gates Foundation to advise on the use of tests for ocular infection in trachoma control programs. It was agreed that a highly specific (>98%) test was required and that the main utility would be informing the decision about when to discontinue treating districts and communities with azithromycin (a threshold of 1% positivity in the target population of 1–9 years old was recommended). What was unclear, due to lack of empirical data, was the level of sensitivity required, but some experts suggested a minimum sensitivity of 80%. In our evaluation, we showed how the relaxation of the classification confidence threshold ‘Zeta’ would have raised our estimated sensitivity above this value. However, we suggest for the reasons outlined below that the sensitivity estimates presented in our evaluation are likely misleading. The issues behind calculating sensitivity in ddPCR and the reasons why this appeared ‘low’ given the technical superiority of digital PCR have been discussed [1,3]. The low sensitivity that we reported is most likely explained simply by sampling error, which is a problem that is unavoidable at low analyte concentrations in any test system [4], as reported by Schachter et al. in 2006 [5].

In the CT ddPCR assay, we used the Poisson formula to estimate sample target concentration. The Poisson formula estimates that with a single approximately 5 μl aliquot, the lowest concentration of targets that can be detected with negligible (<1%) risk of a failing to sample any target in a given aliquot is 1 copy per μl. We showed that ddPCR⁺ Roche Amplicor CT/NG test⁻ samples had low concentration estimates (<1 copy per μl, ≤5 copies per aliquot). The discrepancies that we observed, therefore, occurred in a mathematically predictable manner. The Roche Amplicor CT/NG test is not a quantitative test, so we could not fully investigate whether ddPCR⁻ Roche Amplicor CT/NG test⁺ results also occurred only in low load samples. However, we have tested further aliquots of the same specimens with a conventional quantitative multiplex PCR (qPCR) assay that was based on the same primer and probe sequences as ddPCR. We found that both types of discrepant result (ddPCR⁺qPCR⁻ and ddPCR⁻qPCR⁺) occurred only when the estimated concentration of the target was ≤5 copies per aliquot [Roberts et al., Unpublished Data].

The report of specificity and sensitivity in our study was presented in a standard format and we presented our data at ‘face value’ without the use of discrepant analysis. The results obtained using the multiplex qPCR can be used as a test of discrepant arbitration, which results in estimates of CT ddPCR sensitivity >98%, but this common practice has serious limitations [6]. Our evaluation could also have ensured a higher estimate of CT ddPCR sensitivity simply by increasing the template volume, while maintaining a smaller aliquot in the reference test. However, we used equivalent amounts of DNA template in each of the CT ddPCR and Roche Amplicor CT/NG tests, unlike the recent evaluation of the Cepheid Xpert CT/NG Assay, performed on the GeneXpert^® and Abbott RealTime CT/NG assay on the m2000 platform [7], which used an 8:1 ratio of target input amounts in the newer tests compared with the reference (Aptima Combo 2 assay). We do not dispute the high sensitivity of the Cepheid Xpert CT/NG and Abbott RealTime CT/NG assays, but suggest that the head-to-head comparisons reflected a comparison of two differing operating procedures as well as the absolute technical performance of the assays. Cost, simplified operating procedures and selection of classifier cut-off values that are compatible with programmatic goals are probably more important criteria on which to base the selection of a test for use in trachoma control programs. Some technical improvement can be achieved for these DNA-based tests by targeting multiple target sequences, which can enhance sensitivity at low analyte concentrations [8] and this needs to be explored in the ddPCR system. Indeed, the Abbott RealTime CT/NG assay targets two CT sequences, which is likely to increase its sensitivity compared with single target assays; however, the potential enhancement should only be measured by an evaluation that compares equivalent aliquot sizes.

Choice of reference test

The reviewer correctly points out that a more accurate estimation of sensitivity could have been obtained by using a reference test that targeted RNA, such as the Gen-Probe Aptima Combo 2 (CT/GC) assay. Yet the comparison with the Roche Amplicor CT/NG test is important because it has historically been the most widely used method for detecting ocular CT infections in trachoma research programs and it is therefore desirable to directly compare CT ddPCR with the Roche Amplicor CT/NG test. We are, however, in the process of addressing these concerns by undertaking an assessment of the performance of a modified version of the CT ddPCR against a commercial RNA-based test.

Standards

Both the CT ddPCR assay and the multiplex PCR CT assays that we developed were used in the February 2012 external quality assessment program CTDNA13A organized by quality control for molecular diagnostics [9]. Both assays reported 100% of core samples correctly. We failed to detect CT or obtained presumptively positive results with low confidence (ζ <0.95) in two samples that were included for ‘educational’ purposes, each of which was estimated to contain ≤0.07 copies/μl. This analyte concentration is outside our reported dynamic range for a single test and in these cases we would have required larger aliquot sizes or a number of technical replicates to overcome the effects of sample dispersal and sampling heterogeneity. Adherence to the quality control for molecular diagnostics protocols required that samples were processed as part of the laboratory’s normal sample handling workflows, therefore, no modifications were made. Of the total datasets reported by all participants in this external quality assessment program, 91.2% reported correct results for all core samples indicating that some 8% of the 244 laboratories (273 data sets) taking part in the evaluation failed to correctly identify core proficiency samples. Of these, 2% of laboratories using commercial real-time PCR tests and platforms failed to identify one or more of the core samples correctly, which was equivalent to those laboratories using in-house real-time PCR detection methods. We could conclude, therefore, that the use of commercial reagents and test systems per se does not always guarantee improved performance over in-house developed tests.

Operating procedures

The reviewer questioned the costs and workflow of the ddPCR platform and suggested that the Abbott RealTime CT/NG assay on the m2000 platform or the Cepheid Xpert CT/NG Assay, performed on the GeneXpert would be better platforms suited for trachoma control programs. There are in fact many commercial test platforms and options available for CT detection, just as there are a number of well-evaluated in-house nucleic acid amplification tests. Some of these tests have been evaluated for use in trachoma monitoring and we would expect that many of these options would be fit for purpose. The hardware costs of the commercial platforms and per sample costs are generally beyond the constrained budgets of most national eye health programs in resource-poor settings where trachoma remains endemic. For example, in the six districts of the Kilimanjaro region of Tanzania (population ∼200,000 per district), only three allocated any annual funds for any care services with the maximum level of annual support being Tsh 2,000,000 (∼US$1750) in a single district [10]. No money was directly allocated for trachoma control. Attempts to negotiate and install ‘gratis’ platforms from the major market leaders in countries with high trachoma burden have stalled and are yet to translate into working service centers for trachoma control. This is despite the success of other programs with access to hardware and diagnostic technical expertise in African laboratories. For example, the AMPATH laboratories at Moi University, Eldoret, Western Kenya, have access to the Abbott m2000 platform and meet ISO 15189 and DAIDS standards [Van der Pol B, Pers. Comm.]. African centers such as the one at Moi University could in theory be utilized for CT diagnostics. The GeneXpert platform is also widely deployed within tuberculosis research and control programs in multiple centers and represents a feasible alternative [11,12]. But until the un-met need is resolved, further options should continue to be explored. Indeed, as Schachter acknowledged, innovation in the research laboratory drives discovery and can translate into the clinic, ultimately resulting in changes to health systems policy. The open publication of methods, targets, validation and evaluation data is also of benefit to the scientific, commercial and national health communities. Current options in the ddPCR market have been reviewed by others including editorial review by Nature Methods. The recent publication of guidelines for the reporting of digital PCR experiments indicates that a previously niche technology is now rapidly entering the mainstream [13–15].

Regarding the operating procedures for the CT ddPCR assay, specimens can be stored on dry swabs prior to DNA extraction. The cost per test including DNA extraction is approximately £5. The current UK list price of the Bio-Rad QX200 system is £68,000. Ninety-six samples can be processed in around 3 h and a single operator can comfortably perform up to 4 runs of 96 samples per day. CT ddPCR is a multistep process that requires standard liquid handling and transfers. The basic steps are DNA extraction, PCR formulation, droplet generation, thermal cycling and droplet reading. The workflow is similar and less technically challenging than Roche Amplicor CT/NG test or a simple ELISA. Data analysis is simple and can be automated. Very little training is required to become proficient. Assays require little further optimization when transferring from existing qPCR platforms.

Concluding remarks

As Schachter pointed out, the greatest advantage of the CT ddPCR assay is the quantitative capability [2], but secondary to this is the flexibility in the choice of method used for the DNA template preparation. This flexibility means that other tests can be performed on the same specimen, which is crucial for research programs. Commercial tests frequently lack this flexibility, often the entirety of the specimen is dedicated to the diagnostic assay and material can only be interrogated post hoc by preparing template from residual material. The DNA obtained from the ocular specimens that we evaluated in our initial report has also been used to perform a multiplex CT-qPCR assay, to confirm CT genome sequencing results and to investigate the influence of CT plasmid copy number variations on ocular disease severity [16]. ddPCR offers a novel approach to the investigation of how features of infection besides bacterial load contribute to transmission, disease severity and pathogen population genetics. The ability of the ddPCR technique to identify rare variants in samples containing a dominant or a majority sequence type will allow precise investigation of the frequency of mixed genovar CT infections. The application of ddPCR to the determination of haplotype frequencies can be used to assess recombination. Such applications of ddPCR are currently dramatically altering the detection and monitoring of residual disease in cancer and detection of breakthrough viral escape mutants following treatment [17–22].

The trachoma community is moving to embrace tests for infection as a guide to making decisions about delivery of MDA. The load of bacterial carriage is also important and there is additional value in the determination of the community ocular load [23]. Measurement of the community ocular load could be used to monitor the effectiveness of sustained annual treatment and further inform the decision to discontinue MDA at a threshold where the ocular CT population is no longer self-sustaining. There is, therefore, a major role for quantitative tests, of which ddPCR is one of several next-generation technologies that improve upon qPCR. We envisage ddPCR will complement, rather than displace the existing technologies that are currently used in chlamydial and infectious disease diagnostics.

Financial & competing interests disclosure

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending or royalties.

No writing assistance was utilized in the production of this manuscript.