Comment on “Fecal occult blood versus DNA testing: indirect comparison in a colorectal cancer screening population”

Dear editor

Challenges to medical innovation are welcome and essential elements in the dialectic toward practice optimization but must be based on scientifically sound criticism. A multi-target stool DNA test (MT-sDNA) has emerged as a new approach to colorectal cancer (CRC) screening, and in the US, it is approved by federal regulatory bodies, endorsed in societal screening guidelines, and covered by government and private insurers. In their recent paper,¹ Brenner and Chen concluded that CRC screening sensitivity by fecal immunochemical test (FIT) for occult blood is equal to or better than that of MT-sDNA when tests are compared at matched specificity. However, concerns can be raised regarding their methodology and data.

1. Major differences exist across study populations: a cardinal scientific principle dictates that tests are best compared directly within the same population. In this study from Germany,¹ observed FIT results were compared with reported MT-sDNA results from a large multicenter US study.² Their indirect comparison is confounded by between-population differences in patient, lesion, and other variables. For example, patients were substantially younger and CRCs were more advanced in the German study; 67% of German patients versus 37% of US patients were younger than 65 years, and 46% versus 14% of CRCs were Stages III–IV, respectively. These factors may significantly influence test performance.

2. Direct sensitivity comparisons at matched specificities done within a single test population contradict conclusions in the present indirect comparison: in the US study report,² MT-sDNA and FIT sensitivities were directly compared in area under the curve plots across the full range of specificities. At matched specificity of 86.6%, CRC sensitivity was 92.3% for MT-sDNA and 76.9% for FIT, p=0.015. This direct within-study comparison holds greater validity than indirect between-study comparisons.

3. Observed FIT sensitivity is atypical: in the authors’ own prior studies from the same German consortium, FIT sensitivities for CRC were 53%–73%.^3,4 In prior studies using FOB Gold (the FIT used in the present study), CRC screen sensitivities were reported to be 50%–75%.^5,6 These FIT sensitivities are much lower than the 96.7% observed in the present study, which appears to be an outlier.

4. MT-sDNA specificity used is not representative: the comparator US multicenter study was intentionally skewed toward older patients to enrich for pathology and to meet Medicare data requisites, which lowered specificity;² specificity was 94% in patients aged 50–64 years with normal colonoscopy. In a second blinded screen-setting study from Alaska,⁷ specificity for MT-sDNA was 93% and sensitivity for CRC was 100%. Using conservative point specificities, we have estimated that MT-sDNA done at the recommended frequency of every 3 years generates fewer false-positives over a longitudinal program than does FIT done annually. Thus, program specificity may be functionally higher with MT-sDNA than with FIT. In alignment with this, modeling of CRC screening approaches by the US Preventive Services Task Force showed that MT-sDNA done every 3 years yielded the highest benefit-to-harm ratio and fewest unnecessary colonoscopies.⁸

Given the aforementioned scientific concerns, caution must be taken in interpreting the recent report by Brenner and Chen.

Disclosure

Mayo Clinic is an equity investor in Exact Sciences. Dr. Ahlquist is a co-developer of a stool DNA test commercialized by Exact Sciences and serves as a scientific advisor to and research collaborator with Exact Sciences. Mayo Clinic receives royalties from Exact Sciences and, according to policy, shares those with Dr. Ahlquist. The author reports no other conflicts of interest in this communication.

Authors’ replyCorrespondence: Hermann Brenner, Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 581, 69120 Heidelberg, Germany, Tel [phone], Fax [phone], Email [email protected]

Dear editor

We appreciate Dr. Ahlquist’s interest in our article¹ that challenges his previously disseminated claim that the multitarget stool DNA test (MSDT) represents “the new high bar benchmark for noninvasive CRC screening”.² Expectations on the diagnostic performance of this test should in fact be high, given the following facts not mentioned by Dr. Ahlquist:

• The costs of MSDT exceed the costs of fecal immunochemical test (FIT) for hemoglobin, the noninvasive test of choice in an increasing number of countries,³ ~20-fold.⁴

• MSDT requires substantially more complex logistics for stool sampling and shipping (collection and shipping of a whole bowel movement compared to a few milligrams of feces needed for FIT).

• MSDT was ~20 times more often not evaluable than FIT in the study referred to and co-authored by Dr. Ahlquist and sponsored by the manufacturer that had reported on a direct comparison of both tests.⁵

Given the different background and involvement in the MSDT development, evaluation, and promotion of the research community, we are not surprised that the results of our study, which found the diagnostic performance of a quantitative FIT to be essentially equivalent to the previously reported diagnostic performance of MSDT,⁵ would provoke rather controversial reactions.

Although all of the specific concerns raised by Dr. Ahlquist were carefully and critically addressed in our article,¹ we would like to briefly add the following few comments to each of his concerns:

1. Major differences across study populations: it is correct that the MSDT study by Dr. Ahlquist, as a result of intentional oversampling of adults ≥65 years of age, had included a much larger proportion (63%) of adults ≥65 years of age than our study (33%). Due to intentional oversampling of adults ≥65 years of age, the MSDT study was not representative of a true screening population. Nevertheless, in order to address any potential impact of the differences in age structure we reported results of additional analyses in which we adjusted our findings to the age structure of the MSDT study. As reported in our article,¹ this age adjustment did not alter any of our conclusions.

2. Selective reporting of results favoring MSDT: in his letter, Dr. Ahlquist selectively reports a single statistic from the MSDT study that apparently favors the author’s view, while not addressing other, less favorable statistics, such as the 20-fold higher rate of non-evaluable results of MSDT compared to FIT. The single statistic reported by Dr. Ahlquist is based on an exceptionally poor result for FIT obtained in the MSDT study, which is a true outlier compared to the large number of FIT studies reported to date (for details see the following paragraph).

3. Selective referencing of FIT results: Dr. Ahlquist cites FIT sensitivities for CRC from selected previous studies ranging from 50% to 75%. According to a systematic literature review by Lee et al,⁶ sensitivities of FITs for detecting CRC ranged from 25% to 100%. Furthermore, analogous comparative analyses using another widely used FIT had come to essentially the same conclusion.⁷ Dr. Ahlquist’s ascribing our results to “atypical” FIT results is therefore not justified. Quite the contrary is true: the meta-analysis by Lee et al yielded an area under the curve (AUC) of the hierarchical summary receiver-operating characteristic curve of 0.95 with a 95% confidence interval of 0.93–0.97 for 19 studies evaluating diagnostic performance of FIT for CRC detection.⁶ The AUC of 0.89 reported for FIT in the MSDT study by Ahlquist,⁵ which is well below the lower end of the 95% confidence interval of the AUC for the 19 studies included in the meta-analysis,⁶ demonstrates that the true outlier is the poor FIT performance in the MSDT study whereas the AUC of the FIT in our study was well within the range observed by other studies.

4. Unsubstantiated claims and biased comparisons: Dr. Ahlquist claims: “Using conservative point-specificities, we have estimated that MT-sDNA [note that this corresponds to MSDT in our terminology] done at the recommended frequency of every 3 years generates fewer false-positives over a longitudinal program than does FIT done annually”. The assumptions and calculations made to support this claim are neither provided nor referenced by Dr. Ahlquist. Furthermore, it is most obvious that more frequent testing decreases program specificity while increasing program sensitivity. Thus, comparing MSDT every 3 years with annual FIT and exclusively referring to specificity inevitably results in biased comparisons.

Dr. Ahlquist summarizes his remarks by concluding that caution must be taken in interpreting our recent report. While we certainly support suggestions to take care and caution in the interpretation of any scientific work, we feel that particular caution is needed when interpreting Dr. Ahlquist’s statements.

Disclosure

The authors report no conflicts of interest in this communication.