Phenotyping COPD using High Resolution CT. Is it time to leave it for Watson?

Apparently herding cats is difficult but that challenge pales when compared with what was attempted by the group of pulmonologists and pulmonary radiologists who combined to generate the data in, “ A combined pulmonary-radiology workshop for visual evaluation of COPD: study design, chest CT findings and concordance with quantitative evaluation” which is reported in this issue of the Journal. To get 33 pulmonologists and 25 radiologists to grade 294 inspiratory and expiratory CT images using a formalized approach to interpretation and with each scan being separately scored by between 9 and 11 individuals is an impressive feat of organization (and discipline!). The goals of this study were to examine the quantitative and qualitative CT features of smokers with and without chronic obstructive pulmonary disease (COPD), to evaluate the inter-observer reproducibility of the qualitative scores and to examine their relationship to the quantitative scores.

Computed tomography (CT) has been in wide clinical use for the assessment of lung disease since the early 1980's. The major improvements in CT technology in recent years have increased the dependence on CT, and this is extremely evident is in the study of COPD. While it was recognized that the anatomic underpinnings of COPD were structural changes in airways (remodelling and narrowing) and parenchyma (emphysema) it was difficult to quantify and separate these abnormalities in individual patients in the pre-CT era since both processes produce similar functional consequences and the chest X-Ray does not have the resolution to distinguish them. CT has changed all that and clinical CT reports now often include a subjective estimate of the degree of emphysema and its location as well as statements about bronchial wall thickening and gas trapping.

Importantly, almost as soon as CT became available, investigators realized that the x ray attenuation values that are used to create the image could be used to obtain quantitative data thereby potentially avoiding inter-observer variation and allowing an automated evaluation of lung structure. Many investigators have examined the ability of CT to detect the presence and severity of emphysema as well as to quantify changes in airway structure. The results have shown that the CT assessment of emphysema and airway remodelling can help to sub-phenotype COPD patients into emphysema and airway-predominant groups. The hope is that the separate mechanisms that lead to these pathological changes in COPD can be individually targeted by specific therapy and followed longitudinally with repeat imaging. But how reproducible are these measures? How do the quantitative scores relate to the qualitative scores? And which is “best”?

The first two of these questions have been answered in a robust fashion by Lynch et al (1) who gathered a large group of highly motivated radiologists and pulmonologists at the American College of Radiology Education Center in Reston, VA. These readers were given training on how to evaluate emphysema, gas trapping and airway wall thickening and then scored a group of CT scans done on patients with different severity of COPD. The scores were compared to those of other readers and to a quantitative analysis of the same CT scans. Not surprisingly, the results of this study show that there was reasonable concordance between the extent of abnormalities and the degree of COPD measured by GOLD class, but there was considerable inter-observer variability for the subjective measures. In addition there was moderate concordance between the subjective estimates of emphysema and gas trapping and quantitative CT but less strong concordance for airway remodelling.

On the face of it, these data provide strong support for abandoning subjective scoring, which is time and resource intensive in addition to being highly variable, and for instituting standardized computer-based algorithms. However the reproducibility of a test is not its only salient feature, validity is equally important. Which of the qualitative or quantitative scoring systems best reflects altered lung structure and is most predictive of respiratory symptoms and disease progression? Unfortunately, the study of Lynch and associates can't address this question.

The ideal study design is to compare qualitative and quantitative measures of emphysema and airway wall remodelling to the gold standard; pathological assessment of these abnormalities. If the computerized algorithms are equally, or more sensitive and specific then they would be “better” since they are completely reproducible provided that similar scanners, imaging parameters and software are used. Few such comparisons have been done. Bankier et al (2) did such a study for emphysema on a group of 62 subjects who were having a lobe or lung resected. They found that the quantitative emphysema score (%LAA –950HU) correlated more closely with morphologic emphysema than did the mean or individual subjective scores of three observers. These data support the use of objective computer algorithms for emphysema, the only caveat being that in the Bankier study the correlation between subjective emphysema scores and morphometrically measured emphysema was lower than have been reported by others. (3,4). A similar study has not been done to compare qualitative and quantitative measures of airway wall dimensions.

In contrast there have been a number of studies in which quantitative estimates of CT phenotypes have been compared to measures of lung function and symptoms. The reasoning appears to be that, in the absence of a structural gold standard, lung function and symptoms can act as a surrogate to test validity. If CT can accurately assess anatomic derangement of lung structure and if structural damage correlates with lung function and symptoms, then there should be good relationships between the CT measures and these clinical features.

Mets et al (5) studied 248 subjects with varying degrees of COPD and showed that quantitative estimates of emphysema and gas trapping explained between 68 and 83% of the variation in measures of lung function (FEV1, FEV1/FVC, RV/TLC, Kco) in multivariate models. Similarly Grydeland et al (6) in a study of more than 700 subjects have shown that quantitative CT measures of emphysema and airway wall thickness are related to DLco. This group also found that quantitative CT measures of emphysema and airway wall thickness were significant independent predictors of dyspnea in COPD and that airway wall thickness was associated with cough and wheezing. (7)

Han et al (8) studied 156 subjects with variable airflow obstruction and found that quantitative CT estimates of parenchymal and airway disease were independently associated with St George's Respiratory Questionnaire results. Bronchial thickness but not emphysema was associated with exacerbation frequency while emphysema was a predictor of the BODE index, a composite clinical assessment of the clinical impact of COPD.

In none of these studies were quantitative and qualitative scores compared for their ability to predict lung function or symptoms. One study in which the methods were compared was conducted by Cavigli et al. (9) They showed that quantitative estimates of emphysema were slightly better correlated with the diffusing capacity than qualitative estimates in a group of 30 COPD patients. More recently Gietema et al (10) compared qualitative and quantitative emphysema scores in 1519 subjects enrolled in the ECLIPSE study. They found that the quantitative score of emphysema related more closely to measures of airflow obstruction than the qualitative estimate. They also tried to tease out what features on the CT were influencing the radiologists in their assessment of the presence of emphysema. Using multivariate analysis they showed that the location of the low attenuation areas and the propensity of low attenuation areas to be clustered in identifiable emphysematous “holes” were independent contributors to the radiologists scores.

The most commonly used quantitative measure of emphysema is the percentage of the lung voxels with an attenuation value below a cut off value (eg -950 Houndsfield units) and the algorithms that are used to determine this parameter don't take into account the spatial distribution of the low attenuation voxels. Human readers are able to appreciate the clustering of low attenuation areas into emphysematous spaces. With the progression of emphysema (especially centrilobular emphysema) low attenuation voxels are concentrated in the centrilobular “holes” and it is these holes that the human reader can appreciate. Thus theoretically a subjective score could incorporate important anatomic information that is not captured by using a simple attenuation cut off. In an attempt to develop computer-based algorithms to quantify this spatial concentration cluster, or fractal analyses of the CT images were proposed (11). Recently Yuan and associates (12) showed that by combining a density cut off and a measure of clustering they were better able to estimate a sensitive microscopic index of emphysema, the lung's surface to volume ratio than by using either measurement alone. Furthermore, previous studies by Uppaluri (13) showed that complex computer algorithms were not only more reproducible than humans, but were more accurate in separating the lungs of smokers from non-smokers.

So how does the study of Lynch et al (1) aid us in deciding whether or not to leave it to Watson (the IBM computer who recently beat the best of the best at Jeopardy)? The investigators were cautious in their conclusions. They state that since they had no true gold standard, “perhaps it is best to regard them (qualitative and quantitative measures) as complementary”. This conclusion is based on the idea that while modern computers and search engines are very fast they are not very good at answering complicated questions and because COPD is a complicated disease, or group of diseases, the human brain may still be able to detect patterns not accessible to computers

Being less cautious we suggest that this careful study adds to the accumulating evidence that it is time to leave it to Watson; the bulk of studies indicate that the reliability and validity of computer aided diagnostics which measure the extent and anatomic location of disease are superior to subjective measures in COPD. With the wealth of data available it should be possible to apply complex algorithms to computer analysis that will reconcile the image with the complicated clinical symptoms of the patient. While it may be that mere humans are still important in the study and treatment of disease, maybe it is time to let Watson have the final say in the data analysis.