External validation of a prognostic CT-based radiomic signature in oropharyngeal squamous cell carcinoma

ABSTRACT

Background. Oropharyngeal squamous cell carcinoma (OPSCC) is one of the fastest growing disease sites of head and neck cancers. A recently described radiomic signature, based exclusively on pre-treatment computed tomography (CT) imaging of the primary tumor volume, was found to be prognostic in independent cohorts of lung and head and neck cancer patients treated in the Netherlands. Here, we further validate this signature in a large and independent North American cohort of OPSCC patients, also considering CT artifacts.

Methods. A total of 542 OPSCC patients were included for which we determined the prognostic index (PI) of the radiomic signature. We tested the signature model fit in a Cox regression and assessed model discrimination with Harrell's c-index. Kaplan-Meier survival curves between high and low signature predictions were compared with a log-rank test. Validation was performed in the complete cohort (PMH1) and in the subset of patients without (PMH2) and with (PMH3) visible CT artifacts within the delineated tumor region.

Results. We identified 267 (49%) patients without and 275 (51%) with visible CT artifacts. The calibration slope (β) on the PI in a Cox proportional hazards model was 1.27 (H₀: β = 1, p = 0.152) in the PMH1 (n = 542), 0.855 (H₀: β = 1, p = 0.524) in the PMH2 (n = 267) and 1.99 (H₀: β = 1, p = 0.002) in the PMH3 (n = 275) cohort. Harrell's c-index was 0.628 (p = 2.72e-9), 0.634 (p = 2.7e-6) and 0.647 (p = 5.35e-6) for the PMH1, PMH2 and PMH3 cohort, respectively. Kaplan-Meier survival curves were significantly different (p < 0.05) between high and low radiomic signature model predictions for all cohorts.

Conclusion. Overall, the signature validated well using all CT images as-is, demonstrating a good model fit and preservation of discrimination. Even though CT artifacts were shown to be of influence, the signature had significant prognostic power regardless if patients with CT artifacts were included.

Accounting for approximately half a million cases annually worldwide, head and neck squamous cell carcinoma (HNSCC) is a considerable cause of mortality and morbidity, with the majority of patients having locally advanced, unresectable disease [1]. Oropharyngeal squamous cell carcinoma (OPSCC) has been one of the fastest growing disease sites for HNSCC [2].

Known prognostic factors of locally advanced HNSCC include tumor category, nodal category and human papilloma virus (HPV) status, the latter in particular related to overall survival for OPSCC patients [3–6]. Other potential prognostic factors are obtained by molecular characterization of the tumor, mostly requiring tissue extraction [7–9]. The inherent limitations of biopsies are however their invasiveness and probability of misrepresenting the entire tumor due to intra-tumor heterogeneity, as they only characterize a small portion of the tumor [10]. In contrast, medical imaging is non-invasive and able to capture the entire tumor volume, including intra-tumor heterogeneity, which could provide additional information to supplement traditional tissue biopsy [11]. Nowadays, imaging is used routinely throughout the course of treatment and therefore there is ready access to this useful information.

Radiomics is a high-throughput approach to translate medical images into mineable data by extracting a large number of quantitative features describing tumor intensity, shape, and texture [12–14]. The hypothesis being that a comprehensive and robust [15–19] quantification of imaging phenotypes provides complementary and clinically relevant information, which may lead to imaging biomarkers [20]. As shown in recent studies, quantitative imaging features have prognostic value and potential in predicting clinical outcomes or treatment monitoring in different cancer types [21–26].

Here, we focus on a recently described prognostic radiomic signature, which is based exclusively on pre-treatment CT imaging of the primary tumor volume [27]. This signature was derived from non-small cell lung cancer (NSCLC) patients and independently validated to be not only prognostic in NSCLC, but as well in two HNSCC patient cohorts, of which all patients were treated in the Netherlands. In this study we aim to further validate the prognostic value of this radiomic signature in a large and independent North American cohort of OPSCC patients (n = 542), also considering the presence of CT artifacts [28].

Materials and methods

Patients and CT imaging

Institutional research ethics board approval was obtained and the need for written consent was waived for this retrospective study. A total of 542 patients with OPSCC, treated with curative intent at the Princess Margaret Cancer Center (PMH) between 2005 and 2010 were included in this study. Treatment consisted of radiotherapy or concurrent chemoradiotherapy, with standard fractionated IMRT up to 70 Gy. All patients underwent pre-treatment CT imaging of the head and neck on one of available CT scanners (General Electric Discovery ST; General Electric Lightspeed Plus; Toshiba Medical Systems Aquillion ONE). CT scans were acquired in helical mode with a slice thickness of 2.5 mm (General Electric) or 2 mm (Toshiba), at 120 kVp and 300 mAs tube current (variable tube current for Toshiba scans). The gross primary tumor volume (GTV) was manually delineated for each patient for treatment planning purposes (Figure 1a). Images were visually assessed for the presence of CT artifacts (e.g. streak artifacts due to dental fillings) within the GTV (Figure 1b).

Figure 1. Representative images of OPSCC patients from the validation cohort, without (a) and with (b) visible CT artifacts. The GTV delineation is shown in green.

Radiomic signature

The radiomic signature we aim to validate consists of the following four features, derived from the GTV: 1) “First order statistics: Energy”, describing the overall density of the tumor volume; 2) “Shape: Compactness”, quantifying the compactness of the tumor volume relative to that of a sphere (i.e. the most compact shape); 3) “Gray level run length: Gray level non-uniformity”, a measure of intra- tumor heterogeneity; and 4) Wavelet (HLH) “Gray level run length: Gray level non-uniformity”, also describing intra-tumor heterogeneity, but now after wavelet decomposition of the original CT image. A detailed mathematical description of the aforementioned features, as described by Aerts et al. [27], can be found in the Supplementary Appendix, to be found online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2015.1061214. Features were extracted using software developed in-house, in Matlab R2012b.

The radiomic signature was based on a Cox proportional hazards model and the weights (β) for each individual feature (x) in the signature are given in Table I. The prognostic index (PI) for the radiomic signature, to be used for validation, is then defined as:

Table I. Description and Cox proportional hazard weights of each feature in the radiomic signature.

Feature type	Feature name	Weight
First order statistics	Energy	2.42e-11
Shape	Compactness	−5.38e-03
Gray level run length	Gray level non-uniformity	−1.47e-04
Wavelet (HLH) Gray level run length	Gray level non-uniformity	9.39e-06

Signature validation

To validate the radiomic signature we applied several methods, as described by Royston and Altman [29]. We first determined the model calibration slope (i.e. regression coefficient) on the PI in a Cox regression in the validation cohort and performed a likelihood ratio test of this slope being equal to 1. If the slope equals 1, the relative risk model is valid, otherwise there is a need for recalibration. We formally tested the coefficients (i.e. weights) of the individual variables of the PI, by performing a Cox regression on the individual features of the signature in the validation cohort, offsetting by the original PI (i.e. the coefficient of the PI is 1) and performing a joint test that all coefficients are 0. As a measure of model discrimination in the validation cohort, we determined Harrell's c-index, where a c-index of 1 indicates perfect discrimination. Finally, we compared Kaplan-Meier survival curves between patients with a high and low signature prediction, based on a median threshold that was derived from the MAASTRO “Lung1” cohort by Aerts et al. [27]. A log-rank test was applied to test for significant differences between survival curves.

We validated the radiomic signature in the complete patient cohort (PMH1), in the subset of patients for which there were no visible CT artifacts within the delineated tumor region (PMH2) and in the subset of patients that did have visible CT artifacts (PMH3). All statistical analysis was performed in R (version 3.1.0).

Results

By visual assessment, we identified all scans with CT artifacts inside the GTV, which resulted in a subset of 275 (51%) patients. In Table II we summarized patient characteristic in the complete PMH validation cohort (PMH1), the patient subgroups without (PMH2) and with (PMH3) CT artifacts and, for comparison, the patient characteristics of the MAASTRO “H&N1” and VUmc “H&N2” cohorts originally used for validation of the radiomic signature by Aerts et al.

Table II. Patient characteristics of the PMH validation cohort (PMH1), the subset of patients with no visible CT artifacts within the GTV (PMH2), the subset of patients with visible CT artifacts (PMH3), the MAASTRO “H&N1” cohort and the VUmc “H&N2” cohort [27]. ^#HPV status only for oropharyngeal patients.

	Frequency (%)
Variable	PMH1 (n = 542)	PMH2 (n = 267)	PMH3 (n = 275)	MAASTRO (n = 136)	VUmc (n = 95)
Gender
Male	79.9	82.0	77.8	81.5	65.3
Female	20.1	18.0	22.2	18.5	34.7
Primary tumor site
Oropharynx	100.0	100.0	100.0	64.0	100.0
Larynx	0	0	0	36.0	0
T-category
T1	12.5	13.9	11.3	25.9	10.5
T2	31.9	32.6	31.3	23.0	32.6
T3	33.4	33.3	33.5	17.8	35.8
T4	22.1	20.2	24.0	33.3	21.1
N-category
N0	17.3	16.1	18.5	45.2	44.2
N1	10.3	10.1	10.5	11.9	11.6
N2	65.7	65.5	65.8	40.7	42.1
N3	6.6	8.2	5.1	2.2	2.1
Overall stage
Stage I	12.5	13.9	11.3	18.5	8.4
Stage II	31.9	32.6	31.3	8.1	18.9
Stage III	33.4	33.3	33.5	17.0	18.9
Stage IVA	15.7	13.1	18.2	54.3	45.3
Stage IVB	6.5	7.1	5.8	2.2	7.4
Stage IVC	0	0	0	0	1.1
HPV/p16 status	p16	p16	p16	HPV^#	HPV
Positive	56.3	49.4	62.9	28.4	18.9
Negative	24.0	30.3	17.8	71.6	81.1
Unknown	19.7	20.2	19.3	0	0
Treatment
Radiotherapy	49.1	55.4	42.9	74.1	58.9
Chemoradiotherapy	50.9	44.6	57.1	25.9	41.1

In the complete PMH validation cohort (PMH1; n = 542), the calibration slope on the PI in a Cox proportional hazards model was found to be 1.27 (SE = 0.175). The slope was slightly above 1, but not significantly different from 1 (p = 0.152), indicating a valid relative risk model and preservation of the discriminative value of the radiomic signature in the validation cohort. The joint test of all the predictors in the model with the PI offset was significant (χ₄² = 21.87, p = 2.13e-4). Harrell's c-index for the PI was found to be 0.628 (p = 2.72e-9). Survival curves were significantly different (p = 1.93e-5) between patients with high and low radiomic signature model predictions (Figure 2a).

Figure 2. Survival curves based on model predictions of the radiomic signature, split by a median prediction threshold derived by Aerts et al. from the MAASTRO “Lung1” cohort [27]. (a) Survival curves for the PMH validation cohort for all patients (log-rank test p = 1.93e-5) and for the subset of patients without (log-rank test p = 4.89e-5) and with (log-rank test p = 0.004) visible CT artifacts within the GTV. (b) Survival curves for the MAASTRO “H&N1” (log-rank test p = 8.48e-05) and VUmc “H&N2” (log-rank test p = 0.030) cohorts as reported by Aerts et al.

After excluding patients with visible CT artifacts within the GTV (PMH2; n = 267), the calibration slope on the PI in a Cox proportional hazards model was found to be 0.855 (SE = 0.236) and not significantly different from 1 (p = 0.524). In the model with the PI offset, the joint test of all individual feature coefficients was significant (χ₄² = 12.31, p = 0.015). Harrell's c-index for the PI was 0.634 (p = 2.7e-6) and a significant difference between survival curves was observed (p = 4.89e-5) in this subset of patients (Figure 2a).

Considering patients with visible CT artifacts (PMH3; n = 275), the calibration slope on the PI was 1.99 (SE = 0.273), which was significantly different from 1 (p = 0.002). The joint test of all predictors was significant (χ₄² = 16.81, p = 0.002) in the model with the PI offset. The c-index was found to be 0.647 (p = 5.35e-6) and survival curves stratified by high and low signature model predictions were significantly different (p = 0.004).

Discussion

An important step towards clinically using radiomics in the context of personalized medicine [14] is independent and external validation of proposed signatures [3,4]. Here, we evaluated the validity of a recently published CT-based radiomic signature [27]. This signature was described to be prognostic in independent cohorts of both lung and head and neck cancer patients. We found that this signature validated as well in an additional large cohort of OPSCC patients.

As specified in the original publication, the radiomic features of which the signature consists were calculated from the imaging data as-is [27]. CT images were therefore used as generated by the CT scanner and no pre-processing or normalization was performed before feature calculation. Even though it is known that scanner parameters (e.g. slice thickness or reconstruction kernels), which differ across and within patient cohorts, affect textural features computed from CT images [30], Aerts et al. [27] showed translational potential of the radiomic signature across different cohorts. This statement is further strengthened by our findings, given the good model fit and preservation of discriminative value of the signature in our validation cohort (PMH1). In comparison, Aerts et al reported c-indices of 0.686 and 0.685 in two independent head and neck cancer cohorts, whereas we found a c-index of 0.628. Furthermore, survival curves were significantly different, based on a median threshold of signature predictions, derived by Aerts et al. from the MAASTRO “Lung1” cohort. These results are in line with what has been reported by Aerts et al., for both the MAASTRO “H&N1” (p = 8.48e-05) and VUmc “H&N2” (p = 0.030) cohorts and a side-by-side comparison of survival curves is depicted in Figure 2. Even though our study endorses translational potential of the radiomic signature, we believe that standardization of imaging protocols should be pursued to eliminate variability in radiomic features between institutes, which will greatly improve the potential of radiomics [31,32].

Another common concern in CT images of head and neck cancer are artifacts, mostly caused by metallic dental fillings or other high atomic number material implants [28]. It has to be taken into consideration that the radiomic signature was derived from CT imaging of NSCLC patients, where these type of artifacts (e.g. due to pacemakers) are uncommon. As an additional step we therefore validated the radiomic signature as well on the subset of patients without (PMH2) and with (PMH3) any visible CT artifacts within the delineated tumor region. In the PMH2 cohort subset, the calibration slope deviated less from 1 than in the complete (PMH1) cohort, signifying a better fit of the relative risk model. In contrast, the relative risk model was found to be invalid in the PMH3 cohort, indicating a need for recalibration of the model. These results suggest that there is an influence of CT artifacts on the model fit. Regardless the inclusion of patients with CT artifacts, the discriminative value of the radiomic signature was however preserved in both the PMH2 and PMH3 patient subsets, supported by Harrell's c-indices of 0.634 and 0.647, respectively. A significant difference between survival curves, stratified by high and low signature model predictions, was preserved as well in both patient subsets (Figure 2a). The extent of CT artifacts and the impact on imaging features for head and neck cancer will vary between patients. Promising techniques for metal artifact reduction in CT have been reported [33]. As radiomics relies on extracting meaningful information from medical images, techniques like these should however be thoroughly evaluated (i.e. they should not modify or introduce artificial texture). Besides the influence of the presence of CT artifacts, we also found evidence (joint tests of all the predictors in the model with the PI offset) that the fit of the radiomic signature model in our validation could be improved by adjusting the original weights of the predictors in the PI, regardless of the validity of the relative risk model.

Here we focused on the prognostic value of a radiomic signature, which only contains information derived from standard medical imaging. While our validation study provides further evidence for the concept of radiomics, we do believe that proven prognostic factors like HPV status and other clinical parameters [3–6] should as well be carefully considered in addition to radiomic information. Indeed, Aerts et al. already pointed out that HPV screening for instance provides complementary information to the radiomic signature [27]. Deriving a novel and disease-specific signature for head and neck cancer [34], combining radiomic and clinical information, is therefore warranted for future research – a process that should again be followed by independent validation.

Conclusion

We externally validated a previously described CT-based prognostic radiomic signature in a large OPSCC cohort. Overall, the signature validated well using all CT images as-is, demonstrating a good model fit and preservation of discrimination. Our results showed that CT artifacts are of influence. However, the signature had significant prognostic power regardless if patients with CT artifacts were included. Besides CT artifacts, proven prognostic factors like HPV status should as well be carefully considered, and deriving a novel and disease-specific signature is warranted.

Supplementary material available online

Supplementary Appendix and Figure 1, to be found online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2015.1061214.

Acknowledgments

We acknowledge financial support from the QuIC–ConCePT project, which is partly funded by EFPIA companies and the Innovative Medicine Initiative Joint Undertaking (IMI JU) under Grant Agreement No. 115151. This research is also supported by the Dutch technology Foundation STW (grant No. 10696 DuCAT), which is the applied science division of NWO, and the Technology Programme of the Ministry of Economic Affairs. We also acknowledge financial support from the National Institute of Health (NIH-USA U01 CA 143062-01, and NIH-USA U01 CA 190234-01), EU 7th framework program (EURECA, ARTFORCE), Kankeronderzoekfonds Limburg from the Health Foundation Limburg and the Dutch Cancer Society (KWF UM 2011 – 5020, KWF UM 2009 – 4454), and Alpe d’HuZes-KWF (DESIGN).

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.