Quantitative image quality evaluation of pelvic computed tomography-based imaging systems: A novel concept in radiotherapy

To the Editor,

Different cone beam computed tomography (CBCT) systems are commercially available for radiotherapy (RT). The image quality of these systems differs as a consequence of different hardware and software. However, image quality assessment based on images of phantoms and related physical measures usually cannot tell us whether we should prefer one system over the other, in a specific clinical setting. On the other hand, subjective, non-quantitative evaluations based on the preferences of individual observers are often not sufficient either. Therefore, a quantitative evaluation method based on clinically relevant features of images of actual patients is called for.

A method called visual grading characteristics (VGC) analysis has been developed by Båth and colleagues [1,2]. The VGC is a non-parametric, rank-invariant statistical analysis which can be used for quantitative evaluation of the difference in image quality between two imaging systems. To our knowledge, VGC has never been applied in the field of RT.

The aim of this study was to test VGC analysis as a tool to evaluate clinically relevant image quality differences between CT-based imaging systems used in RT. Furthermore, the aim was to give a preliminary indication of whether one of two specific CBCT systems should be preferred when used for visualization of the bladder in RT [3].

Material and methods

Both home clinics of the authors of the present paper have two different treatment units with CBCT capabilities, i.e. Varian iX (CBCT_iX) [4] and Varian TrueBeam (CBCT_Tb). The CBCT systems on these two different treatment unit types differ and the image quality may differ as well. In short, the CBCT_iX system uses a standard phantom-based normalization in the image reconstruction to account for scatter and beam hardening effects. By contrast, the CBCT_Tb employs a patient-specific scatter correction algorithm as well as an improved analytical beam hardening correction.

The patient data used in this retrospective study were obtained from Aarhus University Hospital. Four male patients were included. All patients had scans obtained with CBCT_iX and CBCT_Tb, as well as a planning CT (pCT) (Supplementary Appendix, to be found online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2013.818252).

Based on European guidelines provided by the Commission of the European Communities (CEC) [5,6] and in cooperation with a physician (physician 1, HL), eight image quality criteria in relation to bladder on CBCT were formulated. For every CBCT and pCT, the fulfillment of the criteria was graded using an ordinal scale from 1 to 5; with 1 and 5 corresponding to “Confident that the criterion is fulfilled” and “Confident that the criterion is not fulfilled”, respectively. The grading was performed by four observers: physicians 1 and 2, and RTTs 1 and 2 (see Supplementary Appendix to be found online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2013.818252).

As in several other studies, e.g. [7–9], the results were pooled for all criteria, however, not for the observers. Thus, individual VGC curves for the observers were created using the software DBM MRMC 2.32 Build 3 [10–17]. The DBM MRMC is a suitable receiver operating characteristic (ROC) software and can be used for VGC analysis [1]. By using ANOVA provided by the software, the areas under the curve (AUCs) and the 95% CIs were determined.

The smallest number of CBCT_iX was three (for Patient III). Hence, for each patient, three CBCT_Tb and three CBCT_iX were randomly selected using Matlab R2011b. This ensured that all patients contributed equally to the analysis. The images were given a random number between 1 and 28. Thus, the observers were blinded when performing the analysis. The order in which the observers examined the images was also random. The randomization was performed to avoid reading order effects [18]. The results were analyzed in terms of VGC analysis of pCT compared to CBCT_iX and CBCT_Tb compared to CBCT_iX.

Results

The AUCs for all pCT vs. CBCT_iX curves were above 0.5, and 0.5 was not included in the 95% CIs except for physician 1 where 0.5 was just within the 95% CI (Figure 1 and Table I). For CBCT_Tb versus CBCT_iX the results were less clear. There was a tendency that CBCT_Tb was superior to CBCT_iX with 95% CIs shifted towards values higher than 0.5. However, it was only for the two RTTs that 0.5 was not included in the CIs.

Figure 1. VGC analysis for physician 1, physician 2, RTT1, and RTT2, respectively. pCT performs better than CBCT_iX with all points above the “equal line”. The equal line corresponds to no difference between the two modalities. In general, CBCT_Tb seems to be superior to CBCT_iX, however, one point is below the equal line (for physician 1) and 0.5 is within the CIs for the physicians (Table I).

Table I. The results of the VGC analysis. AUCs for pCT compared to CBCT_iX and CBCT_Tb compared to CBCT_iX as well as the 95% CIs.

	pCT vs. CBCTiX		CBCTTb vs. CBCTiX
	AUC	95% CI	AUC	95% CI
Physician 1	0.61	[0.48; 0.74]	0.53	[0.44; 0.61]
Physician 2	0.82	[0.73; 0.92]	0.56	[0.48; 0.65]
RTT 1	0.88	[0.78; 0.99]	0.67	[0.58; 0.76]
RTT2	0.95	[0.91; 0.99]	0.64	[0.55; 0.72]

Discussion

The superiority in image quality of CT over CBCT is well known [22–24]. Several methods to improve the image quality of CBCT images have been suggested and investigated, e.g. scatter corrections based on either measurements, Monte Carlo simulations or other algorithms or improvement of the CBCT image quality by using the information from a standard CT scan [23–28]. The result of this study that CT seems to provide better image quality than CBCT was therefore expected from the basic differences in the two acquisition techniques. As opposed to CT, CBCT acquisition takes approximately 60 seconds of scan time with the entire anatomical region in the field of view. Thus, CBCT images are more prone to artifacts from movement of interfaces between different densities during acquisition. Furthermore, scatter and beam hardening effects will deteriorate the soft tissue contrast centrally in the patient where the bladder is located. The contribution from the latter has been decreased in the CBCT reconstruction on the TrueBeam platform compared to iX which is reflected in the results of this study.

Other studies have compared image quality of different CBCT systems for RT or different operating modes or software solutions of the same CBCT system, however not using the VGC method used in this study [28–31]. The VGC curves (Figure 1) give a clear and quick illustration of the trend in results, but they cannot stand alone as the CIs are needed in order to clarify whether the results are statistical significant or not. However, the CIs and AUCs do not tell the whole story either.

The calculations of the CIs are based on the assumption that the AUC can be treated as a normally distributed variable. This assumption is valid in most cases if AUC is not close to 1 and if the number of cases is > 50 [1]. In the present study, the number of cases is < 50 and the assumption of normality may not hold. Therefore, even though, e.g. CT is superior to CBCT for the tested cases as evaluated by the observers of this study, the current study does not have the statistical power to generalize this.

VGC analysis is well suited for comparing different CT-based image modalities used in RT. The results of VGC analysis can be based on clinically relevant differences in image quality of actual patient images. The method thus overcomes the potential problems that images of phantoms and related physical measures may have little relevance in the daily clinical use of the imaging systems.

Acknowledgments

Thanks to physician Camilla Kjær Lønkvist, and RTTs Camilla Lee Dann and Dorrit Larsen from Herlev Hospital for their time and participation in the VGC analysis.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.