Clinical validation of atlas-based auto-segmentation of pelvic volumes and normal tissue in rectal tumors using auto-segmentation computed system

Abstract

Purpose. To evaluate in two different settings – clinical practice and education/training – the reliability, time efficiency and the ideal sequence of an atlas-based auto-segmentation system in pelvic delineation of locally advanced rectal cancer. Methods. Fourteen consecutive patients were selected between October and December 2011. The images of four were used as an atlas and 10 used for validation. Two independent operators participated: a Delineator to contour and a Reviewer to perform an independent check (IC). The CTV, pelvic subsites and organs at risk were contoured in four different sequences. These included A: manual; B: auto-segmentation; C: auto-segmentation + manual revision; and D: manual + auto-segmentation + manual revision. Contouring was performed by the Delineator using the same planning CT. All of them underwent an IC by a Reviewer. The time required for all the contours were recorded and overlapping evaluation was assessed using a Dice coefficient. Results. In the clinical practice setting there have been 13 minutes time saved between sequences A versus sequences B (from 38 to 25 minutes, p = 0.002), a mean Dice coefficient in favor of sequences A for CTV and all subsites (p = 0.0195). In the educational/training setting there have been 35.2 minutes time saved between sequences C and D 8 (from 73.1 min to 37.9 min, p = 0.002). Conclusion. The preliminary data suggest that the use of an atlas-based auto-contouring system may help improve efficiencies in contouring in the clinical practice setting and could have a tutorial role in the educational/training setting.

Radiotherapy has undergone a significant technical evolution in the last 15 years. For example, improved target definition (GTV) by the introduction of new diagnostic image modalities, volume delineation [CTV and organs at risk (OAR)] by the use of CT-simulation images, and in planning and delivery by the introduction of three-dimensional (3D)-conformal treatment and intensity-modulated radiation therapy (IMRT). Delineation of targets and normal tissues is still largely influenced by individual skill and expertise. It has become a critical component of planning treatment to complement the continuously evolving technology delivering high precision radiotherapy. Although guidelines are available to share expertise in contouring with the goal to reduce both intra- and inter-observer variability [1,2], agreement varies to a large degree [3–5].

Our department established a certified quality assurance (QA) program in 2002, to assist and guide staff in the main critical steps of planning and treatment. The QA program for delineation recommends the following steps: 1) Contouring by a ‘Delineator’ referring to a center approved atlas derived from international guidelines; 2) Independent check (IC) of delineation before planning by a ‘Reviewer’ using the same atlas as a reference.This procedure, although assuring high precision, is time consuming requiring an average of 90 minutes for the delineation and IC in a patient with locally advanced rectal cancer; moreover, time and variability reductions have been reported by an increasing number of reports in which various auto-segmentation software programs were tested in different tumor sites [5–8].In the present study we evaluated a software system for atlas-based auto-segmentation in the pelvic delineation of locally advanced rectal cancer. The aim was to evaluate, in the two different settings (clinical and educational/training), the reliability of the system, the time savings, and the best sequence for its use in the educational/training setting.

Material and methods

The automatic segmentation process was performed using Smart Segmentation Knowledge-Based Contouring (SSKBC) (Varian Medical Systems, Palo Alto, California). This software offers a combination of heuristic and atlas-based automatic segmentation algorithms. The heuristic algorithm performs a CT density analysis and automatically detects the body outline and the bones based on this information. The atlas-based algorithm delineates the remaining anatomical structures and targets of therapy after having performed a deformable registration between a template case selected from the library and the planning CT image.

From October 2011 to December 2011, 14 consecutive patients with locally advanced rectal cancer were chosen to validate the system. Four (two females and two males) were contoured by an expert operator according to international guidelines [9–11], were independently validated by two expert radiation oncologists, and then used to populate the library (atlas patients). The other 10 were used to test the software. The patient characteristics are reported in (Supplementary Table Ia and b available online at: http//informahealthcare.com/doi/abs/10.3109/0284186X.2012.754989). In this retrospective series, cases used as atlas were representative of the Italian population. Factors such as sex, age and the median Italian body mass index were used to account for anatomical variability. CT planning images were acquired without IV contrast and slice thickness of 5 mm. The same modality of image acquisition was used both for atlas and test patients. For each test patient four different delineation modalities were applied on the same planning CT images by a Delineator, who was a third year radiation oncology resident. The Reviewer, a radiation oncologist with expertise in rectal cancer performed the IC as described in Figure 1: Group A: Manual Segmentation (Delineator) + IC (Reviewer) versus Group B: Automatic Segmentation (Delineator) + IC (Reviewer) to test the system for clinical setting implementation. Group C: Automatic Segmentation + Manual Revision (Delineator) + IC (Reviewer) versus Group D: Manual Segmentation + Automatic Segmentation + Manual Revision (Delineator) + IC (Reviewer), to test the system in the educational/training setting. The Delineator and Reviewer recorded the time required to perform delineations, to choose the most appropriate case from the library, and IC, respectively.

Figure 1. Four sets of work-flow for each patient: group A and B studied the reliability of the software in clinical practice; group C and D the validity of the system in training programs, according to QA programs.

For each of the 10 test patients the Reviewer also made a master contour. This was blinded to the Delineator and it was used for calculating Mean Dice Coefficient with CT scans contoured by Delineator.

The following structures were contoured: CTV modulated according to tumor stage and location (Table I). The individual subsites were: 1) Lymph nodes subsites [pre-sacral lymph nodes (PS), internal iliac lymph nodes (IIN) + obturator lymph nodes (ON) ± external iliac lymph nodes (EIN)]; 2) Mesorectum (M) and 3) OAR [bladder (B), head of the femur (HF)].

Table I. Subsites inclusion criteria in the rectal cancer CTV.

	Presaral space	Mesorectum	Internal iliac nodes	Obturator nodes	Extrenal iliac nodes	Sphincter complex	Ischiorectal fossa
cT3 high (above the peritoneal reflection)	+	+	+
cT3 mid-low (at the peritoneal reflection)	+	+	+	+		+ (when APR required)	+ (when direct tumor infiltration)
Any cT with massive positive internal iliac nodes	+	+	+	+		+ (when APR required)	+ (when direct tumor infiltration)
Any cT with massive positive obturator nodes nodes	+	+	+	+	+	+ (when APR required)	+ (when direct tumor infiltration)
cT4 with for anterior pelvic organ	+	+	+	+	+	+ (when APR required)	+ (when direct tumor infiltration)

APR, abdominal perineal resection.

The parameters analyzed were: Dice coefficient, defined as the volume of overlap between two sets of contours, divided by their mean volume (2│A∩B│/ │A│+│B│), was calculated to evaluate the overlap between the master contour (made by Reviewer) and each of the four delineation modalities made by Delineator (before the IC). A Dice coefficient = 0 means there was no overlap between the analyzed structure, and a Dice coefficient = 1 means there was total overlap [12,13].

Total time (defined as the time for Delineator needed for delineation + time for Reviewer for IC) was used to compare the efficiency of the system.

These two parameters were calculated for CTV and for each subsite. The Delinerator and Reviewer used different approaches. To reduce bias in contouring the different groups of the same patient in one working session, the Delineator had to contour different patients in one working session. The Delineator recorded the times required for the delineation of each contour and divided the four different sequences of delineation in four different groups labeled by capital letters so that Reviewer was blinded as to which group underwent IC. For groups B, C and D the Delineator chose the most similar case to the test patient from the library, based on demographic/anthropometric characteristics.The Reviewer performed the IC for each patient of all the four sequences (Groups A, B, C and D) during the same working session in order to have homologous corrections among the different sequences, while being blinded to the work done by Delineator in each group at time of IC. He documented the times for corrections of all the structures contoured by Delineator.

Statistical analyses were performed using MedCalc for Windows, version 9.5.0.0 (MedCalc Software, Mariakerke, Belgium). Dice coefficients and total time were compared using Wilcoxon test (paired samples).

Results

Clinical practice setting outcome (group A versus group B)

In the clinical practice setting the Dice coefficient was analyzed for CTV and for all the subsites (pelvic lymph nodes and OAR) separately. The mean Dice coefficient for CTV was 0.75 in group A and 0.70 in group B (p = 0.0195). The mean Dice coefficient for all the subsites was 0.68 in group A and 0.56 in group B (p = 0.0195) (Figure 2a and b). The median total time for contouring all the structures (including CTV) in group A was 38 minutes and 25 minutes in group B (p = 0.002), with a median difference in time of 13 minutes (total time needed for Delineator and Reviewer), (Table II).

Figure 2. Mean Dice Coefficient for CTV and all subsites, group A versus group B.

Table II. Dice coefficient, total time and p-value for CTV and all subsites (including CTV).

	CTV		All subsites (including CTV)
	Dice coefficient	Total time (Delineator + Reviewer) min	Dice coefficient	Total time (Delineator + Reviewer) min
Group A	0.75	16.5 (7.5 + 9)	0.68	38 (16.9 + 21.1)
Group B	0.7	12.2 (1.3 + 10.9)	0.56	25 (2.7 + 22.3)
p-value	0.0195	n.s	0.0195	0.002
Group C	0.75	27.6 (19.6 + 8)	0.715	73.1 (50.2 + 22.9)
Group D	0.75	15.5 (8.4 + 7.1)	0.71	37.9 (22.7 + 15.2)
p-value	n.s	0.005	n.s	0.002

Educational/training setting outcome (group C versus group D)

In the educational/training setting the Dice coefficient was analyzed for CTV and for all the subsites (pelvic lymph nodes and OAR) separately. The mean Dice coefficient for CTV was 0.75 in group C and in group D. The mean Dice coefficient for all the subsites was 0.715 in group C and 0.71 in group D (p = n.s.). The median total time for CTV delineation was 27.6 minutes in group C and 15.5 minutes in group D (p = 0.005). The median total time for contouring all the structures (including CTV) in group C was 73.1 minutes and 37.9 minutes in group D (p = 0.002), with a median difference in time of 35.2 minutes (total time needed for Delineator and Reviewer) (Table II). Dice coefficient analysis for CTV, all subsites, and OAR is shown in Table III. In summary, good reliability was found in CTV delineation among all the four different segmentation modalities, whereas it was less reliable in the small subsites (ON and PS). In group B the auto-segmentation showed good results in CTV delineation, mid in the mesorectum IIN and EIN and less reliable in ON, PS and bladder.

Table III. Overall median Dice coefficient for all the contoured structures, including organs at risk, in the different groups compared with the master contour. Dice coefficient > 0.7 white, > 0.6 < 0.7 light gray, < 0.6 dark gray.

		EIN	IIN	ON	M	PS	B	RFH	LFH
	No.	Median Dice coefficient
Group A	10	0.670	0.645	0.55	0.73	0.395	0.83	0.865	0.875
Group B	10	0.455	0.59	0.525	0.695	0.395	0.585	0.695	0.685
Group C	10	0.68	0.695	0.62	0.74	0.405	0.8	0.82	0.825
Group D	10	0.65	0.675	0.58	0.755	0.395	0.785	0.81	0.815

B, bladder; CTV, clinical target volume; EIN, external iliac lymph nodes; IIN, internal iliac lymph nodes, LFH, left femural head; M, mesorectum; ON, obturator lymph nodes; PS, presacral space; RFH, right femural head.

Discussion

Inconsistencies in contouring target and critical structures can undermine the precision of conformal radiation therapy. Delineation still remains the only procedure which is completely manual and therefore is a time consuming step in radiotherapy planning. Although site specific atlases for the pelvic region were published to aid in contouring pelvic CTV, inter- and intra-observer variability still remains high [3,4]. Few departmental QA programs have implemented IC for delineation. This could be due to the increased time and resources needed for the IC procedure. The introduction of auto-segmentation software is attractive for automatic delineation of different body sites and an increasing number of studies have tested their reliability and time savings [5–8]. Our departmental QA program was implemented in 2002 and includes IC for the delineation. Target volumes and OAR are delineated on CT images by a radiation oncologist and are revised by a second operator.

In the clinical practice setting we compared manual delineation versus auto-segmentation. Auto-segmentation showed acceptable reliability in the delineation of CTV and mesorectum, with a Dice coefficient of 0.7 and 0.695, respectively. It is questionable whether a Dice coefficient of 0.7 can be considered as a good overlap, however, this is the value recommended in the literature [8]. It should be emphasized that pelvic CTV and subsite targets are less clearly defined by a single tissue interface on CT images and therefore are more complex to delineate than head and neck where muscular fascia make it easier to identify the nodal target and the prostate where the anatomy of prostate target has a CT density which makes it distinguishable from adjacent structures. It is interesting to observe that the Delineator, a resident in Radiation Oncology, obtained a Dice coefficient of 0.75 when comparing his manual CTV contour with the master CTV contour. This value is similar to the one obtained with the automatic software (Dice coefficient = 0.70). Although a Dice value of 0.7 is recommended as a good overlap, we agree with the literature [8] that 0.7 is a questionable value.

The areas of largest disagreement in the delineation of CTV were the high-anterior and low mesorectum which are the most critical areas to delineate. The absence of the anterior mesorectum and mesorectal fascia made the high-anterior mesorectum difficult to identify. The inferior limit of the mesorectum showed high variability also in other reports probably due to the low contrast resolution on CT in images in the inferior pelvis where, at the level of levator ani muscles, the mesorectal fat disappears [4,14]. Auto-segmentation resulted in less acceptable reliability for all lymph node subsites (EIN; IIN; ON and PS) with the mean Dice coefficient ranging between 0.395 and 0.59. Of note, manual segmentation resulted in slightly better, however, non-significant improvement in EIN and IN (mean Dice 0.67 and 0.645). The ON and PS Dice coefficients were unacceptable (0.55 and 0.395) demonstrating the difficulty of both the manual and auto-segmentation procedures in identifying the anatomical boundary of these subsites on CT-images without contrast enhancement. Auto-segmentation is suboptimal in defining bladder because of the isodensity with small bowel (mean Dice coefficient = 0.585). However, the small number of patients in the library (two males and two females) may have limited the similarity between the atlas and test patients. Regardless, the delineation of subsites in the clinical practice setting is not recommended at the present time. In the clinical practice setting, where QA programs are routinely implemented for contouring, the use of SSKBC showed a statistically significant reduction of time required for all the delineated structures by 13 minutes. This accounts for approximately 35% of the total delineation time (p = 0.002) and reduces the total time needed for the contouring and IC of a single patient. This significant time reduction might lead, in daily practice, to an increased number of patients delineated per day and the possibility to perform an IC in centers with limited QA programs and resources.The importance of training in the contouring process has been previously investigated in the literature [15,16], however, according to our knowledge, this is the first study assessing the role of auto-segmentation system in the educational/training setting. When comparing Group C versus Group D, there was not a statistically significant difference for Dice coefficient. However, there was a statistically significant time saving in group D both in the delineation of CTV (12 minutes), and of all the structures (16 minutes) when considered together (p = 0.005 and p = 0.002, respectively). The longer time needed in group C, which was thought to be the shorter sequence, is due to the longer time needed by the Delineator for subvolumes delineation (30.6 minutes group C versus 14.3 minutes group D, p = 0.005), and the increased time needed by the Reviewer for IC (14.9 minutes group C versus 8.1 minutes group D, p = 0.019). This phenomenon could be explained by the fact that the low reliability of auto-segmentation made by SSKBC for small volumes led to an increased time for boundaries correction by the Delineator. Another possible explanation, for the shorter time needed in group D, could be due to the Delineator being less inclined to change his or her manual delineation. However, the software improved the delineation of the high, anterior and lateral boundaries of the larger structures (CTV and mesorectum). This resulted in more consistency with the master contour as delineated by the Reviewer on the CT scan in the 10 test patients, when qualitatively compared to the manual delineations. Therefore, in the educational/training setting, SSKBC may assist trainees to become familiar with contouring and quickly evaluate their work. Sequence used in group D was less time consuming when all subvolumes were considered together.

Two prospective studies are ongoing, with the implementation of a more extensive library and with a larger number of test patients. The first study aims to verify, in the clinical practice setting, the reliability of the software in auto-segmenting four different volumes: CTV, subvolume 1 (mesorectum + presacral space), subvolume 2a (IIN and ON) and subvolume 2b (IIN, EIN and ON). The second study is focused on staff who are in training and will identify which approach offers the best learning experience—atlas and auto-segmentation as tutorial versus atlas alone during their early experience in contouring.

Other studies published in recent years [17,18] investigated the use of deformable image registration to take into account daily organ motion during the course of radiotherapy. The implementation of auto-contouring programs and deformable image registration in the future could dramatically change the daily work flow of radiation oncologists and RTTs, however, further implementation and studies are needed.

Our results are comparable to those reported in literature, although a better Dice coefficient was found for head and neck cancer and breast cancer; confirming the difficulties in the definition of anatomical boundaries [6,8].

The use of SSKBC should be tested prospectively in clinical practice, with the goal of enhancing QA programs for delineation of locally advanced rectal cancer while reducing time and resource requirements. It could have a role in the educational/training setting tutoring junior staff and residents in ameliorating delineation skills.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.