Multi-institutional study of the variability in target delineation for six targets commonly treated with radiosurgery

Abstract

Background: Although accurate delineation of the target is a key factor of success in radiosurgery there are no consensus guidelines for target contouring.

Aim: The aim of the present study was therefore to quantify the variability in target delineation and discuss the potential clinical implications, for six targets regarded as common in stereotactic radiosurgery.

Material and methods: Twelve Gamma Knife centers participated in the study by contouring the targets and organs at risks and performing the treatment plans. Analysis of target delineation variability was based on metrics defined based on agreement volumes derived from overlapping structures following a previously developed method. The 50% agreement volume (AV₅₀), the common and the encompassing volumes as well as the Agreement Volume Index (AVI) were determined.

Results: Results showed that the lowest AVI (0.16) was found for one of the analyzed metastases (range of delineated volumes 1.27–3.33 cm³). AVI for the other two metastases was 0.62 and 0.37, respectively. Corresponding AVIs for the cavernous sinus meningioma, pituitary adenoma and vestibular schwannoma were 0.22, 0.37 and 0.50.

Conclusions: This study showed that the variability in the contouring was much higher than expected and therefore further work in standardizing the contouring practice in radiosurgery is warranted.

Introduction

Stereotactic radiosurgery (SRS) is a treatment technique known for its high capability of delivering highly conformal doses to the target while allowing normal tissue sparing due to the steep dose-fall off outside the prescribed dose volume. Due to the high conformality of the treatment delivery as a result of the rapid dose fall-off outside the prescribed dose, the outcome of the treatment is highly dependent on the clinician’s delineation of the target and any inaccuracy in target contouring could result in a potential lower probability of tumor control or increased normal tissue toxicity. In a recent study, a very high variability in target contouring was observed for two complex lesions in the brain, one case of cavernous sinus meningioma and one case of anaplastic astrocytoma [1]. Twenty centers using Leksell Gamma Knife^® for performing SRS provided contours and treatment plans for the two cases and large differences were found within these contours with respect to the size, location and shape of the target. These results indicate a possible impact on the treatment outcome. Other studies have reported on the variability of intracranial target contouring but the majority focused on fractionated radiotherapy and only few studies were dedicated to radiosurgery [2–7]. Methods of comparing multi-observer contouring data often rely on the analysis of overlapping structures [1,8–10]. These methods depend on the number of contours/segmentations analyzed and therefore set the limit on the possibility of inter-study comparison. In a systematic review of 119 publications on target volume delineation variability, a high contouring variability was observed throughout the available literature [11].

The published data on contouring variability for targets in the brain treated with SRS are however sparse.

The purpose of this study is to evaluate multi-center contouring data for six targets in the brain commonly treated with SRS using the methods and the metrics proposed by Sandström and coworkers [1] together with the evaluation of the position of the Center of Mass (COM). In addition, the potential clinical implications are discussed.

Material and methods

Clinical cases

Patient data consist of a cavernous sinus meningioma (case 1), pituitary adenoma (case 2), vestibular schwannoma (case 3) and three cases of metastases (cases 4-6). Axial T1-weighted MR images and axial contrast-enhanced T1-weighted MR images where available for all cases. Various other images were available for the cavernous sinus meningioma, pituitary adenoma and vestibular schwannoma. The three metastases were also imaged with Fluid Attenuated Inversion Recovery (FLAIR) MRI. The clinical information regarding all the six cases, the complete list of images that were available for them and some corresponding examples are given in the Supplementary Material and Supplementary Figures 1–6.

Data collection

All images for one particular case were co-registered and anonymized before distribution to 12 centers using Leksell Gamma Knife^® (LGK, Elekta Instrument AB, Stockholm, Sweden) for performing SRS. The data were sent as Leksell GammaPlan (LGP) files, easily imported to the treatment planning system (Leksell GammaPlan^®, Elekta Instrument AB, Stockholm, Sweden). The instructions given to the participants were to contour the target and OARs according to the local clinical practice and to perform one treatment plan for each case. The set of instructions was intentionally very relaxed because the aim of the study was to gather data that would reflect the current clinical practice at the participating centers and not the ability of the participants to follow a rigid set of instructions for target contouring. The skull contour was pre-specified before distribution of data.

The contours of the targets and OARs as well as the treatment plans made at the 12 participating centers were collected and further analyzed. Information about the participants including profession of planner, experience, planning methodology and general comments on the cases was also gathered.

Data analysis

The methodology used for analysis was previously described by Sandström et al. [1]. The target contours were exported from Leksell GammaPlan^® (LGP) as Digital Imaging and Communications in Medicine (DICOM)-objects and processed in MATLAB (Mathworks, Natick, MA, USA).

An illustration of the methodology to assess the variability in the target delineation is given in Figure 1 which shows examples of contoured structures (left panel) and the variability in target outlining in one slice (right panel) for all six analyzed cases.

Figure 1. Examples of contoured structures (left panel) and overlapping contours in one slice (right panel) for cavernous sinus meningioma (panels A and B), pituitary adenoma (panels C and D), vestibular schwannoma (panels E and F), large metastasis (panels G and H), medium metastasis (panels I and J) and small metastasis (panels K and L).

The 50% agreement volume, AV₅₀, the volume that 50% of participants agree on, was calculated for each of the six cases as described in Sandström et al. [1].

The intersect volume, AV₁₀₀, the volume that 100% of participants agree that belongs to the target, was also determined for each of the six cases. The union (encompassing) volume, denoted AV_100/N, of all volumes delineated by the participants for a particular target, was also determined.

An illustration of the particular methodology for determining AV₅₀, AV₁₀₀ and AV_100/N is shown in Figure 2 as the binary representation of the variability in one slice where black and white indicate complete or highest level of agreement in one slice, respectively. Levels of gray illustrate the different levels of agreement from N_i (complete agreement) to N_i⁻¹ (full disagreement) where N is number of contours for target i. N_i is thus related to AV₁₀₀ and N_i⁻¹ 1 to AV_100/N, the intersect and the union of the contoured structures, respectively.

Figure 2. One slice of the binary agreement matrix for cavernous sinus meningioma (A), pituitary adenoma (B), vestibular schwannoma (C), large metastasis (D), medium metastasis (E) and small metastasis (F).

A new metric, the Agreement Volume Index (AVI), was introduced as the ratio of volume AV₁₀₀ with AV_100/N which is a value between 0 and 1 and gives the level of agreement with respect to the volume, size and shape of contoured structures.

The position of the COM, the weighted center point of a volume, was also determined for each individual contoured structure. From the 3D coordinates indicating the position of the COM, the largest difference in X-, Y- or Z-direction was calculated.

The treatment plan evaluation parameters, target coverage, selectivity and gradient index, were also assessed for each of the targets. The treatment time was also calculated for each plan based on the same dose-rate. The treatment plans were also compared with respect to the differences in prescribed dose and in median prescribed dose.

Results

Twelve Gamma Knife centers participated in the study (from Greece, Norway, Czech Republic, Japan, South Korea, Canada, UK and USA). Thirty-two professionals were involved including 13 neurosurgeons (5–21 years of experience), 12 physicists (3–20 years of experience), three radiation oncologists (6.5–24 years of experience), three radiologists (7–11 years of experience) and one neuroradiologist (10 years of experience). A total of 71 treatment plans with contours were distributed back to be a part of the study; 12 plans for case 1, 13 plans for case 2, 12 plans for case 3, 10 plans for case 4, 12 plans for case 5 and 12 plans for case 6.

Image data sets used for contouring

The majority of participants contoured the cavernous sinus meningioma based on the contrast enhanced T1-weighted MRI (six out of 12) or the fused image of CT with contrast enhanced T1-weighted MRI (five out of 12) while one participating center used the fused image of CT with T2-weighted MRI for contouring the cavernous sinus meningioma. The largest variability with respect to the choice of image on which the target was delineated was observed for the pituitary adenoma. For the delineation of the vestibular schwannoma most of the participating centers preferred the contrast enhanced axial T1-weighted image (10 out of 12). For the delineation of the metastases the highest level of agreement was reached regarding the choice of the preferred image: they all used the axial contrast enhanced T1-weighted MRI. The summary of the analysis of the preferred images used for contouring the targets is shown in the Supplementary Material, Table 1.

Analysis of the contouring variability

The range of contoured volumes together with the respective 50% agreement volume, AV₅₀, the common (intersect) volume, AV₁₀₀, the encompassing (union), AV_100/N, as well as the newly introduced AVI are summarized in Table 1. The explicit values of the contoured volumes for all six targets are displayed in Figure 7 in the Supplementary Material. The larger volume and consequently the largest range of delineated volumes were observed for one of the metastases (10.30–14.55 cm³). For that case, the AV₅₀ is the lowest relative to the range of delineated volumes, only 3.90 cm³, indicating a high variability not only in the actual size of the structure but also in its position and shape.

Table 1. Summary of the results including the range of contoured volumes, AV₅₀, AV_100/N, AV₁₀₀, the Agreement Volume Index (AVI).

	Range of contoured volumes (cm^3)	AV50 (cm^3)	AV100/N (cm^3)	AV100 (cm^3)	AVI
Cavernous sinus meningioma	5.29–7.80	5.52	11.06	2.48	0.22
Pituitary adenoma	1.67–2.15	1.80	2.98	1.09	0.37
Vestibular schwannoma	3.56–4.48	3.90	5.40	2.68	0.50
Large metastasis	10.30–14.55	3.90	15.32	9.46	0.62
Medium metastasis	1.27–3.33	2.94	4.29	0.68	0.16
Small metastasis	1.42–2.26	1.51	2.75	1.01	0.37

Figure 3 shows 3D illustrations of the AV₁₀₀ together with AV_100/N as well as the position of COMs for all the delineated volumes. The graphical illustrations in Figure 3 add to the information in Table 1 by providing an indication not only regarding the size of the volumes but also their relative displacement. For all six cases, there is a substantial difference between the volume contoured by the all participants (the blue 3D structure) and the encompassing volume resulting from the union of all delineated structures (the green 3D structure), indicated also quantitatively by the low values of AVI. The lowest AVI, 0.16, corresponds to one of the metastases, the one medium in size, while the highest AVI, 0.62, corresponds to the largest metastasis. The fact that the positions of the COMs for the delineated volumes do not coincide show that the volumes are not only different in size but also in shape and the location.

Figure 3. Illustration of the common volume, AV₁₀₀ (blue) and the encompassing volume, AV_100/N as 3-dimensional surfaces together with the positions of COM (red dots) for cavernous sinus meningioma (A), pituitary adenoma (B), vestibular schwannoma (C), large metastasis (D), medium metastasis (E) and small metastasis (F).

The largest distance between COM positions in the three Cartesian directions is shown in Table 2 in the Supplementary Material. The largest deviation between the positions of the COM for the contoured structures was observed in Z-direction (cranio-caudal) for four of the cases (cavernous sinus meningioma, pituitary adenoma and two of the metastases) while in the other two cases (vestibular schwannoma and the medium metastasis) the largest deviation was observed in the X-direction (anterior–posterior).

The results regarding the treatment planning specific parameters are presented in the Supplementary Material and in Supplementary Table 3.

Discussion

Accurate definition of the target is very important in radiosurgery and stereotactic radiotherapy treatments like Gamma Knife since the target is planned without margins and the dose distribution is highly conformal to the delineated target. Variability in contouring of targets in radiotherapy has been reported across numerous sites and tumor types [4,7,8,11–22] but studies concerning brain SRS contouring are relatively scarce [5,9,10], despite potentially greater impact of contouring accuracy on treatment outcomes for this high dose highly conformal treatment. A high contouring variability was observed for 14 metastases delineated by eight physicians for stereotactic radiosurgery, suggesting that counteracting methods should be adopted to reduce variations in target contouring [23]. This suggestion was further supported in a study where an automatic segmentation tool was evaluated for brain and head-and-neck OAR [24]. Clinically accepted segmentations of the relevant OARs were obtained, including the brainstem and optic chiasm. Furthermore, another study concluded that target definition is one major source of uncertainty in radiotherapy which emphasizes the need to use multiple imaging modalities, such as co-registration of CT with MRI or functional imaging such as positron emission tomography (PET), together with education and collaboration across specialities as possible ways to reduce the contouring variability [25]. A review of the literature on target volume delineation across numerous tumor sites, concluded that interobserver variability in target volume contouring constitutes the largest source of uncertainty for many of the tumor locations evaluated [26].

The current study adds to the previous observations reporting not only a qualitative assessment of the variability in target delineation, as the previous studied did, but also the quantitative difference and the potential clinical implications related to different prescription doses depending on the size of the delineated target and the proximity to the OARs and also the selectivity of the plans.

With no margins applied to a contoured target, as in SRS, the differences in contouring practices have a greater potential clinical impact in terms of reduced tumor control probability and increased normal tissue toxicity. Considerable variations in target contouring due to accounting differently for the extent of tumor or tumor infiltration may result in large variations in the dose conformity, tumor control probability and normal tissue toxicity. A treatment plan with high conformity might be superior with respect to normal tissue toxicity but on the other hand result in a lower tumor control probability. On the other hand, a treatment plan with margins allowing for higher tumor control probability will have a lower conformity leading to increased probability of normal tissue toxicity [27]. Furthermore, as SRS techniques transition towards fractionated treatment regimes, accuracy in target contouring is especially important to lower the contouring inconsistency between fractions. Therefore meaningful comparison of outcomes across radiosurgery studies depends on consistent target delineation in addition to consistent nomenclature and description of treatment parameters.

In this multi-centered study evaluating clinical contouring and SRS planning variations across six common SRS targets, considerable differences were observed in the size, shape and position of the delineated targets and notable differences in target coverage and selectivity. These results highlight the need for consensus and guidelines for target contouring in SRS to enable meaningful clinical research collaboration in order to move the field forward.

Acknowledgments

The cooperation of all participants involved in this study is gratefully acknowledged: Department of Stereotactic and Radiation Neurosurgery and Department of Medical Physics at Na Homolce Hospital in Prague, Czech Republic, Ian Paddick MSc and Tim Cox MD at Cromwell Gamma Knife Centre in London, England, Katherine Hunt BEng MSc MIPEM, Ruth Batty MD PhD FRCR, Dan Connolly MD PhD FRCR, John Yianni MD FRCS(SN) and Jeremy Rowe MD MA DM FRCS(SN) of the National Centre for Stereotactic Radiosurgery at Royal Hallamshire Hospital in Sheffield, England, Professor Lijun Ma and Professor Penny Sneed at the UCSF Gamma Knife Center in San Francisco, USA, Taylor McAdam Vell Neuroscience Institute at Washington Hospital in Fremont California, USA, Francisco Li MSc RSO at the Swedish Radiosurgery Center in Seattle, USA, Hyun-Tai Chung PhD at the Department of Neurosurgery Seoul National University College of Medicine in Seoul, Korea, Monique van Prooijen PhD at the Princess Margaret Cancer Centre in Toronto, Canada, Chryssa Paraskevopoulou MSc of Hygeia Hospital in Athens, Greece, Bente Sandvei Skeie MD PhD and Jan Heggdal MSc at the Department of Oncology and Medical Physics at Haukeland University Hospital in Bergen, Norway, David Schlesinger PhD, David Larson MD and Kevin Orcutt MD from the Department of Radiation Oncology at the University of Virginia in Charlottesville Virginia, USA. Technical assistance of Björn Somell at Elekta Instrument AB in Stockholm is also gratefully acknowledged.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

Financial support from the Cancer Research Funds of Radiumhemmet is also gratefully acknowledged.