Use of auto-segmentation in the delineation of target volumes and organs at risk in head and neck

Abstract

Background: Manual delineation of structures in head and neck cancers is an extremely time-consuming and labor-intensive procedure. With centers worldwide moving towards the use of intensity-modulated radiotherapy and adaptive radiotherapy, there is a need to explore and analyze auto-segmentation (AS) software, in the search for a faster yet accurate method of structure delineation. Material and methods: A search for studies published after 2005 comparing AS and manual delineation in contouring organ at risks (OARs) and target volume for head and neck patients was conducted. The reviewed results were then categorized into arguments proposing and opposing the review title. Results: Ten studies were reviewed and derived results were assessed in terms of delineation time-saving ability and extent of delineation accuracy. The influence of other external factors (observer variability, AS strategies adopted and stage of disease) were also considered. Results were conflicting with some studies demonstrating great potential in replacing manual delineation whereas other studies illustrated otherwise. Six of 10 studies investigated time saving; the largest time saving reported being 59%. However, one study found that additional time of 15.7% was required for AS. Four studies reported AS contours to be between ‘reasonably good’ and ‘better quality’ than the clinically used contours. Remaining studies cited lack of contrast, AS strategy used and the need for physician intervention as limitations in the standardized use of AS.

Discussion: The studies demonstrated significant potential of AS as a useful delineation tool in contouring target volumes and OARs in head and neck cancers. However, it is evident that AS cannot totally replace manual delineation in contouring some structures in the head and neck and cannot be used independently without human intervention. It is also emphasized that delineation studies should be conducted locally so as to evaluate the true value of AS in head and neck cancers in a specific center.

With the development of intensity-modulated radiation therapy (IMRT), a modern radiation therapy (RT) treatment method which enables the delivery of high precision therapeutic dose to the tumor, treatment of head and neck cancers (HNC) has improved [1–8]. IMRT allows greater dose distribution control, hence allowing significant sparing of organs at risk (OARs), such as parotid glands, while maintaining good target volume (TV) coverage [2,4–9]. Dose escalation is also made possible for tumors located near critical structures [1]. With such clinical benefits, IMRT is now considered the standard technique in RT for HNC [4,6,10–12].

To fully exploit the advantages of IMRT, all TVs and OARs must be accurately contoured before treatment planning. In addition, due to steep dose gradients outside the planning target volume (PTV), the accuracy of delineation is becoming of greater importance [5–7,10]. ICRU 83 [13] reported that organs or structures that are not delineated in IMRT can receive significant radiation absorbed doses. Typically, they will be contoured manually by either the radiation oncologist or planner and this tedious process takes an average of 2.7–3 h for HNC [1–6,9,14–17]. Often, this time-consuming process may be repeated multiple times during a treatment course because of tumor response leading to changes in tumor shape and size or change in patient’s weight or anatomy [1,16,18]. As centers worldwide are moving towards adoption of adaptive RT with the intention of sparing normal tissue and reducing toxicity to patients, exploring and analyzing automatic segmentation (AS) software in delineation is essential for HNC. The aim of this review is to assess the potential of AS to be adopted as the standard for contouring OARs and TVs for HNC patients.

Auto-segmentation systems

Atlas-based auto-segmentation (ABAS) refers to the propagation of segmented structures from atlas images onto a patient image data set using deformable image registration (DIR). The atlas can be based upon a single patient dataset (Single-ABAS), averaged patient population or multiple patient data (Multi-ABAS) [6,16]. Single-ABAS is limited as there is often a large discrepancy between organ volume and location from the atlas to the patient data. This can be overcome using a Multi-ABAS approach, although large libraries of atlases can also suffer from ‘clusters’ of similar anatomy that may be dramatically different from that of the patient anatomy, as discussed by Fritscher et al. [15]. It has been reported in brain imaging that the optimum number of atlases in Multi-ABAS is 20 [19]. Multi-ABAS is also susceptible to topological errors as it makes use of a ‘voting scheme’ that determines whether a voxel is inside or outside the structure [20] and has also been found to result in poorer accuracy than manual delineation for small cranial structures [21].

In model-based auto-segmentation (MBAS), the algorithm recalls the characteristic shapes for the structures from a training set of images that have been previously manually delineated. During MBAS, the best model for each structure is selected to yield the final delineation [15]. A limitation of MBAS, however, is that it can be inflexible as segmentation is limited to the specific shapes described by the statistical model [20].

A combination of both ABAS and MBAS methods is known as hybrid auto-segmentation (HAS) which combines the positive attributes of both methods and has been shown to improve nodal delineation in the head and neck, relative to other methods [4,22]. The AS strategies used in the studies in this review can be found in Table 1.

Material and methods

Search strategy for identification of studies

To ensure that the maximum number of studies in AS planning was captured, a systematic approach was used to search and select relevant publications. The approach used consisted of firstly searching four databases: (1) Excerpta Medica Database (EMBASE); (2) PubMed; (3) Science Direct; and (4) Google Scholar – for journal articles and reviews published between January 2005 and December 2015. This specific timeframe was chosen to capture the earliest known reference to AS in the health literature. The primary search targeted English articles. The following search terms were used:

1. Auto-segmentation AND (Contouring OR organ-at-risks OR target OR head and neck patients);

2. Same search terms limited to 10 years.

After this initial search, review of titles and abstracts was conducted on each resulting citation so as to avoid duplicated results. For those where only the abstract was available, attempts to search for the full study were conducted through Trinity College Dublin library and National University of Singapore library. After reviewing, eligible studies were further assessed based on relevancy, quality and content (Table 2). Review of the reference lists of the eligible articles was conducted to identify any other potential additional papers for inclusion. Data extraction and evaluation were conducted on the eligible articles (Table 1 and Supplementary Tables 1 and 2). Supplementary Figure 1 outlines the search strategy.

Table 1. Study summary table.

Study title	Year	AS used	Number of physicist involved	Disease-related factor	Number of cases	Contouring structures	Time savings assessed?	Delineation accuracy assessed?	AS strategy assessed?	Physicists influence assessed?	Patient staging assessed?	Observer variability assessed?	Study quality
Clinical validation of ABAS of multiple target volumes and normal tissue (swallowing/mastication) structures in the head and neck. Teguh et al. [14]	2011	Multi- &Single-ABAS	5 experienced observers + expert panel (4 staff members)	HNC	12	TV Neck levels I–IV & 20 OARs	Yes	Yes	Yes	No	No	No	Poor (0)
ABAS of head and neck OARs and nodal target volumes: a clinical validation. Daisne et al. [6]	2013	ABAS	1 radiation oncologist	Un-operated HNC	20	OARs & CTV	Yes	Yes	Yes	No	No	Yes	Good (4)
AS of low-risk CTV for head and neck radiation therapy. Yang et al. [3]	2014	Multi- & Single-ABAS	8 radiation oncologists	HNC (Unilateral tonsil cancer)	16	Low-risk CTV	No	Yes	Yes	Yes	Yes	Yes	Poor (0)
AS of normal and target structures in head and neck CT images: A feature-driven model-based approach. Qazi et al. [2]	2011	Hybrid approach	1 observer	HNC	10	OAR & TV (lymph nodes level I–IV)	No	Yes	Yes	No	No	No	Poor (1)
Evaluation of automatic atlas-based lymph node segmentation for HNC. Stapleford et al. [1]	2010	Multi- & Single-ABAS	5 radiation oncologists	HNC (Oropharyn x and nasopharyn x)	5	TV of lymph node regions	Yes	Yes	No	No	No	Yes	Poor-Fair (2)
Does ABAS of neck levels require subsequent manual contour editing to avoid risk of severe target under-dosage? A dosimetric analysis. Voet et al. [5]	2011	Single-ABAS	2 radiation oncologists	HNC	9	OARs & TV	No	Yes	No	No	No	No	Poor-Fair (2)
Dosimetric evaluation of AS for adaptive IMRT for HNC. Tsuji et al. [11]	2010	Unknown	1 planner and 1 radiation oncologist	HNC	16	CTV	No	Yes	No	No	No	No	Poor (1)
Prospective randomized double-blind study of atlas-based OAR AS-assisted radiation planning in HNC. Walker et al. [23]	2014	Unknown	15 radiation oncologists with 1–4 years of experience	HNC	40	16 OARs	Yes	Yes	No	Yes	No	No	Good (4)
Systematic evaluation of three difference commercial software solutions for AS for adaptive therapy in head and neck prostate and pleural cancer. La Macchia et al. [25]	2012	Single-ABAS	3 radiation oncologists	– HNC – Prostate – Pleura	5 from each site Total: 15	All volumes of interests	Yes	No	No	No	No	No	Poor (0)
Evaluation of an AS algorithm for definition of head and neck OARs. Thomson et al. [24]	2014	SPICE	5 radiation oncologists	HNC	10	5 OARs	Yes	Yes	No	No	No	No	Poor (0)

ABAS: atlas-based automatic segmentation; AS: automatic segmentation; CT: computed tomography; CTV: clinical target volume; HNC: head and neck cancer; IMRT: intensity-modulated radiation therapy; OAR: organ at risk; SPICE: smart probabilistic image contouring engine; TV: target volume.

Table 2. Summary of inclusion and exclusion criteria.

Inclusion

Exclusion

All studies published after 2005 Adult (≥18 years) populations of patients with a histologically proven head and neck cancer regardless of staging and treatment intent English language publications Publications investigating auto-segmentation delineation in targets and/or OARs for head and neck patients planned for IMRT treatment Studies investigating at least one of the following outcomes: time savings and delineation accuracy

Studies published before 2005 Studies investigating patients <18 years of age with other cancers beside head and neck cancers Studies describing the technical development of auto-segmentation software Studies discussing about auto-segmentation delineation solely for prostate, breast, lung or other cancers and not head and neck cancers (due to difference in information requirements) Studies assessing physician or healthcare opinion Studies assessing patients who were planned for conventional two-dimensional radiotherapy, three-dimensional conformal radiotherapy as well as pediatric head and neck cancer patients were excluded

Table 3. Quality of study assessment using Jadad et al. [26] (Assessing the quality of reports of randomized clinical trials: is blinding necessary?).

Study	Year	Was the study described as randomized? (1 = yes; 0 = no)	Was the method of randomization well described and adequate? (0 = not described; 1 = described and adequate; −1 = described, but inadequate)	Was the study described as double-blind? (1 = yes; 0 = no)	Was the method of double blinding well described and adequate? (0 = not described; 1 = describe and adequate; −1 = described, but not adequate)	Was there a description of Yes, withdrawals and dropouts sufficient to determine the number of patients in each treatment group entering and completing the trial? (1 = yes; 0 = no)	Final score / quality of study (0 = Bad, 1 = Poor, 2 = Fairly Poor, 3 = Fair, 4 = Good, 5 = Very good)
Teguh et al. [14]	2011	0	0	0	0	0	0
Daisne et al. [6]	2013	1	1	1	1	0	4
Yang et al. [3]	2014	1	0	0	0	0	0
Qazi et al. [2]	2011	0	0	0	0	1	1
Stapleford et al. [1]	2010	0	0	1	1	0	2
Voet et al. [5]	2011	1	1	0	0	0	2
Tsuji et al. [11]	2010	0	0	0	0	1	1
Walker et al. [23]	2014	1	1	1	1	0	4
La Macchia et al. [25]	2012	0	0	0	0	0	0
Thomson et al. [24]	2014	0	0	0	0	0	0

Articles published before 2005 were excluded to increase the relevance of this research to 21st century practice. However, these studies served as a foundation to contextualize and explain observed results. Selected studies were English language publications comparing AS to manual delineation (MD) in HNC planned for IMRT treatment.

Participants of included studies were aged ≥18 years with a histologically proven HNC. Disease could be localized or advanced, and the patients were planned to be treated with RT adjuvantly or definitively and planned for IMRT. All HNC sites and histological types were eligible. Patients who were planned for conventional two-dimensional (2D) RT, 3D conformal radiotherapy (CRT) and pediatric HNC patients were excluded.

Using a range of measuring tools, included publications reported a mixture of different outcomes. For this review, the outcomes of time saving and delineation accuracy were primarily explored. Second, the influence of other external factors (observer variability, AS strategies adopted and stage of disease) were also investigated.

Statistical analysis

All 10 studies analyzed the time-saving ability of AS by comparing the recorded total resident segmentation and correction time using AS to the MD time recorded. Delineation accuracy was measured using a range of statistical tools (Supplementary Table 3). Dice similarity coefficient (DSC) was the most commonly used tool amongst the 10 reviewed studies to evaluate the similarity between AS and MD. DSC measures the spatial overlap between structures to a value between 0 and 1, where 0 means no overlap and 1 indicates perfect agreement.

Results

The search strategy identified 794 references (titles and/or abstracts) for review across the searched databases, of which 454 remained after removal of duplicates. Following this, abstract screening was conducted and 423 articles were disqualified as they were regarded as irrelevant to the review. The remaining 31 papers were reviewed in-depth resulting in a total of 10 studies [1–3,5,6,11,14,23–25] meeting the eligibility criteria (Table 2 and Supplementary Figure 1). Review of the 10 studies did not yield additional papers for inclusion. Majority of the eligible studies were relatively good quality studies, according to Jadad et al. [26] assesment (Table 3) using similar measuring instruments, but yielding different outcomes and results. Relevant data were extracted from the eligible studies and presented accordingly as outcomes of using AS on HNC patients (Supplementary Table 1) and factors affecting the outcomes (Supplementary Table 2). As objectives, methods and presentation of results vary across the included studies, findings are synthesized in a qualitative manner to address the efficacy of AS compared to MD in terms of time-saving ability and delineation accuracy. Due to limited studies investigating AS in HNC, the majority of the studies were retrospective studies and systematic reviews.

Time saving

Six of 10 studies investigated time-savings outcomes [1,6,14,23–25]. Five studies demonstrated significant time savings in delineation of target and OARs in HNC [1,6,14,23,25]. La Macchia et al. [25] reported that use of AS resulted in substantial time reduction of an hour. Similarly, the four other studies by Daisne et al. [6], Stapleford et al. [1], Walker et al. [23] and Teugh et al. [14] also reported significant time savings of 40%, 35%, 30.9% and 59%, respectively, whereas one study [24] concluded that the use of AS compared with MD did not result in time saving but required 15.7% more time.

Delineation accuracy

Nine studies analyzed delineation accuracy in targets and OARs in HNC [1–3,5,6,11,14,23,24]. Two of the studies [2,14] reported high accuracy in AS contours. Teugh et al. [14] reported that an expert panel subjectively scored all OARs in the study as ‘minor deviation, editable’ or better with excellent agreement based on DSC. Similarly, Qazi et al. [2] also reported that AS contours were similar or better quality than the clinically used contours. Tsuji et al. [11] reported that unedited AS contours of normal structures may be used without negatively affecting plan quality. Yang et al. [3] also reported that AS yielded reasonably good segmentation of the low-risk clinical target volume (CTV) with the need for minor modifications.

Three studies [5,23,24] reported that lack of clarity of contrast between structures and tissue lead to delineation inaccuracy hence under- and over-dosage of TVs and OARs, respectively. Voet et al. [5] reported that CTV accuracy was not clinically accepted as they tended to be smaller than those manually delineated, resulting in large under-dosages. Another study by Walker et al. [23] reported clinically unacceptable AS for several critical OARs, such as the optic chiasm, which resulted in inadvertent over-dosage leading to blindness. Thomson et al. [24] reported the AS contours were less accurate and inferior for all investigated structures, apart from the parotid and submandibular glands.

Two studies [1,11] clearly demonstrated the acceptance of AS as a structure delineation tool but emphasized the importance of physician intervention. Tsuji et al. [11] stated that AS is not robust enough to replace physician-drawn contours in TV definition. Similarly, Stapleford et al. [1] also reported that 68% of the physicians deemed AS contours as unacceptable with the indication that the AS CTVs were too large. Three of nine studies [1,3,6] agreed upon the importance of human supervision and modification of AS contours before proceeding with planning for patient’s safety.

Only three of 10 studies [1,3,6] assessed the observer variability, which plays an important role in target and OARs delineation and patient’s outcome. Two studies [1,3,] reported that AS is able to mitigate the potential of observer variability during structure delineation. However, one study [6] reported the existence of inherent intra-observer even with AS, indicated by the high value of average and maximal surface distance.

Three of 10 studies [2,3,14] concluded that delineation accuracy is affected by AS strategies used. Multi-ABAS is preferred over Single-ABAS as it results in better segmentation, resembling manually delineated contours while mitigating observer variability.

Only one study [3] investigated tumor stage for the trial cases used in the study and concluded that tumor stage might affect the delineation accuracy determined by the atlas selection in extreme cases. T4a-staged HNC tumor with aggressive tumor invasion and substantially different CTV shape has a relatively low DSC as compared to T1–T3-staged HNC tumors [3]. In these cases, MD should be considered due to the high possibility of AS delineation inaccuracy.

Tsjui et al. [11] reported that AS will be similarly as useful for post-operative OAR delineation as long as the neck extension is similar. They also reported a lower DSC for spinal cord due to greater neck extension at mid-treatment computed tomography (CT). This low DSC indicated that AS contours are different from manually delineated contours. However, AS might not be suitable for post-operative TV delineation. Mukesh et al. [27] stated that in these instances, there will be a change in patient anatomy, introducing difficulties for post-operative CTV segmentation. Furthermore, Rasch et al. [28] found significant inter-observer variation in post-operative CTV segmentation in patients with paranasal sinus tumors, especially in the anterior border of the nasal cavity, possibly due to lack of anatomical landmarks to define the edge of the volume. As the dose gradients are steep around TVs and their dose specifications more stringent compared with OARs, the threshold accuracy for TVs is presumably higher than for normal tissues [11].

Discussion

Time saving

Upon reviewing the publications, the majority investigating time saving reported substantial time reduction of 30–60% using AS compared to MD [1,6,14,23,25]. These results are consistent with earlier studies [17,29–31]. Hu et al. [31] reported significant time reduction of 87% for overall structure delineation in oropharynx and nasopharynx cancers. Similar time-saving ability of AS has also been observed in other sites [32–34]. Moreover, the observed significant time-saving results are presented in percentage terms hence enabling them to be used for study comparison as they are independent of delineated structures, type of AS software and cancer staging. In addition, the observed results of significant time saving were achieved despite taking into consideration the time needed to modify the AS contours.

Considering these results alone, AS appears to have advantages over MD in terms of time-saving ability. However, two of the studies [1,23] favoring AS investigated time saving in either TVs or OARs, but not both. Furthermore, studies reported that not all HNC cases are eligible for AS, which resulted in more time wasted trying to use the available AS software to no avail [5,6]. Cases with insufficient image content, such as low tissue contrast in HNC CT images, lack of spontaneous contrast between anatomical key structures and image artifacts, such as dental implants, can lead to large registration discrepancy with AS, hence resulting in more modification time needed [5,6]. Thomson et al. [24] reported that the use of AS compared with MD required 15.7% more time instead. Daisne et al. [6] also reported that little or even no time savings were noted for certain nodal CTVs. For example, retropharyngeal nodes saved an average of 0.8 minutes while retrostyloid needed an average of 0.3 minutes more time due to the lack of clear contrast between the retropharyngeal fatty layer and prevertebral muscles and anatomical intimacy between retropharyngeal nodes and retrostyloid, respectively [6]. This indicates that TVs with greater complexity appear to be more time consuming than standard cases. Daisne et al. [6] also noted that cases where iodine contrast uptake was limited resulted in a poorer result using AS methods; of note the reference atlas contained images obtained using intravenous iodine contrast.

When analyzing time-saving ability between AS and MD, it is also important to consider the experience of the observer. Only one study [23] stated that the physicians involved in the study had 1–4 years of residency experience with average time savings per resident varying from 2.3 to 15.7 min. It was observed that residents with 3–4 years of experience tended to take shorter MD time as compared to those with 1–2 years of experience. However, Daisne et al. [6] concluded that any amount of time saved has substantial impact on the daily clinical load as a department is then able to channel the time saved into important tasks, such as reviewing medical files. Therefore, it may be inappropriate to determine the value of time saving of AS for a particular hospital until studies are conducted locally. Five of six studies [1,6,14,24,25] investigating time savings had small sample sizes varying from 5 to 20, which are not statistically powerful enough to construct a clear conclusion regarding time-saving ability of AS.

Delineation accuracy

Extent of tumor control and side effects experienced by patients are highly correlated with the level of structure delineation accuracy [8,23]. Hence, accuracy is the most important factor when considering the best method of delineation. Two of nine studies investigating delineation accuracy [2,14] reported high delineation accuracy in AS contours. Promising results in which all autocontours were scored as ‘minor deviation editable’ or better quality than the clinically used contours were observed [2,14]. Unedited autocontours of normal structures were reported to be usable without negatively affecting plan quality by Tsuji et al. [11]. In two studies by Levendag et al. [29,30] which also investigated the delineation accuracy of HNC, they concluded that similar or more accurate delineations of TVs were observed with AS. Similar delineation accuracy is reported in other sites as well [32,35]. A study by Anders et al. [35] evaluated the performance of ABAS for contouring clinically relevant CTVs for the breast and anorectal cancer and reported satisfactory delineation accuracy results for these clinical TVs. Based on these results, ABAS has been clinically introduced for precontouring of CTVs and OARs [35]. The study [35] also reported that HNC TVs are less complex and more clearly defined than are breast and anorectal cancer TVs. In addition, Yang et al. [3] concluded that the stage of the disease will not affect the delineation accuracy as the latter is determined by the atlas selection except for extreme cases. Such studies endorse the potential for AS to replace MD in structure delineation for HNC.

Nevertheless, differences in structures delineated in the studies must be considered. A possible explanation of the promising results derived may be that the structures chosen to be delineated in those studies have clear contrast between the anatomical key structures. For example, all three studies [2,11,14] delineated spinal cord and parotid glands and these have previously been reported as being delineated with good quality [5,6,23,24]. However, the same studies [2,11,14] did not delineate the larynx that was reported in two other studies [23,24] to be auto-delineated in an inferior and clinically unacceptable manner, which could result in inadvertent over-dosage causing aspiration or dysphagia. In earlier literature, Isambert et al. [21] assessed AS delineation of brain OARs and obtained good results for large structures and less accurate results for smaller structures. This is consistent with the observation that different structures, due to their anatomical variation, will have different delineation accuracy. Clearly, a lack of clarity of contrast between structures and tissues leads to delineation inaccuracy; hence under- and over-dosage of TVs and OARs, respectively [5,23,24].

Yang et al. [3] also reported that due to TV typically not having constant image intensity or clear anatomic boundaries, there is a high potential of under-dosage endangering patient outcome and increasing likelihood of recurrence. Tsuji et al. [11] observed that AS plans were not just inferior to the manual plans in target coverage and conformity; they were unacceptable with target coverage of 90% for both the GTV and CTV. Similarly, two studies [1,5] reported that CTV delineation accuracy was not clinically acceptable due to the size differences from the MD, resulting in large under-dosages to TV and over-dosages to OARs. However, Yang et al. [3] reported a reasonably good agreement between the AS contours and manual contours for CTV delineation by analyzing the total volume overlap and median 2D Hausdorff distances. Moreover, it was reported that most of the local discrepancy between AS and manual plans comes from observer variability in delineating low-contrast objects, such as defining the CTV on the low-contrast lymph nodes [3]. Thus the true delineation accuracy of AS structures remains ambiguous at present. However, with the advent of improved similarity matrices and further improvement of segmentation algorithms in the future, accuracy of AS and indeed the measurement of that accuracy are likely to improve.

In addition, AS software choice should be considered as it may influence the delineation results [3,5,6,14]. For example, structures located close to the local bony anatomy (i.e. larynx) can be affected by the range of available atlas cases [36]. Upon analysis, two studies [2,14] concluded that Multi-ABAS is preferred. Teguh et al. [14] and Yang et al. [3] further illustrated that Multi-ABAS yielded good segmentation resembling MD contours and mitigating observer variability while Single-ABAS is subject to observer variability resulting in large variations across the autocontours. Reduction of observer variability from using Multi-ABAS has also been demonstrated in other sites [22,32,37]. In a retrospective study that investigated the validation and benefit of Multi-ABAS for lung cancer, Yang et al. [32] found that Multi-ABAS was able to reduce inter-subject variability and emphasized the importance of observer variation affecting the delineation of low-contrast small structures. However, regardless of delineation methods, observer variability still remains as one of the most challenging issues in the IMRT era [1,8]. Daisne et al. [6] reported the existence of inherent intra-observer variation even with AS. Similar results were observed when analyzing observer variation in MD as well [38]. However, due to limited studies [3, 14] analyzing AS strategies used for delineating HNC specifically, it is difficult to give a concrete conclusion as to the extent of the impact that AS strategies have on delineation accuracy.

The majority of studies [1,3,5,6,11,23] agreed upon the importance of human supervision and modification of autocontours before proceeding with planning for patient safety. Voet et al. [5] focused on the question of the necessity of editing ABAS-produced autocontours of TVs and OARs in the HNC and concluded that modification of autocontours is essential. Similar suggestions are given in other studies that investigated HNC as well as other sites [32,39]. Moreover, human modification of TVs is proven to be essential for adaptive cases [11]. Tsuji et al. [11] reported that the GTV size was significantly different between manual and automatic contours as for the manual contours, the GTV was deliberately copied to the mid-treatment CT to avoid a cone-down.

Two major concerns that should be addressed are the accuracy of MDs to which AS delineations are being compared and the reporting metrics of accuracy. The majority of studies [1,3,6,11,14,23–25] in this review compare their AS delineations with a ‘gold standard’ MD. In some cases, this ‘gold standard’ is a MD by a single radiation oncologist (RO) [1,3,6,24,25]; others where consensus has been reached by an expert panel of ROs [14,23] and finally where consensus has been reached by both RO and planner [11]. The latter has been recommended by Njeh et al. [40] as well as endorsed by others [41]. It is important to keep in mind that the ‘gold standard’ may itself be the subject of considerable observer variability.

Coupled with this, the majority of studies in this review used the DSC to measure the accuracy of the AS delineations relative to the MD. Use of a simple single overlap metric is not necessarily an indicator that target dose coverage will be sufficient. Voet et al. [5] found that despite having what would be perceived as a high DSC of 0.8 between the AS and MD PTV, an under-dosage of up to 11 Gy to the PTV was observed with the AS PTV. Tsuji et al. [11] found similar limitations with DSC and suggested that sensitivity index might be a better dosimetric predictor than DSC.

Despite these limitations, there is no doubt that AS has a potential critical role in the future of RT with the demand for high-throughput image segmentation within the field increasing in the era of adaptive RT. However, local delineation studies have to be carefully conducted to evaluate the potential benefits for each department in daily clinical practice. Time is also required to further develop the software and to improve on accuracy reporting mechanisms. Building up clinical evidence to construct reliable predictive models so as to ensure a shared ontology will be time consuming and challenging, yet necessary so as to improve the reproducibility and accuracy of using AS software in daily practice. As recommended by Valentini et al. [42], this ‘gold structure set’ will thus represent the unique benchmark of the study and will be the referral contour to which all other contours should be compared to. Even with the implementation of AS software in the future, it should be reinforced that manual editing is still a necessity for patient safety. To promote acceptance of AS software as well as to ensure patient safety clinically, user education will play an important role. It is foreseen that segmentation software will eventually become an evolved widespread clinical tool within the next two decades. Until then, using AS as a standard for delineating HNC TV and OARs still remains an active topic for the research community.

Limitations

The following limitations of this review article must be considered. First, the, majority of studies had small sample sizes that might not have sufficient statistical power to extrapolate their results to a more general patient population [43]. Second, there are many external factors that affect the assessment of delineation accuracy and time-saving ability of AS compared to MD in the reported studies. Reviewed studies had potential observer variability with varying numbers of delineations conducted, performed by persons with contrasting experience ranging from 1 to 15 years. Comparison across studies was also difficult due to use of different AS strategies, use of differing contouring guidelines without a shared ontology as well as the inherent issue of various tumor sites and stages in HNC being reported. This review has illustrated that for future studies on AS in HNC, it would be preferable for consensus contouring guidelines and a shared ontology to be adhered to; as well as stratification of subsites and stages of disease, where possible.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.