Pathophysiology of late anorectal dysfunction following external beam radiotherapy for prostate cancer

Abstract

Background. Patients treated with external beam radiotherapy (EBRT) may suffer from long-term anorectal adverse effects. The purpose of the present study was to assess long-term functional and structural anorectal changes in patients previously treated with EBRT for prostate cancer and to suggest the mechanism behind the development of the adverse effects.

Material and methods. Our previously proposed RT-induced anorectal dysfunction (RT-ARD) score, developed with the intention to survey anorectal dysfunction was used to identify patients with and without anorectal symptoms. Among 309 patients surveyed with the questionnaire, we chose 23 patients with the highest RT-ARD score and 19 patients with the lowest RT-ARD score. They were investigated by multimodal rectal sensory stimulation, standard anal physiological tests. Changes of the rectal mucosa were assessed by flexible sigmoidoscopy and graded by the Vienna Rectoscopy Score (VRS).

Results. The mean follow-up time was 3.8 (range, 2.8; 8.6) years in patients with high RT-ARD and 3.8 (range, 2.6; 5.9) in patients with low RT-ARD. Endoscopic evaluation revealed higher VRS scores in patients with high RT-ARD compared to patients with low RT-ARD (p = 0.002). Patients with high RT-ARD had increased rectal sensory response to distension manifested both as volume (p = 0.006) and cross-sectional area (p = 0.04), and they had reduced maximum anal resting pressure assessed by anal manometri (p = 0.02).

Conclusions. Long-term anorectal symptoms correlate to changes in anorectal biomechanical properties and rectal mucosal injury. Our data suggests that RT-induced long-term anorectal dysfunction is multifactorial caused by injury of the rectal mucosa and the internal anal sphincter combined with increased rectal sensitivity and reduced rectal functional capacity.

Anorectal dysfunction is a common adverse effect in patients having undergone external beam radiotherapy (EBRT) for prostate cancer [1–3]. The anatomic localization of the prostate makes the rectum a highly critical organ at risk when treating patients with prostate cancer by EBRT, and damage to the anorectum may further affect the patients’ daily activities and quality of life [4–6].

Irradiation of high doses to the rectal wall may lead to mucosal impairment, such as telangectasia, congested mucosa, ulcerations and thickening of the submucosa resulting in fibrosis, which may influence the sensitivity, the compliance and the rectal capacity [7–9]. In addition, irradiation of the anal sphincter complex may decrease the anal pressure. These changes may result in rectal bleeding, mucus discharge, increased frequency of defecation, fecal urgency and fecal incontinence [1,10]. The causes of anorectal symptoms after EBRT are likely multifactorial and the consequently underlying mechanisms behind these complex interactions, including anatomical and functional changes are poorly understood [5,11].

The aim of the present study was to investigate the association between patient reported gastrointestinal late side effects and structural and physiological anorectal changes. Rectal mucosal changes, biomechanical wall properties, rectal sensation, anal sphincter function and colorectal transit time were investigated within a cross-sectional approach in patients with anorectal dysfunction compared to patients with no anorectal dysfunction after EBRT for prostate cancer [12].

Materials and methods

Study population

The patients for the anorectal physiology study were selected from a questionnaire survey, which included all patients with prostate cancer treated with EBRT at Aarhus University Hospital, Denmark, from 1999 to 2007. The questionnaire survey was designed to develop and validate an anorectal dysfunction scoring system (RT-ARD) related to the patients quality of life [13]. The RT-ARD score includes the following five items with individual scores shown in brackets: Incontinence for solid stools (0–10), ability to defer defecation < 15 min (0–6), unproductive call to stool (0–9), clustering of stool (0–8) and mucus in stool (0–12). The sum of the scores ranges 0–45, where a higher score indicates a higher degree of anorectal dysfunction. From this cohort we formed two groups of patients; one group having no anorectal dysfunction (low RT-ARD: 0–8) and the other group suffering from moderate to severe anorectal dysfunction (high RT-ARD: 9–45). One hundred and six patients with low RT-ARD score and 128 patients with high RT-ARD score met the criteria of enrollment, which were no biochemical or metastatic failure and no prior rectal surgery. A total of 42 patients accepted to participate; 23 patients with high RT-ARD and 19 patients with low RT-ARD. The anorectal dysfunction, defined by the RT-ARD score was repeated by the patients at the time of enrollment of the clinical study to confirm their anorectal dysfunction as they were grouped according to their answer to the former mentioned questionnaire survey. All patients gave written consent. The study was approved by The Central Denmark Region Committees on Health Research Ethics (ID M-20100205).

The assessments developed for this particular study aimed to survey anatomical and functional changes behind anorectal dysfunction. These included sigmoidoscopy, multimodal rectal sensory stimulation, anal manometry, endoanal ultrasonography, and determination of colorectal transit time. The multimodal sensory stimulation was also performed postprandially to test the postprandial response. Each specific assessment is explained in more detail in the following.

Sigmoidoscopy

Sigmoidoscopy was performed to evaluate radiotherapy induced damage of the rectal mucosa. Mucosal radiation injury was recorded using the Vienna Rectoscopy Score (VRS) including recordings of teleangiectasia, congested mucosa and ulcerations [14]. The sigmoidoscopy was carried out without sedation using a flexible sigmoidoscope (Olympus, CF-Q160s) by one experienced senior colorectal surgeon. The rectal mucosa was evaluated at the anterior rectal wall levels 6 cm and 15 cm from the anal verge, respectively. The 6 cm and the 15 cm levels were chosen after evaluation of 20 randomly selected radiation treatment plans. The 6 cm level was situated just posterior of the prostate and therefore within the irradiated volume and the 15 cm level was outside the irradiated volume. The evaluation of the rectal mucosa was performed systematically to enable a uniform description of the rectal mucosa and photographs were taken to document the lesions and were done in a blinded approach with the symptoms for each patient concealed.

Multimodal rectal sensory stimulation

Multimodal rectal sensory stimulation permits a thorough description of sensory function in the rectum by applying standardized stimuli. Evaluation of biomechanical wall properties was performed by measuring luminal cross-sectional area (CSA) and pressure during distention.

Prior to multimodal rectal sensory stimulation, the patients were introduced to the electronic modified visual analog scale (VAS) ranging from 1 to 10 [15]. This scale has been described in details and has been shown to be both robust and valid in assessing rectal sensation in clinical studies [15]. The intensity was registered for each type of stimulus at sensory levels equivalent to VAS 1–7. Special focus was on vague perception of moderate sensation (VAS = 3), threshold for detection of pain (VAS = 5) and threshold for moderate pain (VAS = 7).

Rectal multimodal stimulation was performed with a custom-designed probe with a non-compliant 30 μm thick polyurethane bag mounted 2 cm from the tip of the probe. The bag had a maximum diameter of 11.5 cm and maximum recordable CSA was 10.380 mm² [12]. Rectal volume, luminal CSA and pressure were simultaneously recorded during distension and the time interval until sensory responses corresponding to VAS 3, 5 and 7 was noted. Distension was stopped at VAS 7. Rectal compliance (mL/mmHg) was computed from distension curves. The parameters were assessed in the fasting state and repeated again after ingestion of a standard meal to measure the postprandial response. The sensory response to temperature was investigated by recirculation of heated and cooled water through the multimodal probe and the temperatures at VAS levels of 3, 5, and 7 were noted. The study protocol and the sensory response have previously been described in details [16].

Endoanal ultrasonography

Endoanal ultrasonography was carried out with the purpose to evaluate the thickness of the anal sphincters. An endosonic 360° rotating transducer, frequency 10 MHz (Flex Focus 400, Type 1202, B-K Medical, Herlev, Denmark) was used to examine the external and internal anal sphincter as previously described [16]. Mean thickness of the anal sphincters muscles was determined on three-dimensional recordings of the internal and external anal sphincters.

Anal manometry

Anal manometry was performed to assess the function of the anal sphincters. The mean values of the maximum resting pressure and the maximum squeezing pressure were measured by standard anal manometry and the mean values were calculated after three repeated recordings of maximum squeezing pressure [16].

Colorectal transit time

To assess the gastrointestinal transit time the patients ingested 10 radiopaque markers for six consecutive days and an abdominal x-ray was taken on day 7 [17]. The segmental and total colorectal transit times were calculated by counting the remaining markers the segmental and total colorectal transit times were calculated. An experienced radiologist blinded to the patients’ symptoms did the assessment of the colorectal transit time.

Statistical analysis

Data are presented as mean ± standard error of the mean (SEM) unless stated. The student's t-test was used for comparison between normally distributed variables, tested by QQ-plots, and the non-parametric Mann-Whitney test was used for evaluation of variables that were not normally distributed. Statistical dependence between two variables was assessed by χ²-test or Fisher's exact test for categorical data. The Spearman's rank order correlation was used for testing of correlations and two-way analysis of variance (ANOVA) was used for the sensory and biomechanical analyses comparing the two groups of patients at VAS levels of 3, 5 and 7. All tests were two-sided and p < 0.05 was considered statistically significant. Statistical analyses were performed using STATA IC 11.2 (Statacorp, TX 77845, USA) and Sigma Stat v. 3.0 (SPSS Inc., Chicago, IL, USA).

Results

There were no differences among the two patient groups concerning age, follow-up time, specific treatment parameters or concomitant diseases (Supplementary Table I, available online at: http://informahealthcare.com/doi/abs/10.3109/0284186X.2014.926029). The patient with high RT-ARD had a significantly higher mean RT-ARD score compared to patient with low RT-ARD (p < 0.001, χ²) (Table I).

Table I. RT-ARD score.

Side effects	High RT-ARD (n = 23)	Low RT-ARD (n = 19)	p-Value
RT-ARD score, mean (range)	24 (9; 36)	6 (0; 8)	< 0.001
Incontinence solid stool^†	9	0	0.002
Ability to defer defecation < 15 min^†	16	7	0.034
Unproductive call to stool^‡	11	0	< 0.001
Clustering of stool^†	18	4	< 0.001
Mucus in stool^†	14	0	< 0.001

RT-ARD, radiotherapy-induced anorectal dysfunction score. ^†The specific side effect being present; ^‡≥ once a day.

Three patients did not complete the multimodal rectal sensory stimulation due to technical difficulties. Four of the patients did not complete sigmoidoscopy.

The endoscopic findings showed that the VRS was significantly different between patients with and without RT-ARD (p = 0.002; Mann-Whitney). Thus, telangiectasia were observed in 18 (90%) of the patients with high RT-ARD and in 11 (61.2%) of the patients with low RT-ARD (p = 0.003; χ²). Congested mucosa was observed in 18 (90%) of the patients with high RT-ARD and in six (33.4%) of the patients with low RT-ARD (p < 0.001; χ²). Microulcerations < 1 cm² were observed in one patient with high RT-ARD. No strictures or necrosis were observed (Table II).

Table II. VRS score and mucosal changes in patients with high and low RT-ARD.

	High RT-ARD (n = 20)	Low RT-ARD (n = 18)	p-Value
Mucosal radiation injury (VRS)
0	2 (10%)	7 (38.9%)	0.002
1	8 (40%)	10 (55.6%)
2	6 (30%)	1 (5.6%)
3	4 (20%)	0
4	0	0
5	0	0
Teleangiectasia (grade)
0	2 (10%)	7 (38.9%)	0.003
1	9 (45%)	10 (55.6%)
2	5 (25%)	1 (5.6%)
3	4 (20%)	0
Congested mucosa (grade)
0	2 (10%)	12 (66.7%)	< 0.001
1	2 (10%)	1 (5.6%)
2	6 (30%)	4 (22.2%)
3	10 (50%)	1 (5.6 %)
Ulceration (grade)
0	19 (95%)	18 (100%)	0.34
1	1 (5%)	0

RT-ARD, radiotherapy-induced anorectal dysfunction score; VRS, Vienna Rectoscopy Score.

Multimodal rectal sensory stimulation revealed that in the fasting state patients with high RT-ARD had increased sensory response to distension assessed by both volume and CSA when compared to patients with low RT-ARD (F = 8.0, p = 0.006 and F = 4.5, p = 0.04; ANOVA, for volume and CSA, respectively) (Figure 1). Mean values of CSA were 2436 mm² in patients with high RT-ARD and 2980 mm² in patients with low RT-ARD. Corresponding differences in sensory response to increased volume (F = 5.8, p = 0.02; ANOVA) and CSA (F = 7.1, p = 0.009; ANOVA) were observed postprandially (data not shown). No differences were found during stimulation with hot or cold water (F = 0.02, p = 0.88 and F = 2.6, p = 0.1; ANOVA). No differences between the two groups were found in compliance of the rectal wall neither in the fasting state nor postprandially (F = 3.0, p = 0.08 and F = 0.3, p = 0.58; ANOVA).

Figure 1. Thresholds to sensory stimulation of the rectum in patients with high RT-ARD (black) and with low RT-ARD (white) in the fasting state. Patients with high RT-ARD were more sensitive to rectal distensions; (A) they had a smaller rectal volume at different visual analog scale (VAS) levels (F = 8.0, p = 0.006), and (B) they had a smaller rectal CSA at different VAS levels (F = 4.5, p = 0.04).

Manometry and endoanal ultrasonography demonstrated a lower maximum anal resting pressure in the group with high RT-ARD score compared to the group with low RT-ARD score (p = 0.02, t-test) (Table III). No difference in maximum anal squeeze pressure, mean thickness of the internal or external anal sphincter was observed (Table III).

Table III. Manometry, endoanal ultrasonography and colorectal transit time.

	High RT-ARD (n = 23)	Low RT-ARD (n = 19)	p-Value
Manometry
MSP median (H2O), range	128 (30; 315)	151 (74; 295)	0.45
MRP median (H2O), range	48 (20; 127)	77 (29; 122)	0.02
Endoanal ultrasonography
IAS median (cm), range	1.95 (1.3; 2.8)	2.16 (1.2; 3.6)	0.36
EAS median (cm), range	9.3 (6.2; 10.8)	9.2 (7.4; 10.2)	0.36
Median colorectal transit time, days	1.7	2.2	0.98

EAS, external anal sphincter; IAS, internal anal sphincter; MRP, maximum resting pressure; MSP, maximum squeeze pressure; RT-ARD, radiotherapy-induced anorectal dysfunction score.

There was no difference in median colorectal transit time between the two groups (Table III). Likewise, segmental colorectal transit times of the ascending colon, the transverse colon, the descending colon or the rectosigmoid colon were not different between groups (data not shown).

Discussion

In the present study, we performed a detailed evaluation of anatomical and functional changes hypothesized to be related to anorectal dysfunction in patients having undergone EBRT for prostate cancer. Our results demonstrated objective signs of damage to the rectal mucosa, increased rectal sensitivity and reduced anal resting pressure in patients reporting anorectal dysfunction in the questionnaire survey.

By determining the rectal mucosal changes by the VRS, there was clearly more mucosal alterations in the group of patients reporting a high RT-ARD score than among patients with a low RT-ARD score. This is in accordance with previous reports where telangiectasia and congested mucosa have been described [18–20]. These studies also found that mucosal alterations correlated to rectal bleeding or to anorectal morbidity assessed by the Radiation Therapy Oncology Group (RTOG) scale [21]. In the present study, anorectal side effects were described using our previously developed RT-ARD score, which consists of five items selected based on the impact of quality of life [13]. The RT-ARD score does not contain rectal bleeding, because bleeding per se was not associated with a severe affection of the patient's quality of life. The mucosal alteration are likely responsible for the mucus discharge which represents one of the five items in the RT-ARD scoring system. We did find a positive correlation between rectal bleeding and VRS (data not shown) whereas the relationship between mucosal alterations and rectal sensitivity and functional capacity in the present study was less clear [22].

Our study revealed that patients with RT-ARD had an increased sensory response to rectal distension. Other studies have also demonstrated a reduced rectal capacity as a late reaction after radiotherapy [6,11] and a rectal wall hypersensitive to mechanical stimuli [23]. In line with previous studies, we found that anal resting pressure was reduced in patients with high RT-ARD while anal squeeze pressure was unaffected [16,24]. The combination of reduced anal resting pressure and normal squeeze pressure may be explained by weakness of the internal anal sphincter leading to passive fecal incontinence in some patients [25]. This is in line with a previous study that showed impaired function of the internal anal sphincter due to damage of the nerve supply and fibrotic replacement of the normal internal anal sphincter tissue [26]. The colorectal transit time revealed no difference between patients with and without symptoms. The finding of a normal colonic motility pattern suggests that the side effects defined by the RT-ARD score are caused by local damage to the structures surrounding the prostate gland and not by alterations of the rectal sensory neurons. Lack of differences in response to heat and cold stimulation between the two groups supports that the radiation-induced symptoms are caused by local damage of the periprostatic tissue [27,28].

In a recent review it was suggested that incontinence related complaints are associated with specific anorectal structural changes. Thus, a weakened anal sphincter may lead to fecal urgency and incontinence and frequency of defecation was correlated to a diminished rectal capacity [22]. All five symptoms contained in the RT-ARD score are likely explained by the findings in the present study. However, the mechanisms behind the development of radiation-induced anorectal damage are complex and it is difficult to explain each specific symptom by a particular alteration of a specific anorectal physiology parameter [5].

The findings in the present study should be seen in light of a recent study applying the same study protocol to patients treated with neoadjuvant radiotherapy (25 Gy in 5 fractions or approx. 50 Gy in 25–28 fractions) and surgery for rectal cancer. In these patients, the neorectum was constructed from a non-irradiated colonic segment, whereas the surrounding tissues in the pelvis had been enclosed in the irradiated volume. In contrast to patients in the present study, patients receiving surgery and neoadjuvant radiotherapy were characterized by neorectal hyposensitivity [16]. This could be explained by surgical denervation combined with extensive changes caused by radiation to a much larger volume in combination with a surgical trauma. In the case of EBRT for prostate cancer, only a small part of the anterior rectal wall receives a high radiation dose that may cause rectal hypersensitivity.

The different assessments used in the present study have strengths and weaknesses. It is a strength of the VRS scoring system that is validated; however it is a weakness that microulceration, which are one of the endpoints of the score, is only rarely seen after radiotherapy for prostate cancer. The functional testing (e.g. multimodal rectal stimulation measures, endoanal ultrasonography, anal manometry and colorectal transit time) have the advantages of measuring directly on different physiological measures, but the intra-individual variabilities of the measurements are relatively large.

We designed this study to compare anorectal physiology in selected groups of patients characterized by moderate/severe morbidity and no/minor morbidity. We believed that testing the extremes would reveal measureable difference between groups even in a relatively small cohort of patients. Furthermore, we performed a detailed testing comprising evaluation of the rectal mucosa, multimodal rectal sensory testing both in the fasting state and postprandially, anal manometry, endoanal ultrasonography and assessment of the colorectal transit time. The study revealed a significant correlation between the newly developed and validated RT-ARD score and the objective rectal mucosal changes revealed by sigmoidoscopy indicating that the RT-ARD score is a reliable tool for assessing anorectal dysfunction. However, the cross-sectional design and the lack of an untreated control group are limitations of the study.

The present study contributes to the understanding of the pathophysiology behind the development of anorectal dysfunction following EBRT of prostate cancer. So far, we do not know to which extent the patient's symptoms following radiotherapy are caused by reduced rectal capacity, increased anorectal sensitivity or weakened anal sphincter. Further studies may reveal the mechanisms behind the development of the morbidity and may therefore point at the structures that should be given the highest priorities and should be spared at the RT-planning.

In conclusion, we found an association between RT-ARD and radiation induced alterations of the rectal mucosa as well as rectal hypersensitivity, low anal resting pressure and reduced rectal functional capacity as a result of irradiation injury of the anterior rectal wall and the anal canal.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

The study is supported by the CIRRO – The Lundbeck Foundation for Interventional Research in Radiation Oncology and The Danish Cancer Society.