3,251
Views
102
CrossRef citations to date
0
Altmetric
Original Research

Use of broad-spectrum antibiotics impacts outcome in patients treated with immune checkpoint inhibitors

ORCID Icon, , , , , , , & show all
Article: e1507670 | Received 04 May 2018, Accepted 31 Jul 2018, Published online: 20 Aug 2018

ABSTRACT

We carried out a retrospective cohort study on patients with advanced cancer treated with immune checkpoint inhibitors (ICIs) to determine whether antibiotics affect treatment outcome. Sixty consecutive patients were identified, and 17 received systemic antibiotics within 2 weeks before and/or after first dose of ICI. Antibiotic-treated patients were significantly younger (p = 0.0008) and less likely to receive nivolumab (p = 0.08) or had neutrophil:lymphocyte ratio < 5 (p = 0.08). They had a lower response rate (RR) (29.4% vs 62.8%) (p = 0.024) and more inferior progression-free survival (PFS) (p = 0.048). Narrow-spectrum antibiotics did not affect the RR. However, broad-spectrum antibiotics were associated with a lower RR (25% vs 61%) (p = 0.02) and a trend towards longer time to response (median: 14 weeks vs 12 weeks) (p = 0.1). They also had shorter PFS (p = 0.012). Multivariate analysis identified antibiotics as the only factor affecting RR (p = 0.0038) and PFS (p = 0.01). We next examined the 21 patients whose PFS lasted for 12 weeks or more. Five of the 21 patients received broad-spectrum antibiotics within 10 weeks before disease progression. There was a trend towards shorter PFS in these patients (p = 0.1). Finally, antibiotic-treated patients experienced shorter overall survival (OS) (median: 24 months vs 89 months) (p = 0.003). Multivariate analysis found age (p = 0.035) and antibiotics (p = 0.038) to be the only factors affecting OS. Our results point to a detrimental effect of broad-spectrum antibiotics on treatment outcome to ICI therapy.

Introduction

Effective immune responses against tumor rely on the successful generation of cytotoxic T cells that mediate tumor cell-kill. These adaptive immunities are the results of the interaction between the pro-inflammatory and anti-inflammatory cytokines, and the homeostatic processes that modulate the responses. One of the major homeostatic mechanisms in regulating appropriate immune responses is the Programmed cell death (PD)-1/Programmed cell death ligand (PDL)-1 axis that forms part of the immune checkpoint network. Immune checkpoint inhibitors (ICIs) have been made available commercially for use in the clinics as a mean to modulate the immune checkpoint network in patients with various cancers. These agents interfere with the homeostatic processes that induce apoptosis of cytotoxic T cells arisen from adaptive immune responses. Factors that modulate these immune responses may, therefore, affect the treatment outcome in patients receiving ICIs.

Antibiotics are effective for treating various infections. However, contingent on their antimicrobial spectrum and pharmacokinetics, they will also change the composition of the intestinal microbiota. A balanced intestinal microbiome is needed for the immune homeostasis, most notably on adaptive immunity.Citation1 Perturbation of the balanced microbiome may, therefore, disrupt the homeostatic loop and induce detrimental health effects.Citation2 Compositional and metabolic changes in the intestinal microbiota (dysbiosis) have been found in association with inflammatory bowel disease, where there are increases in Proteobacteria, Firmicutes bacillus, and Actinobacteria.Citation3 Dysbiosis has also been implicated in the pathophysiology of colorectal cancer, multiple sclerosis, diabetes mellitus, obesity and allergies,Citation4Citation8 and most recently by us in sickle cell disease.Citation9 In addition to changing the intestinal microbiome, some antibiotics, like tetracyclines, fluoroquinolones and macrolides, also have intrinsic anti-inflammatory effects on the hosts.Citation10Citation15 Antibiotics may, therefore, affect the outcome of patients treated with ICIs, through their influence on the intestinal microbiota composition or direct anti-inflammatory activities. In this study, we set out to determine whether antibiotic use during therapy with ICI affected the treatment outcome.

Results

Patient characteristics

We identified 60 consecutive patients with advanced cancer treated with ICIs during the study period. Seventeen (28%) patients received systemic antibiotics within 2 weeks before and/or after starting ICI therapy and 43 (72%) patients did not. The reasons for the use of antibiotics were: Urinary tract infections (n = 4), empiric therapy for fever (n = 3), acne (n = 2), hospital acquired pneumonia (n = 2), cellulitis (n = 1), bacterial peritonitis (n = 1), and unknown (n = 4). The duration of antibiotic use in all cases were 8–14 days, except in the two patients who received minocycline continuously for acne. The clinical characteristics of these patients are shown in . There was no statistical difference between the two groups in terms of gender, ECOG performance score, or PDL-1 expression. By far, lung cancer was the commonest diagnosis in both groups (59% vs 56%). There was also no statistical difference between the proportion of patients who received ICI as first line therapy or in combination with chemotherapy between the two groups of patients. However, antibiotic-treated patients were significantly younger (median age: 52 vs 66 years) (p = 0.0008) and showed a trend towards significance in being less likely to receive nivolumab (p = 0.08) and to have a pre-treatment neutrophil:lymphocyte (N:L) ratio of < 5 (p = 0.08).

Table 1. Baseline Characteristics of Patients

Effects of antibiotics on response rate

The overall response rate in this cohort of patients was 53% (32/60 patients). All 32 patients had a partial response. A wide range of systemic antibiotics was used by these 17 patients (). By far, the 3rd and 4th generation cephalosporins were most commonly used, followed by vancomycin and ciprofloxacin. We found that the response rate (RR) was significantly lower in patients who received systemic antibiotics within 2 weeks before or after being started on ICI compared to those who did not (29.4% vs 62.8%) (HR = 1.9; 95% CI: 1.2–3) (p = 0.024). To further characterize the type of antibiotics associated with the lower RR, we next divided the antibiotics into narrow-spectrum antibiotics, and broad-spectrum antibiotics. Narrow-spectrum was defined as antibiotics only covering Gram-positive bacteria, while broad-spectrum antibiotics included antibiotics that would cover Gram-positive and negative ± anaerobic bacteria. The use of narrow-spectrum antibiotics did not affect the RR. In contrast, patients treated with broad-spectrum antibiotics experienced a lower RR (25% vas 61%) (HR = 2.34; 95% CI: 1.5–3.65) (p = 0.02). They also showed a trend towards longer time to response (median: 14 weeks vs 12 weeks) (p = 0.1). Multivariate analysis found that the use of broad-spectrum antibiotics was the only significant factor in determining the response rate to ICI (p = 0.0038). Age, nivolumab, and N:L ratio of > 5 were not.

Table 2. Types of antibiotics

Effects of antibiotics on progression-free survival

We next determined how the use of antibiotics affected the PFS in these patients. Use of antibiotics within 2 weeks before or after starting ICI therapy not only affected the RR but also resulted in an inferior PFS (HR = 1.6; 95% CI: 0.84–3.03) (p = 0.048) (). The effects of antibiotics use on PFS was even more obvious when the analysis was carried out only on those who received broad-spectrum antibiotics (HR = 1.8; 95% CI: 0.86–3.89) (). Multivariate analysis found the use of antibiotics being the only significant factor affecting PFS (p = 0.01).

Figure 1. Progression-free survival in patients who had received any antibiotics within 2 weeks before and/or after starting ICI compared to those who had not. Antibiotic-treated patients had significantly more inferior PFS (p = 0.048)

Figure 1. Progression-free survival in patients who had received any antibiotics within 2 weeks before and/or after starting ICI compared to those who had not. Antibiotic-treated patients had significantly more inferior PFS (p = 0.048)

Figure 2. Progression-free survival in patients who had received broad-spectrum antibiotics within 2 weeks before and/or starting ICI compared to those who had not. Patients who had not received any broad-spectrum antibiotics had significantly superior PFS (p = 0.012)

Figure 2. Progression-free survival in patients who had received broad-spectrum antibiotics within 2 weeks before and/or starting ICI compared to those who had not. Patients who had not received any broad-spectrum antibiotics had significantly superior PFS (p = 0.012)

Effects of antibiotics on disease relapse

We also evaluated whether the use of broad-spectrum antibiotics is associated with disease relapse. In this analysis, we only included those patients whose PFS was 12 or more weeks. Twenty-one (65.6%) of the 32 responders enjoyed the pre-defined durations of PFS. Five (23.8%) of these patients received antibiotics (all broad-spectrum antibiotics) within 10 weeks before disease progression was recorded. The duration of progression-free survival in these five patients showed a trend towards significance for shorter PFS compared to those who did not receive broad-spectrum antibiotics prior to disease relapse (HR 2.1; 95% CI: 0.59–7.6) (p = 0.1) ().

Figure 3. Progression-free survival in patients who had received broad-spectrum antibiotics within 10 weeks before disease progression compared to those who had not. Patients who had received antibiotics showed a trend towards significance in shorter PFS (p = 0.1)

Figure 3. Progression-free survival in patients who had received broad-spectrum antibiotics within 10 weeks before disease progression compared to those who had not. Patients who had received antibiotics showed a trend towards significance in shorter PFS (p = 0.1)

Effects of antibiotics on overall survival

Finally, we analyzed the effects of broad-spectrum antibiotics on overall survival (OS) of this cohort of patients. Patients who did not received broad-spectrum antibiotics enjoyed a longer OS compared to those who did (). The median OS was 89 weeks (95% CI: 44 – 105) for those who did not receive antibiotics but only 24 weeks (95% CI: 4 – 28) (p = 0.003) (HR 2.9, 95% CI: 1.1 – 8.1) in those who did. Multivariate analysis showed that both age (p = 0.035) and antibiotic use (p = 0.038), but not N:L ratio > 5 or the use of nivolumab, were the only predictors of improved OS.

Figure 4. Patients who did not receive antibiotics enjoyed longer overall survival compared to those who did (p = 0.003)

Figure 4. Patients who did not receive antibiotics enjoyed longer overall survival compared to those who did (p = 0.003)

Discussion

Although ICIs have benefited many patients with advanced cancer, a substantial number of patients failed to response to the therapy or responded but only with short durations of PFS. Many investigators have worked to determine the tumor and host factors that impact responses to ICIs. While these would help identify the group of patients most likely to respond or not respond, they are not modifiable by the treating physicians. Identification of actionable factors influencing responses to ICIs is, therefore, much needed.

A previous study showed that primary resistance to ICI therapy in epithelial tumors was related to abnormal intestinal microbe community.Citation16 Antibiotic administration inhibited the clinical benefit of ICIs in patients with advanced cancer. Furthermore, the antitumor effects of PD-1 blockade was abrogated when cancer-bearing mice received fecal microbiota transplantation (FMT) from these patients. A relative abundance of Akkermansia muciniphila was identified to be associated with clinical response. Interestingly, tumor response was restored when Akkermansia muciniphila was administered to tumor-bearing mice that had undergone FMT from patients with primary resistance to ICI therapy. These results, therefore, support the rationale for our study.

We found in this study that patients who received antibiotics within two weeks before and after starting ICI therapy took longer time to respond to the therapy and they also experienced lower response rate and PFS compared to those who did not. Our findings are not unlike those reported at the recent American Society of Clinical Oncology meetings for metastatic renal cell carcinomaCitation17 and a cohort of patients with heterogenous diagnoses.Citation18 Similar observations were also observed in another study that involved renal cell and non-small cell lung cancer.Citation19 These detrimental effects of antibiotics associated with treatment outcome were prominently seen, especially in those who received broad-spectrum antibiotics such as the quinolones and 3rd or 4th generation cephalosporins, even in multivariate analysis. These results, therefore, raise the possibility that the broad-spectrum antibiotics given to the patients during ICI treatment impair their response to the therapy. Obviously, it is also possible that the association between antibiotic use and inferior outcome might have arisen due to antibiotics being administered to those febrile patients who became sicker as their disease progressed, although more than half of our patients who were treated with antibiotics had clear sources of infection. The detrimental effects of broad-spectrum antibiotics on treatment outcome were further supported by our findings that those patients who received antibiotics within 10 weeks of disease progression experienced shorter durations of response.

Taking into consideration the effects of the antibiotics, our findings are not surprising, although insightful. The detriments of antibiotics on ICI treatment outcome could be related to their effects in altering the intestinal microbiota compositions. ICIs have been found to be less effective for treatment of sarcoma in mice housed under germ-free conditions compared to those housed under specific pathogen-free conditions.Citation20 Mice bearing sarcoma when treated with broad-spectrum antibiotics that eliminated gut microbiota responded poorly to the ICIs, anti-CTLA-4 antibodies. Interestingly, when the germ-free mice and antibiotic-treated mice were replenished with Bacteroides, responses to anti-CTLA-4 were restored via stimulation of Th1 response. Two studies in patients who received ICIs with ipilimumabCitation21 or an anti-PD-1 antibodyCitation22 also found that patients whose intestinal microbiome was perturbed had inferior outcome when compared to those with an intact microbiome.

Various investigations have implicated certain intes-tinal microbes in modulating immune responses. Both Lactobacillus johnsonii and Enterococcus hirae stimulate the differentiation of Th17 and Th1 cells,Citation23 and Ruminococcus promotes tumor necrosis factor-α (TNFα) production.Citation24 The quinolones, on the other hand, down-regulate the production of TNFα.Citation25 Overgrowth of Clostridium species,Citation26 Bacteroides fragilis,Citation27 and Staphylococcus aureusCitation28 promotes the development of FOXp3 + T-regulatory cells. Bacteria-derived lipopolysaccharides (LPS) have also been found to directly stimulate CD8 T cells through LPS receptor complex and Toll-like receptor (TLR) 4.Citation29 Any change in the relative abundance of these signature microbes could, therefore, affect the antitumor activity associated with the use of ICIs. Our results suggest that the broad-spectrum antibiotics given to the patients might have depleted the microbes involved in stimulatory immune responses and expanded the microbes that induce suppressive immune responses, rendering ICIs less effective.

In conclusion, our results indicate that use of broad-spectrum antibiotics during ICI therapy is associated with poorer response rate and shorter PFS. Despite the shortfalls of a retrospective study in a group of patients with heterogenous diagnoses, our observations provide the foundation for prospective works to further delineate the basis for the detrimental effects of broad-spectrum antibiotics on ICI therapy. It is likely that manipulation of the intestinal microbiota will eventually be part of ICI therapy. Future translational studies should include studies to determine the role of concurrent probiotics administration, intermittent FMT, and the use of antibiotics that target specific microbes to enhance the immunostimulatory effects or down-regulate the immunosuppressive effects of certain intestinal organisms. In patients needing antibiotics, careful selection to avoid antimicrobial agents that modulate immune responses should also be taken into consideration.

Patients and Methods

Data collection

A retrospective cohort study was performed on all patients ≥ 18 years of age who received the either PD-1 inhibitors (nivolumab or pembrolizumab) or PD-L1 inhibitor (atezolizumab) for advanced cancer at Westchester Medical Center during the period from April 2014 to December 2017. Patient demographics, clinical data, diagnosis, and treatment history were collected. Antibiotic use and types were collected from all patients within 2 weeks prior to and after initiation of therapy and within 10 weeks prior to disease progression. We have chosen the 2-week time point to evaluate the effects of antibiotics on response and duration of response because any microbiome changes due to antibiotics will last longer than 2 weeks. We have also chosen the 10-week time point to evaluate the effects of antibiotics on disease progression because responders to ICI underwent radiologic evaluation no more frequent than every two to three months. Therefore, the effects of antibiotics on disease progression is likely to be only detected radiologically two to three months after the use of antibiotics. Data was censored on March 31, 2018. The study was conducted with the approval from New York Medical College and Westchester Medical Center Institutional Review Board.

Response evaluation

All patients had computed tomography (CT) imaging performed prior to starting treatment with ICIs and repeated every 6–12 weeks during therapy. Imaging studies were reported by radiologists at the medical center and reviewed by treating physicians. All responses were determined using the Response Evaluation Criteria in Solid Tumors (RECIST) 1.1.Citation30 Duration of response was measured from the time of first measurement of partial response (PR) or complete response (CR) until the date that disease progression was documented.

Statistical analysis

The response rate and progression-free survival (PFS) were calculated. PFS was also plotted as a time-dependent covariate using Kaplan-Meier method. Association of factors potentially predictive of response was evaluated using the Chi-square tests. Differences in the various clinical and laboratory parameters were calculated as mean and compared using the Student t tests. A two-sided p value of ≤ 0.05 was considered statistically significant and > 0.05 but ≤ 0.1 was considered a trend towards significance. All covariates with a p value of ≤ 0.1 were included in multivariate analysis.

Additional information

Funding

This work was supported by the None;

References

  • Lee YK, Mazmanian SK. Has the microbiota played a critical role in the evolution of the adaptive immune system? Science. 2010;330:1768–1773. doi:10.1126/science.1195568.
  • Lloyd-Price J, Abu-Ali G, Huttenhower C. The healthy human microbiome. Genome Med. 2016;8:51. doi:10.1186/s13073-016-0307-y.
  • Peterson DA, Frank DN, Pace NR, Gordon JI. Metagenomic approaches for defining the pathogenesis of inflammatory bowel diseases. Cell Host Microbe. 2008;3:417–427. doi:10.1016/j.chom.2008.05.001.
  • Ahn J, Sinha R, Pei Z, Dominianni C, Wu J, Shi J, Goedert JJ, Hayes RB, Yang L. Human gut microbiome and risk for colorectal cancer. J Natl Cancer Inst. 2013;105:1907–1911. doi:10.1093/jnci/djt300.
  • Jangi S, Gandhi R, Cox LM, Li N, Von Glehn F, Yan R, Patel B, Mazzola MA, Liu S, Glanz BL, et al. Alterations of the human gut microbiome in multiple sclerosis. Nat Comm. 2016;7:12015. doi:10.1038/ncomms12015.
  • Upadhyaya S, Banerjee G. Type 2 diabetes and gut microbiome: at the intersection of known and unknown. Gut Microbes. 2015;6:85–92. doi:10.1080/19490976.2015.1024918.
  • Bäckhed F, Ding H, Wang T, Hooper LV, Koh GY, Nagy A, Semenkovich CF, Gordon JI. The gut microbiota as an environmental factor that regulates fat storage. Proc Natl Acad Sci USA. 2004;101:15718–15723. doi:10.1073/pnas.0407076101.
  • Riiser A. The human microbiome, asthma, and allergy. Allergy Asthma Clin Immunol. 2015;11:35. doi:10.1186/s13223-015-0099-4.
  • Lim SH, Morris A, Li K, Fitch AC, Fast L, Goldberg L, Quesenberry M, Sprinz P, Methé B. Intestinal microbiome analysis revealed dysbiosis sickle cell disease. Am J Hematol. 2018;93:E91–E93. doi:10.1002/ajh.25019.
  • Iino Y, Toriyama M, Natori K, You A. Erythromycin inhibition of lipopolysaccharide-stimulated tumor necrosis factor alpha production by human monocytes in vitro. Ann Otol Rhinol Laryngol. 1992;101:16–20. doi:10.1177/0003489492101S1005.
  • Morikawa K, Oseko F, Morikawa S, Iwamoto K. Immunomodulatory effects of three macrolides, midecamycin acetate, josamycin, and clarithromycin, on human T-lymphocyte function in vitro. Antimicrob Agents Chemother. 1994;38:2643–2647.
  • Morikawa K, Watabe H, Araake M, Morikawa S. Modulatory effects of antibiotics on cytokine production by human monocytes in vitro. Antimicrob Agents Chemother. 1996;40:1366–1370.
  • Roche Y, Gougerot-Pocidalo MA, Fay M, Etienne D, Forest N, Pocidalo JJ. Comparative effects of quinolones on human mononuclear leucocyte functions. J Antimicrob Chemother. 1987;19:781–790.
  • Korzeniowski OM. Effects of antibiotics on the mammalian immune system. Infect Dis Clin N Am. 1987;3:469–478.
  • Tang X, Wang X, Zhao YY, Curtis JM, Brindley DN. Doxycycline attenuates breast cancer related inflammation by decreasing plasma lysophosphatidate concentrations and inhibiting NF-κB activation. Mol Cancer. 2017;16:36. doi:10.1186/s12943-017-0607-x.
  • Routy B, Le Chatelier E, Derosa L, Duong CPM, Alou MT, Daillère R, Fluckiger A, Messaoudene M, Rauber C, Roberti MP, et al. Gut microbiome influences efficacy of PD-1-based immunotherapy against epithelial tumors. Science. 2018;359:91–97. doi:10.1126/science.aan3706.
  • Lalani AA, Xie W, Lin X, Steinharter JA, Martini DJ, Duquette A, Bosse D, McKay RR, Simantov R, Wei XX, et al. Antibiotic use and outcomes with systemic therapy in metastatic renal cell carcinoma (mRCC). J Clin Oncol. 2018;36(Suppl.):607. doi:10.1200/JCO.2018.36.6_suppl.607.
  • Tinsley N, Zhou C, Villa S, Tan G, Lorigan P, Blackhall FH, Elliott T, Krebs M, Carter L, Thistlethwaite F, et al. Cumulative antibiotic use and efficacy of immune checkpoint inhibitors in patients with advanced cancer. J Clin Oncol. 2018;36(Suppl.):3010.
  • Derosa L, Hellmann MD, Spaziano M, Halpenny D, Fidelle M, Rizvi H, Long N, Plodkowski AJ, Arbour KC, Chaft JE, et al. Negative association of antibiotics on clinical activity of immune checkpoint inhibitors in patients with advanced renal cell and non-small cell lung cancer. Ann Oncol. 2018 Mar 30. [Epub ahead of print]. doi:10.1093/annonc/mdy103.
  • Sivan A, Corrales L, Hubert N, Williams JB, Aquino-Michaels K, Earley ZM, Benyamin FW, Lei YM, Jabri B, Alegre ML, et al. Commensal Bifidobacterium promotes immunity and facilitates anti-PD-L1 efficacy. Science. 2015;350:1084–1089. doi:10.1126/science.aac4255.
  • Chaput N, Lepage P, Coutzac C, Soularue E, Le Roux K, Monot C, Boselli L, Routier E, Cassard L, Collins M, et al. Baseline gut microbiota predicts clinical response and colitis in metastatic melanoma patients treated with ipilimumab. Ann Oncol. 2017;28:1368–1379. doi:10.1093/annonc/mdx108.
  • Gopalakrishnan V, Spencer CN, Nezi L, Reuben A, Andrews MC, Karpinets TV, Prieto PA, Vicente D, Hoffman K, Wei SC, et al. Gut microbiome modulates response to anti-PD-1 immunotherapy in melanoma patients. Science. 2018;359:97–103. doi:10.1126/science.aan4236.
  • Viaud S, Saccheri F, Mignot G, Yamazaki T, Daillère R, Hannani D, Enot DP, Pfirschke C, Engblom C, Pittet MJ, et al. The intestinal microbiota modulates the anticancer immune effects of cyclophosphamide. Science. 2013;342:971–976. doi:10.1126/science.1240537.
  • Ménard S, Candalh C, Bambou JC, Terpend K, Cerf-Bensussan N, Heyman M. Lactic acid bacteria secrete metabolites retaining anti-inflammatory properties after intestinal transport. Gut. 2004;53:821–828.
  • Gollapudi SV, Chuah SK, Harvey T, Thadepalli HD, Thadepalli H. In vivo effects of rufloxacin and ciprofloxacin on T-cell subsets and tumor necrosis factor production in mice infected with Bacteroides fragilis. Antimicrob Agents Chemother. 1993;37:1711–1712.
  • Nagano Y, Itoh K, Honda K. The induction of Treg cells by gut-indigenous Clostridium. Curr Opin Immunol. 2012;24:392–397. doi:10.1016/j.coi.2012.05.007.
  • Telesford KM, Yan W, Ochoa-Reparaz J, Pant A, Kircher C, Christy MA, Begum-Haque S, Kasper DL, Kasper LH. A commensal symbiotic factor derived from Bacteroides fragilis promotes human CD39(+)Foxp3(+) T cells and Treg function. Gut Microbes. 2015;6:234–242. doi:10.1080/19490976.2015.1056973.
  • Rabe H, Nordström I, Andersson K, Lundell AC, Rudin A. Staphylococcus aureus convert neonatal conventional CD4(+) T cells into FOXP3(+) CD25(+) CD127(low) T cells via the PD-1/PD-L1 axis. Immunology. 2014;141:467–481. doi:10.1111/imm.12209.
  • Komai-Koma M, Gilchrist DS, Xu D. Direct recognition of LPS by human but not murine CD8+ T cells via TLR4 complex. Eur J Immunol. 2009;39:1564–1572. doi:10.1002/eji.200838866.
  • Eisenhauer EA, Therasse P, Bogaerts J, Schwartz LH, Sargent D, Ford R, Dancey J, Arbuck S, Gwyther S, Mooney M, et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur J Cancer. 2009;45:228–247. doi:10.1016/j.ejca.2008.10.026.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.