Author’s Response to Letter to the Editor

Re: Hutchinson M, Fox RJ, Havrdova E, et al. Efficacy and safety of BG-12 (dimethyl fumarate) and other disease modifying therapies for the treatment of relapsing–remitting multiple sclerosis: a systematic review and mixed treatment comparison. Curr Med Res Opin 2014;30:613-27

Dear Editor,

Please see our response to the letter you received regarding the above article:

Safety section

Comment 1: “Network meta-analyses, which respect the concurrent control structure of the data, were not performed. Comparisons of safety outcomes was done by comparing incidences only in treatment arms in each trial instead of using relative effects that preserves the effect of randomization within study. Comparing treatment arms across studies is called naïve indirect comparison in the literature, and should be avoided in the analyses of data from randomized clinical trials”.

Response: Relative risk (via network meta-analysis) is one of many alternatives (including annual rates and probabilities) to present events data1. However, phase II–III clinical trials are not designed to test the specified hypotheses about safety (assessed as tertiary outcome) or to detect statistical differences in safety between the study treatments2^,3. Hence, findings of network meta-analysis were not presented and a non-relative presentation of safety in the form of annual rates and probabilities was chosen. Moreover, data would not have been available for all adverse events for each treatment arms; hence, the relative safety assessment would not have been an appropriate representation.

Comment 2: “The reason stated for not performing network meta-analyses was lack of power in detecting statistical differences between treatments arms. As this does not prevent getting reliable estimates, this is not considered as an appropriate explanation. Furthermore, most of the studies included in the analysis were not powered to detect differences in disability progressing, yet a network meta-analysis was performed on this outcome”.

Response: Various submissions have recognized confirmed disability progression as a primary outcome with good power to detect clinically meaningful results4. Unlike safety outcomes, disability progression was assessed as time-to-event outcome from Kaplan–Meier curves presented in various studies. Network meta-analysis to calculate relative hazards of progressions was a more appropriate method here, which took into account the changing risk of progression over the time. In addition, disability progression is one of the primary or secondary outcomes in 17 of the 27 included studies and, therefore, is more adequately powered in comparison to safety outcomes, which are usually assessed as in studies as tertiary outcome. It would be unwise to compare safety outcomes with disability progressions due to differences in assessment and occurrences.

Comment 3: “Uncertainty about incidence rate estimates was ignored”.

Response: There is no single correct way to present risk and probabilities. Relative risk reduction, absolute risk reduction, and number needed to treat can be confusing because they avoid making the baseline risk clear. International Society for Pharmacoeconomics and Outcome Research Good Research Practice (ISPOR) Task force cites “the method for transforming interval probabilities from the literature or a clinical trial into an instantaneous rate and then into event probability”5. By using method recommended by Fleurence and colleagues, we have calculated instantaneous events rate and then transformed into annualized event probabilities to provide the best guess estimates of evidence across the published randomised controlled trials in a given time frame1.

We acknowledge that the degree of uncertainty needs to be quantified. Ideally a patient or a physician would like to know whether or not they will suffer an adverse drug effect. Such certainty cannot be provided except in very rare instances. Hence we need to be able to convey the sense and preferably the degree of uncertainty. Unfortunately using statistical entities such as confidence intervals around event rates are largely unhelpful6.

Comment 4: “Safety outcomes were included if there was at least a 5% difference in incidence rates between BG-12 and comparator. This arbitrary data driven filter introduces bias, as only outcomes where there is a difference are included. The selection of safety outcomes does not correspond to a set allowing for a comprehensive assessment of safety across treatments. For example, the BRAT guideline (http://www.cirs-brat.org) for assessing benefit–risk ratios not included in original recommends using frequency, severity and regulatory and public health importance”.

Response: The BRAT guidelines (version 1.0 dated 21 November 2013) state that “benefit–risk Framework is not intended to include a listing of all adverse events observed with use of a product during development; rather it should list outcomes that are considered to have a substantive impact on the benefit–risk balance (i.e., an impact that is meaningful to patients, physicians, or others whose perspective is being considered in the analysis)”. In addition, the BRAT guidelines highlight the need to document “a clear description of the criteria for selecting outcomes and the decision-making approach used (e.g., consensus approach, nominal group recommendation)”.

A non-selective approach was used to present results of specific adverse events in an unbiased manner after considering all AEs that occurred at an incidence of ≥3% in the total BG-12 group compared to the placebo (even if the overall incidence in the BG-12 arm was <5%). As a conservative estimate in comparison with BG-12, the AEs included for comparators were only those reported in BG-12 studies. The results were presented not only where a comparator had at least 5% higher incidence of AEs than BG-12 but also where BG-12 had at least 5% higher incidence of AEs than any comparator (See Figure 5 in original article: Summary plot of the annual incidence of adverse events where there is a ≥5% point difference between BG-12 240 mg BID and comparators). The initial list of AEs selected came from the BG-12 trials, and hence, do not put any of the comparators at a disadvantage.

Comment 5: “The burden and severity of each outcome was not assessed”.

Response: We acknowledge the fact that burden and severity were not included under scope of the manuscript and hence, were not presented. We have considered this suggestion for future publications.

Comment 6: “Conclusions contain information on safety outcomes outside of the selected list indicating biased approach to inferences made”.

Response: All the listed AEs were evaluated but the presentation of safety was limited only to those AEs where at least 5% difference in incidence rates was observed between BG-12 and comparators. This resulted in the omission of specific AEs (such as PML and hepatotoxicity) which are serious AEs associated with some of the currently marketed drugs but not observed with BG-12. In this way, a holistic view was provided without any intention to show undue advantage of BG-12 over other disease modifying therapies (DMTs).

Efficacy section

Comment 1: “The study ignored that there are different endpoint definitions with respect to confirmed disability progression across trials, the results with respect to risk of disability progression are biased and so don’t lead to a valid inferences on disability progression”.

Action taken: No changes are required.

Response: Within disability progression, the definitions of disability progression were consistent across all included studies, except for trials evaluating fingolimod and teriflunomide (where disability was defined as increase from baseline of at least 1.0 point in the EDSS score [or at least 0.5 points for patients with a baseline EDSS score greater than 5.5]). In general, such definition caveats exist in head-to-head trials and analysis of all the definitions separately could result in omission of important comparators. There is a balance to be struck between having few analyses focusing on outcome with the same definitions and having analyses that contain sufficiently homogeneous studies. Considering the differences in the definitions across trials, a random effect model was used.

Comment 2: “Although modeling to account for differences in patient characteristics was performed, the results of this assessment are not presented.

Patients who are treatment naïve and previously treated are combined in the same analysis. These patients respond differently to treatment and none of the models account for this. In particular combining these patients occurs within a loop which would affect the estimates of BG-12”.

Response: We acknowledge the fact that sub-group analysis for treatment-naïve patients were not included under the current scope of the manuscript and hence were not presented. We have considered this suggestion for future publications.

Comment 3: “In general it is not clear how the analysis was done.

There is not sufficient explanation of the statistical methodology. There are no models given, and no references to any papers giving the methods”.

Response: The details on statistical methodology have been provided under ‘Data analysis and mixed treatment comparison’. Since PROC GLIMMIX is a well established and widely used method under generalized linear models7^,8, the explanation provided in the manuscript can be considered appropriate and sufficient within the scope of the research question. However, we can answer specific questions around the model on requests.

Comment 4: “The models are described as having study as a random effect and treatment as a fixed effect. This is not the usual way of performing such an analysis, as it assumes there is no heterogeneity in the treatment effects. Commonly (e.g. the Lu and Ades framework) random effects are put on the treatment effect, and study effects are treated as nuisance parameters”.

Response: We have used a well established GLIMMIX procedure which performs estimation and statistical inferences for generalized linear mixed models (GLMMs). A GLMM is a statistical model that extends the class of GLM by incorporating normally distributed random effect in the linear predictor. The PROC GLIMMIX statement invokes the procedure. There is a class statement in the model which instructs the procedure to treat the variables as classification variables. In our analysis, we have used treatment and study as classification variables putting a random effect on treatment. The distribution is conditional on random treatment effect7–9 and is in line with Lu and Ades10. A comprehensive description of model and statements was out of scope of the publication.

Comment 5: “Heterogeneity is expected to be quite high, as noted by the authors. However, no estimate of the heterogeneity is given. IFNs have been pooled, which may also add to the heterogeneity”.

Response: Deviations and caveats do exist in mixed treatment comparison (MTC) analysis and a random effect model was used to account for moderate heterogeneity observed across included studies. The estimates of heterogeneity have been briefly described under ‘Data analysis and mixed treatment comparison’. Moreover, covariate analyses were also performed to assess the impact of publication year, study duration, mean age, disease duration, female percentage, relapse in prior 1 year, and EDSS score at baseline. The results have been briefly presented in the manuscript as well under ‘Discussion’. The beta IFNs were pooled as they generally have a similar mechanism of action and efficacy in the treatment of relapsing-remitting multiple sclerosis (RRMS)11^,12.

Sincerely,

Baris Deniz (on behalf of all authors)

Biogen Idec

E-mail:[email protected]