Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome

ABSTRACT

Background

Several randomized trials have reported that graded exercise therapy (GET) is an effective treatment for chronic fatigue syndrome (CFS). These trials were not blinded and relied on patient-reported outcome measures (PROMs). We investigate whether bias introduced by this study design influenced the results.

Methods

We extracted standardized mean differences from the most recent meta-analysis on exercise therapy for CFS to analyze their size, consistency over time, and congruence with objective measurements. A narrative review methodology was used to examine mediation analyses, plausible mechanisms of improvement, and risk of response bias.

Results

Patient-reported improvements in exercise trials for CFS tend to be small, transient, and poorly supported by objective measurements. The risk of expectancy effects and response bias was high as patients were actively encouraged to adopt a positive attitude towards exercise therapy. Mediation analyses suggest that self-reported improvements in fatigue and physical function are not mediated by objective measures of fitness.

Conclusions

Treatment effects seen in exercise trials for CFS could be the result of bias associated with the use of PROMs in non-blinded trials. This might explain the discrepancy between positive results reported in randomized trials and views on exercise therapy expressed by patient organizations. We hope that this case study furthers critical assessment of patient-reported improvements in areas of medicine where blinding of therapists and trial participants faces practical limitations.

KEYWORDS:

• Chronic fatigue syndrome
• graded exercise therapy
• patient-reported outcome measures
• minimal clinically important difference
• bias
• blinding

Acknowledgements

The authors would like to thank Tom Kindlon, Simon McGrath, and Andrew Kewley for their thoughtful comments to earlier drafts of this analysis.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Competing interests

MT is the assistant chairperson of the Belgian patient organization 12ME. Members of the ME/CFS patient community donated to a crowdfunding campaign in support of DMT’s academic position

Authors’ contributions

MT conceived the idea for the analysis and wrote the initial version. DMT and CS refined the arguments and structure of the manuscript and made substantial additions to the first and subsequent drafts. All authors read and approved the final manuscript.

Notes

1 A reviewer of this paper drew our attention to the study by Cho et al. [67] which argues that the placebo response is low in the CFS patient population compared to other illnesses. We contest the conclusion of this review on several grounds. Cho and colleagues used a definition of the placebo effect as the response rate in the control arm of randomized clinical trials. The authors then estimated the size of the placebo effect in CFS studies by doing a meta-analysis of the percentage of responders in the control arm of several randomized clinical trials for CFS. This method is problematic because the response rate in the control group can be affected by multiple factors, including the natural progression of the illness. The low response rate that Cho and colleagues interpret as a placebo effect could also reflect a low rate of spontaneous improvement in patients with CFS. Second, Cho et al. also included passive control conditions where patients did not receive an active intervention but were put on a waiting list or received usual care as was the case in the trial by Sharpe et al. 1996 [68] and Powell et al. 2001 [9]. In our view, the placebo response cannot be estimated if patients do not receive a credible or sham intervention. Third, the response rate of the control group is defined differently in each of the included clinical trials, and depends both on the primary outcome, and on the thresholds used to define a significant improvement. Fourth, the review only includes a limited number of CFS trials (n = 29) all published before August 2002. The review is out of date, and its finding of a lower response rate in the control group of CFS trials compared to other conditions could be due to chance and a small sample size. Recent, large, and well-conducted randomized trials for CFS point to a significant response rate in the control group. The most notable example of this is the phase III trial on rituximab which reported a response rate of 35% in the control group [69].