ALSUntangled: Introducing The Table of Evidence

ALSUntangled

Patients with ALS (PALS) often consider alternative or off-label treatments (AOTs) they read about on the internet (1,2). Internet information about AOTs is not always accurate (3). In 2009, the North American ALS Research Group (ALSRG) started ALSUntangled to scientifically review AOTs and allow PALS to make more informed decisions about them (4). Our review team now consists of 95 clinicians and scientists from 10 different countries. To date we have received requests to investigate more than 160 different AOTs, and have completed 26 reviews which are available free on our website (5).

ALSUntangled has always had a protocol for conducting its reviews. This starts with attempts to contact the proponents of the AOT. Materials used to advertise the AOT are gathered to determine what claims are being made about it. A PubMed search is conducted to determine if there are relevant scientific publications. A Google search is conducted to review any related news items or blogs. The ALSUntangled review team is polled to see if any patients under their care have tried the AOT and what happened to them. The PRO-ACT database and PatientsLikeMe are queried, and if patients trying the AOT are identified, their available outcome measures before and after the AOT are reported. Attempts are made to visit the clinic offering the AOT to review its infrastructure, oversight and consent processes, interview patients who are trying it and review medical charts on the clinic's ‘best successes’. Finally, an ALSUntangled team member writes a first draft of the review and the entire team provides edits via email. When the team is satisfied, the article is submitted to Orla Hardiman, the editor of the journal Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration, for her final review and, if approved, publication.

Recently, to make our reviews even more transparent and objective and to facilitate updating them when new information arises, we decided to construct a Table of Evidence (TOE).

Methods

We found no precedent for grading the types of evidence we review, so we used a ‘crowd sourcing’ approach to create the TOE. One ALSUntangled reviewer (RB) constructed a first draft. The rest of the review team then suggested edits via emails over two months. Finally a subset of reviewers met in person and validated the utility of the TOE by attempting to convert the evidence from all 26 prior ALSUntangled reviews into this.

Results

Five iterations of the TOE occurred, eventually resulting in the current version (Table I). Reviewers ultimately selected the following categories of evidence for the TOE: Mechanism, Pre-Clinical Data, Anecdotal Data, Trials, and Risks. We decided on an ordinal, rather than numeric grading system within each category. This is because we found it difficult to ̔weigh’ one category versus another. For example, we all agreed that Trials should be worth more than any other type of evidence, but how much more? We also struggled to quantify the difference between levels within a category. For example, we agreed that two or more well-designed pre-clinical studies should count for more than 1, but could not determine exactly how much more. Specific cut-offs for achieving grades of A, B, C, D, F and U were developed and are listed in Table I.

Table I. ALSUntangled TOE.

	Grade
Evidence category	U	F	D	C	B	A
Mechanistic plausibility	Unknown	Implausible; would violate known principles or laws of biology	Acts on a biological mechanism but it is not clear than this mechanism is relevant in ALS	Theoretically and plausibly acts on an ALS-relevant mechanism in humans	Shown in a peer-reviewed publication to act on a relevant mechanism in pre-clinical model(s)	Shown in a peer-reviewed publication to act on a relevant mechanism in humans
Pre-clinical models (animal or cell models recognized by ALSUntangled reviewers to be relevant to ALS)	None	The only studies available show no benefit	One or more non-peer reviewed studies reporting benefits (published on a website or in an abstract)	One or more peer-reviewed publication(s) reporting benefits in flawed studies (*)	One peer-reviewed publication reporting benefits in a well-designed study (*)	Two or more peer-reviewed publications reporting benefits in well-designed studies (*)
Patient case reports	None	The only reports available show no benefit	Subjective report(s) of benefit without validated diagnoses and/or benefits	One unpublished report of benefit with validated diagnosis and benefits	More than one unpublished report of benefit with validated diagnosis and benefits	One or more peer-reviewed publications reporting benefits with validated diagnosis and benefits
Patient trials	None	The only trials available show no benefit	One or more peer-reviewed publications reporting benefits in a flawed trial (**)	One or more peer-reviewed publications reporting benefits in a well-designed randomized, blinded, placebo-controlled phase I or II trial	One peer-reviewed publication reporting benefits in a well-designed randomized, blinded, placebo-controlled phase III trial	Two or more peer-reviewed publications describing benefits in well-designed randomized, blinded placebo-controlled phase III trials
Risks (harms that occurred on this treatment)	Unknown	At least 5% of exposed patients experienced death or hospitalization	More than 0% but less than 5% of exposed patients experienced death or hospitalizations	At least 10% of exposed patients experienced harms (no hospitalizations or deaths)	More than 0% but less than10% of exposed patients experienced harms (no hospitalizations or deaths)	No exposed patients appear to have experienced harms

*Animal studies are assumed to be ‘well designed’ when they follow published guidelines (8). When they deviate from these they are considered ‘flawed’.

**Flawed trials means those in which there are identifiable problems with patient selection, randomization, blinding, controls or follow-up. These have ‘high or unclear risk of bias’ according to published criteria (9). Well-designed trials are those that have ‘low risk of bias’.

During the in-person meeting, information from all prior ALSUntangled reviews was translatable to the TOE, consensus between reviewers was easily obtained, and new information could be quickly incorporated into the TOE to update a review. For example, since our review on it (6), the Deanna Protocol was found to have a flawed animal study published in a peer-reviewed publication (7), so its Pre-Clinical Data score was updated from a ‘U’ to a ‘C’. All prior ALSUntangled reviews have now been converted into the TOE (Table II).

Table II. Conversion of all prior ALSUntangled reviews into TOE.

	Mechanism	Pre-Clinical	Cases	Trials	Risks
Iplex	D	B	B	D	F
Hickey Wellness	D	U	C	U	B
San Jose Tec	D	U	U	D	F
XCell Center	C	U	D	U	D
Stowe	C	U	U	U	U
Mary Murray	F	U	D	U	U
Aimspro	C	C	D	D	C
Naltrexone	D	U	D	U	B
Spirulina	C	C	U	U	A
Lutimax	C	U	D	U	B
NuTech	C	U	F	U	U
Dean Kraft	F	U	C	U	A
Bee Venom	C	C	D	U	B
MotoTab	C	U	U	U	U
Coconut Oil	C	U	D	U	B
Cannabis	C	C	D	F	U
Lyme Treatment	D	U	D	U	B
Prevagen	C	U	D	U	B
Sodium Chlorite
A. NP001	A	D	U	C	B
B. WF10	A	D	D	U	C
C. ORAL	F	U	D	U	F
Deanna Protocol	B	C	D	U	B
Fecal Transplants	D	U	D	U	B
Propofol	C	U	D	U	D
Rife Machine	F	U	D	U	B
Vitamin D	C	C	F	D	B
Ursodiol	C	U	D	D	C
Lunasin	C	U	C	U	B

Conclusion

The ALSUntangled Table of Evidence facilitates simple, objective, reliable and timely reviews of AOTs. We plan to use this for all our future reviews and hope this will make our program even more useful to PALS and clinicians.

The ALSUntangled Group Currently Consists of the following members: Richard Bedlack (who wrote the first draft of this manuscript), Colin Quinn, Chafic Karam, Alex Sherman, Lyle Ostrow, Orla Hardiman, Terry Heiman-Patterson, Laurie Gutmann, Mark Bromberg, Gregory Carter, Edor Kabashi, Tulio Bertorini, Tahseen Mozaffar, Peter Andersen, Jeff Dietz, Josep Gamez, Mazen Dimachkie, Yunxia Wang, Paul Wicks, James Heywood, Steven Novella, L.P. Rowland, Erik Pioro, Lisa Kinsley, Kathy Mitchell, Jonathan Glass, Sith Sathornsumetee, Hubert Kwiecinski, Jon Baker, Nazem Atassi, Dallas Forshew, John Ravits, Robin Conwit, Carlayne Jackson, Kate Dalton, Katherine Tindall, Ginna Gonzalez, Janice Robertson, Larry Phillips, Michael Benatar, Eric Sorenson, Christen Shoesmith, Steven Nash, Nicholas Maragakis, Dan Moore, James Caress, Kevin Boylan, Carmel Armon, Megan Grosso, Bonnie Gerecke, Jim Wymer, Bjorn Oskarsson, Robert Bowser, Vivian Drory, Jeremy Shefner, Noah Lechtzin, Melanie Leitner, Robert Miller, Hiroshi Mitsumoto, Todd Levine, James Russell, Khema Sharma, David Saperstein, Leo McClusky, Daniel MacGowan, Jonathan Licht, Ashok Verma, Michael Strong, Catherine Lomen-Hoerth, Rup Tandan, Michael Rivner, Steve Kolb, Meraida Polak, Stacy Rudnicki, Pamela Kittrell, Muddasir Quereshi, George Sachs, Gary Pattee, Michael Weiss, John Kissel, Jonathan Goldstein, Jeffrey Rothstein, Dan Pastula, Gleb Levitsky, Mieko Ogino, Jeffrey Rosenfeld, Efrat Carmi, Merit Cudkowicz, Christina Fournier, Paul Barkhaus, Eric Valor, Brett Morrison, Lorne Zinman, Craig Oster, Jerry Belsh.

Note: this paper represents a consensus of those weighing in. The opinions expressed in this paper are not necessarily shared by every investigator in this group.

Declaration of interest: ALSUntangled is sponsored by the Motor Neurone Disease Association.