Clinically meaningful change: evaluation of the Rasch-built Overall Amyotrophic Lateral Sclerosis Disability Scale (ROADS) and the ALSFRS-R

Abstract

Objective

To investigate clinically meaningful change for ROADS and ALSFRS-R using a patient-defined approach.

Methods

Data were reviewed from participants assessed at the Emory ALS Center from 2019–2022 with two assessments using both ROADS and ALSFRS-R and a completed patient-reported global impression of change scale at the second visit. Minimal important difference (MID), or the smallest amount of change that is clinically relevant, was assessed based on patient reported impression of change for ROADS and ALSFRS-R. Minimal detectable change (MDC), the smallest amount of change exceeding the threshold for measurement error, was assessed for ROADS and ALSFRS-R using standard deviations for participants self-rated as “unchanged”.

Results

Data were included from 162 participants. For ROADS (total possible normed score = 146), MID = 5.81 and MDC = 2.83 points. For ALSFRS-R (total possible sum-score = 48), MID = 3.24 and MDC = 1.59 points. Clinically meaningful decline during the assessment period was observed in 98/162 (60.49%) participants on ROADS and 75/162 (46.30) participants on ALSFRS-R (OR = 1.63, 95% CI [1.0009, 2.66]).

Conclusions

Changes that are on average less than 5.81 points (3.98%) on the normed ROADS score or less than 3.24 points (6.75%) on the ALSFRS-R sum-score may not be clinically meaningful according to a patient-defined approach. Understanding the clinical and statistical limitations of these scales is crucial when designing and interpreting ALS research studies.

Keywords:

• Amyotrophic lateral sclerosis (ALS)
• outcome measures
• Rasch-built Overall ALS Disability Scale (ROADS)
• clinically meaningful change

This article is related to:

Do we really need to calculate a minimal important difference for ALSFRS-R?: A letter in response to ‘Clinically meaningful change: evaluation of the Rasch-built Overall Amyotrophic Lateral Sclerosis Disability Scale (ROADS) and the ALSFRS-R’ published in Vol. 24(3–4), pp. 311–316

Introduction

The reported results from recent ALS trials have raised questions among the ALS community regarding both the clinical and statistical significance of study findings (1). These uncertainties highlight the importance of understanding both the clinical relevance and the statistical limitations of ALS outcome measures.

The minimal important difference (MID) is a tool for understanding the clinical relevance of an outcome measure value. MID is a measure of the smallest change on an outcome measure that is perceived by patients and/or clinicians (2). The MID is typically calculated using an anchor-based method, where the change in score on the outcome measure of interest is anchored to the patient’s or clinician’s global assessment of overall clinical status (3–5). Changes on an outcome measure that are less than the MID would be considered clinically insignificant or clinically irrelevant, regardless of statistical significance. While some works use the term minimal clinically important difference (MCID) for this metric, it has been argued that MID is a better term for emphasizing a patient-centric approach as opposed to clinically measured factors (5). Both terms are still in use, but this manuscript will use the MID terminology throughout for consistency.

Minimal detectable change (MDC) is a tool for understanding the statistical limitations of an outcome measure. MDC is defined as the smallest amount of change that can be detected by an outcome measure that exceeds the threshold for measurement error (6). The MDC is calculated using a standardized formula that includes the standard deviation and reliability of the outcome measure; it does not provide information about clinical relevance (7). MDC can be conceptualized by thinking of a digital scale with perfect reliability that only displays pounds in whole numbers. In this case, differences of 0.5 pounds cannot be reliably measured with this tool, regardless of whether or not that amount is clinically important. This scale could be an appropriate measurement tool for an adult obesity study where values of less than 1 pound are not of clinical interest, but this scale would be an inadequate measurement tool for a study on weight changes in premature infants, where changes less than the MDC are clinically important. From a statistical standpoint, changes that are smaller than the MDC value cannot reliably be assessed.

For ALS research, many clinical studies rely on the ordinal revised ALS Functional Rating Scale (ALSFRS-R) to detect clinical changes (8). There is currently no consensus on how to define a clinically meaningful change on the ALSFRS-R (9), but a survey study of 65 expert ALS clinicians is often cited as the rationale for accepting a 20–25% change in slope of the ALSFRS-R as “at least somewhat clinically meaningful” (10). This approach does not take into account the patient’s view of clinically meaningful change, nor does it consider the minimum amount of change that needs to occur to surpass what can be seen due to measurement error. The Rasch-built Overall ALS Disability Scale (ROADS) is a newer patient-reported outcome measure with mathematical advantages over the ordinally-constructed ALSFRS-R (11), but to date, clinically important changes on the ROADS have not been studied.

This study examined a patient-defined approach for understanding clinically meaningful change by quantifying minimal important difference (MID) and minimal detectable change (MDC) for both ROADS and ALSFRS-R.

Methods

The Emory ALS Clinic database is a prospective database for studying the natural history of ALS. It contains detailed demographic and outcome measure data for all patients seen at the Emory ALS Center, both virtually and in person, since 1997 (12). Data were reviewed for patients seen between September 2019 and March 2022, with September 2019 selected as the starting point based on the timing of ROADS creation and validation. Patients were included in the analyses if they had a diagnosis of ALS confirmed by an Emory ALS Center physician, if they had at least 2 assessments that included simultaneous administration of the ALSFRS-R and ROADS, and if they had a patient-reported global impression of change scale recorded at the time of the second assessment. The ALSFRS-R and ROADS are routinely collected every patient at each visit at Emory as part of clinical care. The patient-reported global impression of change scale was added as a routine clinical assessment in 2020 for the purposes of studying clinically meaningful change.

The ALSFRS-R is an ordinal scale contains 12 items, each scored 0–4, and in a clinical trial setting is typically completed by an evaluator based on patient report (13). ALSFRS-R was administered by a research coordinator or nurse who had completed a standardized training protocol for administering the scale. Most participants had different raters at each visit, and therefore a test-retest reliability value of 93% was used for minimal detectable change calculations based on prior studies of inter-rater reliability (14).

The Rasch-built Overall ALS Disability Scale (ROADS) is a patient-reported outcome measure that assesses 28 items, each scored 0-2, and then provides a normed total score from 0-146 to capture overall disability level in a linearly-weighted manner, meaning that a 1-point change on the normed ROADS score represents a quantifiable and consistent measurement of disability, and a 2-point change reflects twice the amount of disability (11). For in-person visits, the ROADS was completed on paper by the participant. For telemedicine visits, the ROADS was administered by telephone. Staff administering the ROADS by telephone did not provide coaching or guidance regarding individual questions or response selection, but if asked would repeat the verbatim instructions. When needed, staff members or caregivers provided assistance marking answers or verbalizing responses, but answer responses were provided by the patient. Test-retest reliability for the ROADS administered on paper has been previously reported at 97% (11). A subset of participants completed the ROADS both in paper and by telephone within 7 days to calculate the telephone test-retest reliability; this value was used in minimal detectable change calculations.

To measure the global impression of change (15,16), patients were asked by the research coordinator or nurse “with respect to your ALS, how would you describe yourself today as compared to your last assessment on <date of last ROADS and ALSFRS-R assessment>: better, unchanged, a little worse, or a lot worse?” Patients were not provided with their previous ROADS or ALSFRS-R scores. Patients were not given additional instructions to define “with respect to your ALS”. The global impression of change scale was not explicitly discussed with patients at the baseline assessment. Given the paucity of “better” responses, the “better” and “unchanged” responses were combined as a single category for all analyses.

Baseline characteristics of the study cohort were characterized using descriptive statistics, including age at first visit, disease duration at first visit, sex, race, ethnicity, site of symptom onset, time between assessments, and baseline ROADS and ALSFRS-R scores. All ROADS calculations used the normed score to maintain linear weighting (11). Mean rate of decline for ROADS and ALSFRS-R per month was calculated with the following formula: ((Second scale measurement) – (first scale measurement))/(time in months between assessments).

All statistical analyses were performed using the Microsoft Excel Version 2018 (Microsoft, WA, USA) and GraphPad Prism Version 8.4.2 (GraphPad Software, San Diego, California, USA).

Minimal important difference (MID)

MID for ROADS and ALSFRS-R was assessed using an anchoring approach, with the patient-reported global impression of change scale serving as the anchor. In order to conceptually capture the minimum difference between two adjacent levels on the patient impression of change scale (17), MID for each scale was calculated using the following formula: $$\text{MID}=│((\text{mean change in total score between visit}\mathrm{ }2\mathrm{ }\text{and visit}\mathrm{ }1\mathrm{ }\text{for participants who responded}\mathrm{ }“\text{a little worse}”\mathrm{ }\text{on the global impression of change scale})\mathrm{ }–\mathrm{ }(\text{mean change in total score between visit}\mathrm{ }2\mathrm{ }\text{and visit}\mathrm{ }1\mathrm{ }\text{for participants who responded}\mathrm{ }“\text{unchanged}”\mathrm{ }\text{or}\mathrm{ }“\text{better}”\mathrm{ }\text{on the global impression }\text{of change scale}))│.$$

Minimal detectable change (MDC)

Minimal detectable change (MDC) determines the smallest amount of change on an outcome measure that surpasses what could be seen due to measurement variability. MDC was calculated for ROADS and ALSFRS-R using the following standard formula: $$\text{MDC}\mathrm{ }=1.96\mathrm{ }\times \surd 2\mathrm{ }\times \left(\text{standard error of measurement}\right).$$

Standard error of measurement was calculated by using the following formula: $$\left(\text{standard error of measurement}\right)\mathrm{ }=\mathrm{ }\left(\text{standard deviation}\right)\mathrm{ }\times \mathrm{ }\left(\surd \left(1-\text{intraclass coefficient}\right)\right)$$

(3,4,18).

For this study, ROADS intraclass coefficient was calculated by assessing test-retest reliability in participants who completed the questionnaire by both telephone and on paper within a 7-day period, and historical data were used for the ALSFRS-R.

MDC is smaller when standard deviation is low and reliability of the outcome measure is high. For this calculation, the standard deviation was calculated based on change in ROADS and ALSFRS-R scores for participants that rated themselves “unchanged” or “better” on the global impression of change scale. This MDC reflects the variability and measurement error within the group of patients that did not experience clinical changes, rather than reflecting the disease heterogeneity of the entire cohort, and this approach allows the MDC calculation from the entire cohort to be applied to other study populations.

Standard protocol approvals, registrations, and patient consents

This study was approved by the Emory Institutional Review Board. Participants provided informed consent for an ALS natural history study and for inclusion in the database. All Emory ALS Center patients at any stage of disease are recruited for participation.

Results

One hundred and sixty-two ALS patients were included in the analysis. This unselected cohort represents the wide range of clinical disease severity within this population. The mean age of symptom onset was 59.75 (SD = 11.12), with mean disease duration at first assessment of 3.16 years (SD = 3.38). The average time between the two assessments was 6.41 months (SD = 4.21). The mean baseline score at first assessment was 83.60 for ROADS (SD = 20.13) and 31.54 for ALSFRS-R (SD = 9.17). The mean decline per month was −2.03 points per month on the normed ROADS score (SD = 2.44) and −0.76 points per month on the ALSFRS-R total sum score (SD = 1.09). Additional cohort characteristics are included in Table 1.

Table 1 Characteristics of the 162 participants with ALS included in analyses.

Parameter (continuous variables)	Mean (SD)
Age of onset (years)	59.75 (11.12)
Disease duration at first assessment (years)	3.16 (3.38)
Baseline ALSFRS-R score	31.54 (9.17)
Baseline normed ROADS score	83.60 (20.13)
Time between assessments (months)	6.41 (4.21)
Mean decline in ALSFRS-R per month	−0.76 (1.09)
Mean decline in normed ROADS per month	−2.03 (2.44)
Parameter (categorical variables)
Site of symptom onset
Bulbar	38 (23.46%)
Upper extremity	61 (37.65%)
Lower extremity	62 (38.27%)
Diaphragm	1 (0.06%)
Sex
Female	66 (40.74%)
Male	96 (59.26%)
Race
Asian	3 (1.85%)
Black or African American	22 (13.58%)
Pacific Islander	1 (0.62%)
White	130 (80.25%)
Other	1 (0.62%)
Unknown	5 (3.09%)
Ethnicity
Hispanic/Latino	4 (2.47%)
Non-Hispanic/Latino	148 (91.36%)
Unknown	10 (6.17%)

Patient reported global impression of change

Comparing their disease at the time of the second assessment to the time of the first assessment, four (2.47%) participants classified themselves as “better”, 24 (14.81%) as “unchanged”, 86 (53.09%) as “a little worse”, and 48 (29.63%) as “much worse”. The “better” and “unchanged” categories were merged into a single category for further analyses.

On the ROADS, the mean decline in normed score was −3.64 points (SD = 4.56) for patients who rated themselves as better or unchanged, −9.45 points (SD 9.77) for patients who rated themselves as a little worse, and −17.56 points (SD = 12.52) for participants who rated themselves as much worse (Table 2).

Table 2 Patient-reported global impression of change results.

Patient-reported global impression of change n (%)	Change in ROADS mean (SD)	Change in ALSFRS-R mean (SD)
Better/Unchanged 28 (17.28%)	−3.64 (4.56)	−0.64 (2.17)
A little worse 86 (53.09%)	−9.45 (9.77)	−3.88 (3.98)
A lot worse 48 (29.63%)	−17.56 (12.52)	−7.21 (6.08)

On the ALSFRS-R, the mean decline in sum-score was −0.64 points (SD = 2.17) for patients who rated themselves as better or unchanged, −3.88 points (SD 3.98) for patients who rated themselves as a little worse, and −7.21 points (SD = 6.08) for participants who rated themselves as much worse (Table 2).

Minimal important difference

Based on the difference between mean change in score for participants who rated themselves as a little worse compared to unchanged/better, MID = 5.81 (3.98%) for ROADS (based on the normed score, total possible points = 146) and MID = 3.24 (6.75%) for ALSFRS-R (based on 48 possible points). In other words, average changes on the normed ROADS score of less than 5.81 points or changes on the ALSFRS-R sum-score of less than 3.24 points may be considered not clinically meaningful.

Minimal detectable change (MDC)

A subset of 37 participants completed the ROADS by both telephone and on paper within a 7-day period; the intraclass coefficient (ICC) was 0.95. For MDC calculations, ICC = 0.95 was used for ROADS and 0.93 for ALSFRS-R based on prior data (14,19). The standard deviation for change in score during the assessment period among patients who rated themselves as unchanged/better was 4.56 for the ROADS and 2.17 on the ALSFRS-R. Using these ICC and standard deviation numbers, MDC = 2.83 points for ROADS and 1.59 points for ALSFRS-R. In other words, average changes on the normed ROADS score of less than 2.83 points or changes on the ALSFRS-R sum-score of less than 1.59 are within the bounds of measurement error.

In this cohort, the number of patients that showed clinically meaningful decline by exceeding the threshold for both MID and MDC during the assessment period was 98/162 (60.49%) on the ROADS and 75/162 (46.30%) on the ALSFRS-R (OR = 1.63, 95% CI [1.0009, 2.66]).

Discussion

This study provides a patient-driven approach for understanding clinically meaningful change on the ROADS and the ALSFRS-R. A strategy based on patient perspectives allows clinicians and patients to better interpret the relevance of ALS research findings and facilitates an informed approach to shared decision making when patients weigh the benefits, risks, and burdens of various treatment options or experimental therapies. This is particularly relevant in the setting of emerging costly therapies for ALS that may be of modest clinical benefit (1,20,21). Though this type of analysis has not been done previously, the ALSFRS-R is currently used to define the effectiveness of new drugs in ALS clinical trials. These newly measured parameters for clinically meaningful change should inform future clinical trial design; changes in ROADS and ALSFRS-R that are less than MDC values should be considered within the realm of measurement variability, and changes in ROADS and ALSFRS-R that are less than MID values should be scrutinized for clinical relevance. A strength of this study is that it was performed in a representative ALS clinical population, which supports generalizability of the findings.

This study does have limitations. The mean duration between study visits was 6.41 months, and future longitudinal data collection with less time between visits will provide a better understanding of short-term responsiveness, particularly for the ROADS, which is a newer outcome measure. Studies of clinically meaningful changes for other scales have utilized shorter intervals between assessments to reduce recall error (22), and therefore future studies with shorter intervals in between assessments would strengthen this line of work. When participants were asked to provide a global impression of change response “with respect to your ALS”, it is possible that they were considering factors other than physical disability of function, such as mood or quality of life, capturing factors that ALSFRS-R or ROADS are not intended to measure. This study also did not include formal cognitive testing, and thus the role of cognitive impairment on the study findings cannot be addressed in this work. Additionally, this work is from a single tertiary ALS clinic, and more data are needed to support the external validity of the findings. While this population is representative of a clinical cohort, it is different than a typical clinical trial population. In particular, this patient population has longer disease duration and more advanced progression compared to a typical interventional clinical trial population. Another consideration is that the ALSFRS-R is not linearly-weighted, meaning that a one-point change does not represent a specific unit of functional decline (23,24), and prior work has shown that the ALSFRS-R declines in more of a curvilinear manner (25). Therefore, the MID for ALSFRS-R in reality is likely not a rigidly fixed value across the range of the scale. For example, a change in the ALSFRS-R from 48 to 45 does not necessarily indicate the same level of clinical functional loss as a change from 26 to 23 or a change from 5 to 2. The sample size for this study did not permit subgroup analyses at different ranges of the scale. The normed ROADS score is mathematically linearly-weighted, and thus MID is expected to be more consistent across the range of the scale.

When designing future ALS clinical trials that rely on the ROADS or ALSFRS-R, it is important to consider the expected progression of the research population over the study period to ensure that participants are expected to decline by a clinically meaningful amount over the study period. If a study drug is not expected to lead to frank improvement, then clinically meaningful treatment effects will not be observed in study participants that do not progress by an amount exceeding the MID on the primary outcome measures. Trials that aim to demonstrate clinically meaningful change will likely need to consider statistical enrichment techniques to select for a faster progressing cohort, extend the duration of the study period to allow clinically meaningful change to occur, or employ a combination approach.

Although no one method for determining clinically meaningful change is universally accepted as best, this study provides a compelling patient-driven approach for considering 5.8 points on the normed ROADS score and 3.2 points on the ALSFRS-sum-score as the mean minimum threshold for clinically meaningful change.

Acknowledgements

The authors thank the patients and families of the Emory ALS Clinic and the staff of the Emory ALS Clinic.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Data availability statement

All study data analyses are included within this manuscript. An anonymized version of the dataset used for conduct of this study will be shared upon request to a qualified investigator with institutional review and approval.

Additional information

Funding

This study was funded by the Department of Defense under [grant W81XWH-21-1-0124] and the Department of Veterans Affairs under [grant IK2CX001595-02].