Operationalising treatment success in aphasia rehabilitation

ABSTRACT

Background

Treatment success is the desired outcome in aphasia rehabilitation. However, to date, there is a lack of consensus on what defines a 'successful' result on a given aphasia outcome measurement instrument (OMI).

Aim

In this methodological paper, we present strategies for how to define and measure treatment success on a given OMI at the group-level, as well as for an individual person with aphasia. The latter is particularly important when research findings from group studies are clinically implemented for individuals in rehabilitation.

Scope

We start by presenting methods to calculate the average statistically significant change across several (group) studies (e.g., standardised mean difference, raw unstandardised mean difference) for a given OMI. Such metrics are useful to summarise the overall effect of the intervention of interest, particularly in meta-analyses. However, benchmarks based on group effects are not feasible for assessing an individual participant’s treatment success and thus for determining the proportion of patients who had a beneficial response to therapy (overall response rate of an intervention). We therefore recommend a distribution-based approach to determine benchmarks of a statistically significant treatment response at the individual level, i.e., the 'smallest detectable change' for a given OMI, which refers to the smallest change that can be detected by the OMI beyond measurement error. However, the statistical significance of an individual treatment effect does not necessarily correspond to its clinical impact. This requires an additional indicator. The benchmark to determine a clinically relevant improvement on a given OMI is the 'minimal important change'. The minimally important change is defined as the smallest OMI change score perceived as important by the relevant stakeholder group (i.e., people with aphasia, their relatives/caregivers, clinicians). It therefore requires relating the individual OMI change scores to 'anchors', i.e., meaningful external criteria, preferably based on patient-perceived therapy success. Currently, there is no consensus regarding the optimal 'anchors' and their respective definition of clinically important change in aphasia outcome research.

Conclusions/Recommendations

Operationalising individual treatment success based on both statistically significant and (patient-reported) clinically meaningful benchmarks is a key priority in aphasia rehabilitation. Availability of such measures will (a) facilitate estimates of therapy response rate in intervention studies and thus optimise therapeutic decisions and (b) provide stakeholder groups (e.g., the society, the stroke team, people with aphasia, family, clinicians, healthcare professionals) with objective, statistically reliable and meaningful feedback on individual treatment response in the clinical setting.

KEYWORDS:

• Aphasia
• treatment effectiveness
• patient-relevant outcome
• benchmarking healthcare
• individualised medicine

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. A patient-defined approach focusses on patients’ treatment expectations in deciding what constitutes a successful treatment outcome (Zeppieri & George, 2017). A 'patient-defined' outcome may be a 'patient-reported' outcome, but self-reporting is not a critical requirement here.

2. An additional more recent method to evaluate average intervention effects, yet based on small-N designs (e.g., case-control series), is linear mixed-effects modeling, a type of multiple regression analysis. This method also yields a standardised treatment effect estimate for an OMI, but requires frequently repeated (e.g., daily) assessments of the OMI over the course of the treatment (Wiley & Rapp, 2019).

3. Basic requirements for reliability estimation in classical test theory are parallel (or less strictly tau-equivalent) measures, i.e., (1) stability of subjects’ 'true' scores (traits) across repeated assessments with the same test (or a constant 'true' score change for all individuals of the reference population) and (2) the test administrations have identical error variances and thus observed score variances in the reference population (for tau-equivalent measures the error variances may differ).

4. In case of rating scales with two or more raters the computation of generalizability coefficients or kappa coefficients was proposed by the COSMIN group (Mokkink, Boers, van der Vleuten, Patrick, et al., 2020).

5. Other variants of the ICC (consistency/single measure variant or average variants) as well as the Pearson correlation coefficient are less suitable because systematic score level differences between assessments are of no concern (McGraw & Wong, 1996; also cf., Mokkink, Boers, van der Vleuten, Patrick, et al., 2020, p. 37).

6. Alternatively, the split-half reliability coefficient with 'upgrading' to the full test length via the Spearman-Brown formula may also be applied. Parallel versions of the same test are less frequent in aphasia rehabilitation, but the Pearson correlation between parallel versions would also be adequate. For either reliability estimate, the model of 'essentially tau parallel' measures needs to hold. If only the weaker model of 'tau-congeneric' measures holds, McDonald’s omega is the reliability estimate of choice (Padilla, 2019).

7. An alternate option is that parallel versions of a test may be administered for repeated assessments to disentangle changes in 'true' scores from measurement error (e.g., using bi- factor models; Mokkink, Boers, van der Vleuten, Bouter, et al., 2020, p. 31).

8. More precisely, this is the definition for the standard error of measurement = SEmeas according to (McManus, 2012), which needs to be differentiated from the other two types of SEm, which are the standard error of estimation = SEest and the standard error of prediction = SEpred.

9. The WAB-R AQ benchmark for a statistically significant individual change score will presumably be higher in acute stroke samples because of greater sample variability and thus a larger SEm. This needs to be addressed in future studies.