Time for a step change? Improving the efficiency, relevance, reliability, validity and transparency of aphasia rehabilitation research through core outcome measures, a common data set and improved reporting criteria

This article refers to:

Measuring outcomes in aphasia research: A review of current practice and an agenda for standardisation

Challenges facing COS development for aphasia

A core outcome set for aphasia treatment research: Obstacles, risks, and benefits

View addendum:

A good outcome for aphasia

OUTCOME MEASURES AND RESEARCH

Considered and meticulous outcome measurement is central to rigorously conducted effectiveness trials, and in turn the relevance and reliability of the study findings to the patient, therapist or policy maker. Failure to include valid and reliable outcome measurements is ethically questionable and wastes already limited research resources (Chalmers & Glasziou, 2009). Well-chosen outcome measures ensure that the impact of the research findings extends beyond the conclusions of the specific study. Outcome measures should not only capture clinically meaningful or functionally relevant change in stroke survivors but should also facilitate comparison to other clinical trials, clinical populations, inform meta-analyses and other synthesis approaches. Preferably, new study findings should contribute to the existing evidence base and in turn progressively grow and inform our understanding of aphasia rehabilitation. Increasingly, authors are expected to present and interpret all new research findings within the context of systematic assessment of pre-existing evidence (Clark & Horton, 2010; Glasziou et al., 2014). This can only be done effectively if there is some similarity in the choice of outcome measures and the manner in which they are recorded and reported.

As Wallace and colleagues highlight, outcome measures in aphasia research are receiving increasing interest from a variety of sources (Wallace, Worrall, Rose, & Le Dorze, 2014). In the recent Cochrane review of speech and language therapy for aphasia (SLT) after stroke, the number of outcome measures used across trials exceeded the number of trials (n = 39) included in the review (Brady, Kelly, Godwin, & Enderby, 2012). This number is further inflated if variations in test versions, the use of subscales and un-validated assessment tools are also considered. The vast (and ever-growing) volume of tools available and utilised in aphasia research settings is a reflection of the outcome measurement challenges faced across stroke rehabilitation research more generally (Ali et al., 2013; Salter, Teasell, Foley, & Jutai, 2007). The rehabilitation section of the Virtual International Stroke Trials Archive (Myzoon Ali et al., 2010) is a centralised resource of pre-existing stroke rehabilitation trial data comprised of 44 trial contributions and more than 10,000 individual patient’s data. One difficulty encountered in utilising the archive was the very limited degree of overlap in outcome measures employed across the data contributed (Ali et al., 2013; Salter et al., 2007). Over 50 outcome measures were observed across the rehabilitation trials, the most common outcome measure utilised being the Barthel index, which was used in 26 trials. However, important demographic data such as the type of stroke experienced and the time since stroke were also poorly recorded in many trials, hindering the comparison of outcomes following stroke (Ali et al., 2013). Many other organisations are contributing to the worldwide initiative to improve outcome measurement in stroke rehabilitation research including the World Health Organisation and Consensus-based standards for the selection of health measurement instruments, as Wallace and colleagues have already highlighted (Wallace et al., 2014).

Time for a step change in aphasia research

Aphasia research has faced methodological and infrastructural challenges often remaining language, region and discipline specific, thus limiting the efficiency, strength and broader relevance of any research conducted (Collaboration of Aphasia Trialists, 2013). Perhaps, some of these challenges and the nature of language and verbal behaviour itself have contributed to the vast variety of outcome measures employed across aphasia research to date. Collaborative approaches are required to produce a step change in the approach to aphasia research by collectively coordinating our research efforts and enhancing the efficiency, relevance, reliability, validity and transparency of our research. In the recent Cochrane systematic review and synthesis of 39 randomised controlled trials of speech and language therapy for people with aphasia, it became evident that there was room for more methodologically robust trial designs (Brady et al., 2012). Where data were available, trials typically randomised small numbers of participants (mean 65 (SD 68) range 5–327 individuals) from very diverse populations in terms of age (range 17–97 years), time since stroke (mean of 3.2 days up to 28 years) and aphasia severity (mild to very severe). A priori sample size calculations were only reported in 15% (6/39) of included trials. Therapy regimes were many and varied, and there was little consistency in the outcomes measured (Brady et al., 2012).

CORE SET OF OUTCOME MEASURES

The importance of coordinating and aligning data collection across research activities has been recognised amongst stroke research organisations including the National Institute for Neurological Disorders and Stroke (NINDS) in USA and the European Stroke Organisation (ESO). Facilitated by a series of rigorous consensus-building activities, these organisations have supported the development of a common set of outcome measures for acute stroke research. In particular, the ESO has recommended the inclusion of the National Institutes of Health Stroke Scale as a measure of stroke severity and the modified Rankin scale as an outcome measure in acute stroke clinical trials (Lees et al., 2012). The NINDS Common Data Element Project (CDE) (“National Institute of Neurological Disorders and Stroke (NINDS) Common Data Elements Project”; Saver et al., 2012) seeks to encourage systematic, standardised data collection procedures across neurological research studies. Their hope is that achieving a consensus on CDEs will facilitate high-quality stroke research by shortening the development time of data collection materials (and procedures), maximising the quality and relevance of the research data collected and facilitating data sharing across trials and other investigations. They outline four CDE groupings: (1) general CDEs (relevant across all neurological research studies, their use is strongly encouraged), (2) stroke CDEs (also strongly encouraged), (3) supplemental disease-specific CDEs (which may be common but not essential) and (4) exploratory CDEs (measures in development or not yet validated).

Such a CDE approach could potentially work well in the streamlining of aphasia research activities. A key overlap in CDEs shared across aphasia rehabilitation research and trials of effectiveness is the common goal of enhancing (and thus the need to accurately measure) functional communication ability in real-world settings or activities. This agenda, which is shared by patients, family members, therapists and researchers, will facilitate a common language to describe “recovery”, regardless of the nature of the intervention. Secondary outcome measures are often employed to capture impairment-based items that are more closely related in theoretical or empirical terms to the intervention itself. In the field of language rehabilitation, the absence of valid and reliable outcome measures for some languages can present significant challenges for researchers and clinicians. In addition, it is important that we distinguish between rehabilitation research outcome measures and the measurement of language status as might be used in a clinical setting to assess language profile (inclusive of multilingualism where relevant). Though many outcome measures are used clinically, recent work has highlighted the clinical use of less formal approaches to inform individualised interventions or to evaluate interim changes (Kroll et al., 2013). While improvement of language performance may have an impact on functional communication ability, we cannot assume this to be the case.

By adopting the NINDS CDE approach (“National Institute of Neurological Disorders and Stroke (NINDS) Common Data Elements Project”; Saver et al., 2012), researchers have assisted the rapid development and conduct of high-quality, coordinated and complementary acute stroke research activities. Such coordination of research effort has no doubt facilitated some of the pharmacological breakthroughs in acute stroke care (for example the use of thrombolysis) and in turn benefited patients (and their families), with measurable impacts on mortality and recovery across a range of international clinical settings. Similar efforts are currently underway in relation to outcome measures for rehabilitation research which addresses upper-limb function, cognitive rehabilitation after stroke and other chronic health conditions (“COMET (Core Outcome Measures in Effectiveness Trials) Initiative”; http://www.comet-initiative.org/).

ENHANCING METHODOLOGICAL APPROACHES

We know that the optimum design to demonstrate the clinical and cost effectiveness of an intervention is a randomised controlled trial, and many trials have been conducted in the field of SLT after stroke. Each trial involves many hours in preparation of funding, ethical and other research applications for permissions and approvals, not to mention the length of time spent developing the study design, intervention and choice of tools, approaches and procedures. Management of a trial similarly is a mammoth task, and the development of data collection tools and processes all take considerable effort. However, the efforts involved in these specific tasks are similar for a trial of 10 individuals and a trial of 200.

Typically, in the field of aphasia rehabilitation research, sample sizes have been small, many trials have experienced high attrition rates, and few have considered the extent to which rehabilitation interventions have been adhered to, as factors which may have contributed to the trial results (Brady et al., 2012). Fewer than half the included trials in the recent Cochrane review (Brady et al., 2012) described an approach to randomisation (sequence generation (38%; 15/39)) concealment of allocation (26%; 10/39) considered to be at low risk of bias. Similar findings have been made in a recent review of medical trials which demonstrated that a quarter of trials (26%) were unclear in their reporting of concealment of allocation, while 18% had inadequate approaches (Dwan et al., 2008). Just over half the trials of SLT for people with aphasia reported a procedure which ensured blinding of outcome measurement (54%; 21/39), but this rate is also considered too low (Brady et al., 2012).

ADEQUATE INTERVENTION DESCRIPTION

Descriptions of the speech and language therapy interventions delivered across the trials in the systematic review were limited, making it difficult to translate effective interventions into the “real-world” context of clinical practice, consider fidelity to an intervention and compare between different interventions (Brady et al., 2012). Speech and language therapy interventions tend to be complex, potentially requiring some degree of individualised tailoring of an intervention to address the needs of a specific stroke survivor (and their spouse). Information on the frequency, duration, theoretical approach to therapy, intensity, repetition, adherence to home-based therapy tasks and context of the intervention may all be relevant components of the description of a speech and language therapy intervention for aphasia.

The field of aphasia research is not alone in calling for better reporting of non-pharmacological interventions (Hoffmann, Erueti, & Glasziou, 2013). In a recent review of 133 randomised controlled trials of a range of complex interventions, fewer than half adequately reported the experimental intervention (Hoffmann et al., 2013). Word limits on manuscripts and other submission criteria did not alter the quality of reporting. Designing, defining and reporting complex non-pharmacological interventions is a challenge (Glasziou, Meats, Heneghan, & Shepperd, 2008), though advances continue to be made in models to support such descriptions (Hoffmann et al., 2014; Hooper, Froud, Bremner, Perera, & Eldridge, 2013; Medical Research Council, 2000). In order to facilitate and support a step change in aphasia research, we need to look to the methodological literature on the reporting of complex non-pharmacological interventions and where possible contribute to that knowledge base. Trialists, academics, funders and participating clinicians should ensure that interventions are accurately described at the start of the trial. Protocol publication is increasingly encouraged and allows detailed description of interventions and boundaries of fidelity to be specified.

INCREASING THE TRANSPARENCY OF APHASIA REHABILITATION RESEARCH REPORTS

Some trials included within the Cochrane review included a range of methodological or design components to protect their trial from potential sources of bias, but it was impossible to establish this for many of the published trial reports (despite attempting to contact the authors directly for this information, where necessary; Brady et al., 2012). Consequently, the relevant risk of bias must be considered in interpreting individual study findings. Many trials included in the Cochrane review provided incomplete reports of their outcome data (41%; 16/39) failing to provide final outcome measures on all randomised participants or to report an intention-to-treat analysis. Some trials failed to provide summary data or provided these data for only some outcome measures or participants, making it difficult to synthesise with other potentially relevant data (41%; 16/39 considered to be at high or unclear risk of bias). Few included trials reported sample size calculations or the degree to which the groups participating in the trial were comparable at baseline (18%, 7/39).

Other reporting limitations were also in evidence across the Cochrane review of SLT for aphasia (Brady et al., 2012). Inadequate description of participants in aphasia research has long been highlighted (Brookshire, 1983; Roberts, Code, & McNeil, 2003). People who experience aphasia as a consequence of stroke are a highly heterogeneous group presenting with a wide variety of personal, social, language (including multilingual), stroke and impairment profiles and co-morbidities. These aspects should be carefully considered in participant selection, data capture and reporting of findings as they may have a considerable impact on therapeutic effectiveness (Hallowell, 2008). Yet 31% (12/39) of trials in the Cochrane review failed to provide comparison data on group participants at baseline (5/12) or had groups that differed at baseline in relation to their age (4/12), time since stroke (1/12) or aphasia severity (2/12) (Brady, 2012).

Poor reporting wastes many hours of researchers, clinicians and patients’ efforts and is more likely to result in duplication of research efforts rather the efficient progression of knowledge and improvements in clinical practice. Transparent reporting informs the design and development of subsequent research activities. Building on completed research activities provides an efficient route from theoretical development, to feasibility and effectiveness studies, cost-effective implementation in clinical practice and individual patient benefit. To facilitate high-quality reporting across research studies, a number of guidelines have been developed and are now available for Enhancing the QUAlity and Transparency Of health Research (EQUATOR, 2014). These reporting guidelines include those for parallel-group randomised controlled trials (CONSORT), observational studies (STROBE), qualitative research (COREQ) and clinical case reports (CARE) (EQUATOR, 2014). Journal editors and reviewers are increasingly looking for authors to adhere to these checklists in the preparation of their manuscript and prior to publication.

Fewer than half the number of research studies conducted ever get published (Chalmers & Glasziou, 2009). The recently updated Declaration of Helsinki has expanded upon the ethical principles researchers need to abide with when working with stroke survivors. The Declaration highlights the need to register research activities in a publicly accessible database before the study commences. They also highlight the responsibility that researchers have to ensure the completeness and accuracy of their reports and to publish and disseminate their research findings (not just those that include significant results). Manuscripts failing to adhere to these recommendations may not in future be accepted for publication (World Medical Association, 2013).

COLLABORATIVE INITIATIVES

Coordination and harmonisation of international aphasia researchers’ perspectives will be required to achieve an agreement on core outcome measures, a common data set and high-quality reporting. Funded by the Co-Operation in Science and Technology (COST; www.aphasiatrials.org), efforts are underway within the Collaboration of Aphasia Trialists to establish a network across more than 25 countries of leading international multidisciplinary aphasia investigators in rehabilitation, psychology, social science, linguistics, language and other research fields (Collaboration of Aphasia Trialists). Supported by the COST funding, aphasia trialists from multidisciplinary backgrounds have created a forum to share knowledge, resources, support consensus generating activities, foster early-stage researchers, disseminate new research developments and plan coordinated transnational programmes of aphasia research. The Collaboration seeks to (1) improve our understanding of aphasia and its impact, (2) foster the development, design, planning and conduct of high-quality future aphasia programmes, (3) harmonise approaches to outcome measurement in aphasia research through consensus generating activities and thus (4) support and facilitate future collaborative aphasia research activities, (5) inform our aphasia rehabilitation programmes and (6) facilitate the reintegration of people with aphasia back into their family and communities.

CONCLUSIONS

The Collaboration of Aphasia Trialists support Wallace and colleagues’ call for the design and conduct of consensus development activities to seek standardisation of core outcome measures for aphasia research. However, we believe that this activity needs to occur in parallel with the establishment of a common data set for aphasia research and high-quality transparent reporting, which will improve the efficiency of our research efforts and inform our understanding of the relevance, reliability and validity of any research findings for people with aphasia and their clinicians. As aphasia clinicians and researchers working alongside people with aphasia, it is our moral and ethical duty to do no less.