Adapting the stroke therapy outcome measure for use by Multidisciplinary Teams: considering reliability

Abstract

Purpose

The Therapy Outcome Measure (TOM) comprises many profession and client group-specific adapted scales based on the International Classification of Functioning Model, used by different Health Care Professionals (HCPs). A new Multidisciplinary stroke TOM scale (MDT Stroke TOMs) was developed by amalgamating the relevant scales. This study aimed to develop a scale to describe patients’ abilities and improve communication and collaboration within the stroke Multidisciplinary Team (MDT). The present study investigated the inter-rater and intra-rater reliability of this adaptation.

Materials and Methods

The MDT Stroke TOM Scale was developed by combining and rewording four profession-specific adapted TOM scales following iterative trials and discussions leading to an agreed version. 150 stroke Allied Health Professional (AHPs) and nurses were trained virtually in TOM and the MDT Stroke TOM Scale. Ten AHP and nursing raters from five professions evaluated ten composite stroke case studies for inter-rater reliability. Six participated in the intra-rater reliability review of the same three case studies at different time points. Reliability was quantified using intraclass correlation coefficients and measurement error using the smallest detectable change.

Results

There was good to excellent concordance between the raters in terms of scoring. Raters’ scores were also consistent. The MDT Stroke TOMs is being used consistently across the South West London Stroke Network.

Conclusion

The Stroke MDT TOMs is psychometrically robust and should assist with improving team work in the rehabilitation of stroke patients and their carers. Future research may use MDT Stroke TOM to promote interprofessional team working and facilitate communication in Stroke Rehabilitation.

IMPLICATIONS FOR REHABILITATION

• The introduction of a new Multidisciplinary Team (MDT) Stroke Therapy Outcome measure to encourage interdisciplinary working and collaboration.

• One measure to complete and document for busy MDTs, which has good - excellent reliability.

• Early and timely MDT discussions regarding patient care and needs.

• Using this reliable outcome measure facilitates audit and benchmarking of services.

Keywords:

• Stroke
• rehabilitation
• therapy outcome measures
• reliability
• Multidisciplinary
• communication
• interprofessional

Introduction

Stroke is a leading cause of serious, long-term disability, the effects of which may have physical, cognitive, emotional, and social consequences, not only for those affected, but also for their family [1]. The majority (63.8%) of stroke patients had a physical, cognitive, or communicative disability immediately after the incident, which required different health professionals to work collaboratively to bring their collective knowledge and specialist skills for effective rehabilitation [2]. A range of health professionals are required to work together in rehabilitation with effective communication and shared objectives. Stroke teams often include a range of allied health professionals (AHPs) (physiotherapists, occupational therapists, speech and language therapists, dieticians, and rehabilitation assistants), psychologists, nursing, and medical staff. Stroke team members support stroke survivors in practices related to improving independence in Activities of Daily Living (ADLs), increasing function, mobility, swallowing, communication, social participation, and well-being of patient and carer [3].

Multidisciplinary Teams (MDTs) have been established as best practice for optimal patient care across many health conditions [4–7]. Inclusion of a community stroke rehabilitation nurse provides a unique and fundamental role to the team that strengthens the model of interdisciplinary teamworking [8]. In a qualitative observational study of teams working in two rehabilitation stroke units, Clarke [9] found that a high level of MDT work was achieved primarily through learning from and about each other’s rehabilitation practices. “Regular problem-oriented and opportunistic dialog between team members led to stroke-and therapy-specific language being increasingly shared” [9]. The author continues by explaining that a team should preferably use an agreed-upon consistent approach by team members for each problem faced by a patient, ensuring that the patient is given consistent advice and taught the same strategies to ameliorate or overcome the challenges they face. They argued that providing a shared framework to share the objectives of rehabilitation and measure outcomes would assist in improving communication and multidisciplinary work.

The World Health Organisation developed the International Classification of Functioning, Disability, and Health (ICF) model [10], in order to establish a common language for describing health and health-related states to improve communication. It is accepted as an approach appropriate for all professions within rehabilitation services with respect to the concepts of functioning, disability, and health, providing a framework from biological, individual and social perspective [11].

The Therapy Outcome Measure (TOM) [12] for Rehabilitation Professionals [13–19]⁹ are based on the ICF and were designed to be a simple, reliable, cross-disciplinary, and cross-client group method of gathering psychometrically robust information on a broad spectrum of issues associated with therapy/rehabilitation to guide intervention and identify changes over time. The TOM was developed following review of the goals of rehabilitation by different AHPs and which were found to cover:

• improving the disorder,

• reducing dependence,

• improving social participation

• impacting on the well-being of the individual and carer.

Three of the four TOM dimensions (impairment, activity, participation) map well onto the ICF however a further dimension of “well-being” (patient and carer) was added to the TOMs despite not being represented on the ICF as many goals in rehabilitation address that domain. Approximately 30% of stroke patients suffer from depression at some point post-stroke, with a significant proportion remaining undiagnosed or inadequately treated [20]. This number increased to 60% in patients with post stroke aphasia [21,22]. The need for early mood screening as part of the multidisciplinary assessment is imperative, hence the importance of the “Well-Being” section of the MDT Stroke TOMs, as an adjunct to the ICF framework [10].

The TOM is rated on an ordinal rating scale, with “0” representing the severe end of the scale and “5” representing normal for the persons age, sex, and culture. Half-points increase the scale to 11 points, allowing for further flexibility and sensitivity in scoring. The integers are defined with a semantic operational code that identifies the severity of the difficulty experienced in each domain with half-points, allowing the assessor to indicate whether the person is slightly better or worse than the descriptors associated with all points [18]. The ratings were entered at the beginning and end of the rehabilitation episode of care [12]. The development of 67 adapted scales with amplified descriptors for use by different healthcare professionals working with a broad range of client groups improved inter-rater and intra-rater reliability [19]. TOM takes a short time (approximately three minutes), as ratings are assigned after the individual has been interviewed, assessed, and observed in the usual manner. TOM is used to summarise the clinician’s view of the patients’ impairment, activity, participation, and well-being.

There are separate TOM-adapted scales used by a range of healthcare professionals to reflect dysphagia, communication impairments, mobility, cognition, and physical stroke impairments [23]. This study details a further adapted Stroke TOM scale for use by the multidisciplinary team working together to reduce recording time, improve working and cohesion, and facilitate communication between team members. It is a holistic outcome measure that addresses all areas of stroke recovery, including continence, and fatigue, providing a picture of a patient’s abilities before, during and after treatment.

In this paper, we describe the development of a TOM-adapted scale suitable for multidisciplinary team use and the investigation of its inter-rater and intra-rater reliability.

In community stroke settings, the National Service Model for an Integrated Community Stroke Service (UK) outlines that stroke therapy should be offered in the community based on “clinical need tailored to goals and outcomes” ^[24,25]. It also describes how patients should have a “personalised plan,” which allows them to own their own goals to encourage more participation and self-management. Collecting information on a patient’s needs when taken on by a service would create a collective narrative about the patient’s difficulties and goals for treatment, allowing interventions to be shared and outcomes to be reviewed. Numerous studies and guidelines have outlined the necessity of timely communication between MDT members along the stroke care pathway [7, 24–26].

The nature of the communication and interactions between team members defines if a team is working in a multi- or inter- disciplinary way. The definition of MDT working is when members of different professions are working together but still approach patients from their own perspective. In contrast, Interdisciplinary Team (IDT) working is an integrated approach of different disciplines with a significant level of collaboration and communication with an “agreed or shared strategy” [27]. The MDT Stroke TOMs encourages early and continuing communication about the patient’s abilities, the rehabilitation focus, and goals, which dictate their scores on the scale. This supports the idea that using the MDT Stroke TOMs may redirect MDT working to become more IDT focused as communication is paramount [27].

Material and methods

Setting

The Croydon Health Services NHS Trust is part of the Southwest London Health Service Network. The Early Supported Discharge (ESD) team for stroke patients in Croydon, works closely with other teams along the stroke care pathway and with neighbouring boroughs, sharing resources and training. This aims to rationalise the stroke and neurological community rehabilitation services, as there was no streamlined approach to identify patients’ needs, and no consensus on outcome measures being used. This highlighted the need for the design of a more holistic measure to assess and review patients with a diagnosis of stroke, providing the opportunity of sharing and comparing outcomes.

Development of the MDT Stroke TOM Scale

Online feedback sessions with a range of 12 AHPs and nursing colleagues regarding the pre-existing TOM stroke scale highlighted that it lacked any representation of patients with cognitive (20-80% of stroke patients28], communication (approximately 30-40% of stroke patients29], or swallowing difficulties (> 50% of stroke30]. They reported feeling that patients were not represented appropriately without using multiple scales to describe the effects of their stroke, which was “time-consuming” and “more difficult to collate”.

The separate TOM-adapted scales appropriate for use with stroke patients (dysphagia, dysphasia, mobility, and cognition) by different members of the MDT were reviewed by a lead speech and language therapist and physiotherapist. They had numerous discussions regarding blending TOM-adapted scale descriptors and adding to and rewording the previous Stroke TOM Scale, which was primarily physical in nature. An example is illustrated here for someone with a moderate level of stroke impairment (see Table 1).

Table 1. Example of change of “stroke TOM scale” to “MDT stroke TOM scale”.

Original Stroke Scale: “Level 3 Active movement against gravity: controlled isolated movement, occasional associated reactions, moderate sensory inattention, movements may not be accurate, or one severe sign/symptom, for example dense hemiplegic arm, or two moderate signs/symptoms, for example moderate arm/leg hemiplegia.” [23].

New MDT Stroke Scale: “Level 3 Moderate degree of signs and symptoms (could be one severe, or multiple mild – moderate signs/symptoms). For example, active movements against gravity, actively cooperative, reduced dynamic standing balance, controlled isolated movements with occasional associated reactions, moderate sensory inattention, moderate apraxia. Severe signs could be dense hemiplegic arm/swallowing/communication/cognition or a combination of milder signs. Actively contributes to therapy.”

Once a consensus was reached about the wording of the scale, it was trialed by the Croydon Stroke Rehabilitation Team and subsequently by the wider Stroke Community Southwest London Network. Further virtual discussions with the lead MDT therapists in 4 separate stroke teams, led to further changes in the descriptors until members of MDT reached agreement. The lead author amalgamated the changes and produced a final version. This version was rolled out, at the end of 2022, across the South West London Network as the MDT Stroke TOM Scale (See Appendix 1, supplementary material). Interestingly, the opportunity for these MDT discussions facilitated better understanding between the different disciplines regarding their areas of interest and expertise.

Inter-rater and intra-rater reliability study

Between June, 2021 and September, 2022, 150 different Allied Health Professionals, psychologists, and nurses across the South West London Network were trained virtually on the theory and use of TOM scales and the newly adapted MDT Stroke TOM (see Figure 1). In this training, they learned about the importance of outcome measurement and data, and its relevance when working in healthcare. They were also specifically trained in how to use the TOM scales, and differentiate between the four domains (i.e., impairment, activity, participation and well-being). They all attended subsequent inter-rater scoring sessions online, where attendees could blindly practice scoring composite case studies to assess agreement and improve overall confidence in scoring.

Figure 1. Professions trained in South West London in TOMs.

Sample size

Intraclass correlation coefficients (ICCs) are the most used method for assessing the reliability of continuous or ordinal data with more than four or 5 categories [31]. In this study, sample size estimation was conducted using the method described by Bonett [32]. Assuming inter-rater ICCs in the magnitude of 0.80 to 0.85, ten raters assessing 8-12 case studies would give an acceptable confidence interval width of ± 0.15.

Participants

Following training, the participants (HCPs) were invited to participate in the reliability study via email and provided written consent. Ten participants were recruited: two speech and language therapists, three physiotherapists, three occupational therapists, one assistant psychologist, and one nurse working in community stroke teams. It was important to have a representation of the full MDT to assess the scales appropriacy and versatility across professions.

The participants comprised of a mixture of different levels of experience (ranging from bands 6 to 7). Band 6 members had one to four years’ experience working in stroke and had limited exposure to using TOMs. The Band 7 members had five to ten years working in stroke and had experience using TOMs, particularly the MDT Stroke TOMs. The 10 MDT members participated in the inter-rater reliability trial and six members also conducted the intra-rater reliability.

Procedure

Reliability assessments were conducted by 10 MDT raters independently scoring previously prepared composite stroke patient case studies (an example case study is provided in Appendix 2, supplementary material) using the MDT Stroke TOMS Scale. They needed to give a score for the “stroke patient’s” level of impairment, activity, participation and well-being using the 11-point ordinal scale without collusion. Scores were compared for reliability between raters.

For the assessment of intra-rater reliability, six raters rated the same three composite case studies at two different time points. The raters were asked to rate the case studies live, unaware they would be asked to repeat this task 3 months later when they were asked to rate the same case study in their own time via email. This project which was part of an audit used composite case studies (see Appendix 2, supplementary material); therefore, ethical approval was not required by the Trust.

Statistical analysis

Inter-rater reliability was quantified using ICCs derived from two-way random-effects models for single-measure absolute agreement, with separate models fitted for each TOM domain. Intra-rater reliability was determined using ICCs from nested random-effects models to account for the multiple paired ratings provided by each rater, with separate models fitted for each TOM domain. The 95% confidence intervals were calculated for each ICC. The standard error of measurement (SEM) was quantified using the root mean square error term from analysis of variance tables and converted to the smallest detectable change (SDC) at a 95% confidence limit using the formula: 1.96 * √2 * SEM. All analyses were conducted using Stata 18 [33].

Results

In the inter-rater reliability assessment, the difference between the lowest and highest ratings assigned by the ten raters did not exceed “1” on the TOM scale for any domain. For intra-rater reliability, the difference between paired ratings did not exceed “0.5” points.

The ICCs and corresponding 95% confidence intervals for both inter- and intra-rater reliabilities are provided for each TOM domain (Table 2).

Table 2. Inter- and intra-rater ICCs.

	Inter-rater reliability	Intra-rater reliability
Impairment	0.89 (0.78, 0.97)	0.90 (0.73, 0.97)
Activity	0.85 (0.71, 0.95)	0.81 (0.58, 0.93)
Participation	0.90 (0.80, 0.97)	0.87 (0.61, 0.97)
Wellbeing	0.93 (0.86, 0.98)	0.97 (0.87, 0.99)

The SDCs taking into account the measurement error from the inter-rater assessments ranged from 0.65 to 0.86 on the TOM scale. The SDCs accounting for intra-rater error ranged from 0.33 to 0.61 points.

Discussion

This reliability study indicated that that the MDT Stroke TOMs is a reliable outcome measure, with ICCs ranging from “good” to “excellent” agreement; thus, the measure can be used with confidence by any member of the stroke MDT. The SDCs suggest that individual changes in TOM pre-post therapy would need to be of a magnitude of “1” or greater to be confidently considered “true” from a measurement perspective and not potentially due to measurement error.

The MDT Stroke TOMs provides the multidisciplinary team with the opportunity to collaborate at an early stage in the patient’s rehabilitation journey by identifying the patient’s limitations, retained abilities, strengths and needs. It provides opportunities for liaison, planning of goals, agreement on therapeutic approaches, and information sharing regarding optimal care as well as the role of each member of the team with a particular patient. It aims to transition MDT working to a more IDT approach through maximising collaboration between HCPs. We know that service users perform better when individuals work more effectively as an integrated team, which amalgamates all the skills, knowledge and expertise of each discipline. Good teamwork can produce a greater effect than the sum of the individual efforts [27].

Feedback from MDT teams

Members of the MDT teams commented that they felt it was a helpful tool to encourage early and specific communication about patients’ strengths and needs, making goal planning and targeted rehabilitation easier to share. One Physiotherapy colleague stated,’ Had we not discussed his MDT Stroke TOM score in the first week, I would have continued treating this man’s physical difficulties, not understanding the extent of his cognitive difficulties. When scoring, I received invaluable feedback from my occupational therapy colleagues’ assessment and changed my approach accordingly”. The feedback from teams using the MDT Stroke TOMs is that it is easy to implement, quick to use within a multidisciplinary team, holistic and easy to document.

Study limitations and implications

Previously, the limitations of testing inter-rater reliability using case histories have been highlighted because information “can be either inadequate or reveal information in such a way as to prime judges” [18]. In this reliability trial, this did not seem to have an effect.

There may be reservations about the training required to implement the Stroke MDT TOMs within services that are new to TOMs. Appropriate use of this and other outcome measures necessitates understanding of the domains and diligence leading to reliable use. Formal training for the TOM is available remotely (via Innervate Ltd.: https://innervate.co.uk/).

There may also be concerns about the sensitivity of the measure, i.e., TOM only reflects clinically significant changes (deemed at 0.5 on the TOM scales). Feedback from the relevant teams across the South West London Network has been that it is easy to implement, quick to use within a multidisciplinary team, holistic and easy to document. It not only allows you to see change within body function/impairment, but also in how independent the patient is, how socially capable and integrated they are and how they and their carers are feeling. Using a reliable outcome measure facilitates audit and benchmarking of services.

Conclusion

In conclusion, there was good-to-excellent concordance between raters from different professional groups in terms of scoring. Raters’ scores were also consistent. The MDT Stroke TOMs has been found to be psychometrically robust and should assist in making already functioning MDTs move towards interdisciplinary rehabilitation of patients with stroke and their caregivers. The MDT Stroke TOMs was developed as a quick and easy to use, clinical outcome measure within Croydon’s Stroke Team to monitor trends within stroke services, facilitate benchmarking and IDT working, ensure better communication, and facilitate quality assurance.

Future research will need to examine the reliability in other geographical areas where teams may not be so cohesive. The use of the MDT Stroke TOMs as a tool to promote team communication and interdisciplinary collaboration in Stroke Rehabilitation could be explored.

Acknowledgments

We thank Frances Smith, Physiotherapist, a co-trainer in TOMs, and an advocate for MDT Stroke TOM in Southwest London. We thank Rachel Sibson, and the Southwest London Stroke Network for contributing to data collection and TOMs training. We thank Mary Brooks, Operational Clinical Lead for Integrated Neuro Therapy Services, and Croydon Health Services NHS Trust for providing time, support, and resources for training in TOM.

Disclosure statement

No potential conflict of interest was reported by the authors.