Absolute and relative intrarater reliability of the modified motor assessment scale according to Uppsala academic hospital -99

ABSTRACT

Background

For some of the most commonly used motor measures, psychometric properties, and minimal detectable change (MDC₉₅) remain largely unknown, limiting the interpretability of tests.

Objective

The aim was to establish intrarater reliability, MDC₉₅ and floor- and ceiling effects for a modified version of the Motor Assessment Scale (M-MAS UAS-99).

Methods

Data was derived from an intervention study that enrolled 41 individuals with chronic stroke. Test scores from two subsequent assessments with 3 weeks apart were used for establishing the floor and ceiling effect, the intraclass correlation coefficient (ICC_[2,1]), standard error mean (SEM) and the MDC₉₅ for the total score, and subdomains of the M-MAS UAS-99.

Results

The intrarater reliability was excellent with an ICC_[2,1] between 0.970 and 0.995 for both total score and subdomains. The MDC₉₅ for the M-MAS UAS-99 total score was 1.22 which means ≥ 2.0 points on an individual basis. For bed mobility subdomain, a ceiling effect was seen, but not for the total score of the test. No floor effect was seen for the test.

Conclusion

M-MAS UAS-99 has excellent intrarater reliability. Any individual increase in test scores must reach 2.0 to be considered a true change.

KEYWORDS:

• Chronic stroke
• reliability
• minimal detectable change
• functional motor ability

Introduction

Approximately 80% of stroke survivors suffer a functional impairment, which brings a sudden and radical change to their life situation and often results in serious and persistent disabilities and activity limitations. These limitations often have considerable physical, psychological, and financial impact on the individual’s daily life, social activities, and health-related quality of life and places burdens on their families, the health-care system, and society (Langhorne, Bernhardt, and Kwakkel, 2011). The Modified Motor Assessment Scale according to Uppsala Academic Hospital (M-MAS UAS) version 1999 (Carr, Shepherd, Nordholm, and Lynne, 1985) is a Swedish instrument developed based on Carr and Shepherd’s Motor Assessment Scale (MAS) (Carr, Shepherd, Nordholm, and Lynne, 1985; Lannin, 2004; Scrivener, Schurr, and Sherrington, 2014) and Modified Motor Assessment Scale (Carr and Shepherd, 1989; Loewen and Anderson, 1988). The original version of MAS has been widely used in clinical and research settings and was designed to assess eight subsets of functional motor performance and one subset of muscle tone. The item designed to measure muscle tone was removed from the MAS because of low reliability (Aamodt et al., 2006). In addition, assessment of the intact side was added for the functional areas arm function, hand function, and fine motor skills, which resulted in the motor performance being assessed in 11 different areas. The new assessment instrument was named Modified Motor Assessment Scale according to Uppsala Academic Hospital (M-MAS UAS) and has been subjected to validity and reliability testing (Barkelius, Johansson, Korm, and Lindmark, 1997; Johansson, 1992).

Since 1991, the M-MAS UAS has been modified twice. In 1995 (Barkelius, Johansson, Korm, and Lindmark, 1997), further modifications to the scaling system of M-MAS UAS was made. Previously, the 11 items were assessed on a 7-point scale ranging from 0 to 6, but in its revised version, a scale step was removed, resulting in 0–5 scale steps. In 1999 (Andersson, 1999), the user guideline was clarified for the items sitting, walking, and hand movements, as well as the scoring criteria for some items. This new version of the instrument was named M-MAS UAS-99. The M-MAS UAS-99 measures activity according to the ICF (Geyh et al., 2004).

While the M-MAS UAS-95 has been subjected to both concurrent validity and reliability assessments (Arnell, Westlin, and Lindmark, 1996; Barkelius, Johansson, Korm, and Lindmark, 1997) and correlation to other measurements (Linder, Winkvist, Nilsson, and Sernert, 2006) the M-MAS UAS-99 has only been subjected to interrater reliability testing (Andersson, 1999). In the essay investigating the interrater reliability of the M-MAS UAS-99 (Andersson, 1999) 20 individuals with varying degrees of motor deficits after stroke underwent testing by four assessors, with two having experience from applying the M-MAS and two having no experience. The result showed a generally good interrater reliability. Out of the total 1,320 pairwise assessments, 1,213 were in agreement, demonstrating a total agreement of 92% and an ICC that ranged from 0.95 to 1.0 (Andersson, 1999).

Even though the M-MAS UAS-99 is the version most commonly used in both clinical and research settings in Sweden (Brogårdh and Lexell, 2010; Brogårdh, Vestling, and Sjolund, 2009; Persson, Hansson, and Sunnerhagen, 2011) several clinimetric properties such as absolute and relative intrarater reliability, minimal detectable change (MDC) or floor and ceiling effects have not yet been investigated. To meaningfully interpret test scores and draw appropriate conclusions, one needs reliable tools that allow consistent reproduction and results when applied at different times except for random errors. The MDC is the smallest change that can be detected by the instrument beyond measurement error with a specific amount of confidence (Haley and Fragala-Pinkham, 2006; Jette, Tao, Norweg, and Haley, 2007). Such information can facilitate the interpretation of treatment outcomes that helps inform clinical decision-making in the alteration or discontinuation of treatments that aim to improve the functional ability of patients (Lin, Fu, Wu, and Hsieh, 2011). Therefore, this study seeks to establish the absolute and relative intrarater reliability, floor and ceiling effects, and the MDC with 95% confidence (MDC₉₅) of the M-MAS UAS-99.

Methods

Participants

This study is based on data from a cohort of individuals (n = 41) with chronic stroke who were enrolled in an explorative study with a within-subject repeated measures design. The explorative study sought to assess the effectiveness of enriched task-specific therapy in late-phase stroke-survivors with residual hemiplegia (Vive, Af Geijerstam Jl, Kuhn, and Bunketorp-Käll, 2020). The eligibility criteria for the study were as follows: at least 6 months after the onset of stroke; disability grade 2–4 on the modified Rankin Scale (mRS) (Rankin, 1957); no other injury, illness or addiction, making the individual unsuitable for participation. This included: exercise-induced epilepsy assessed by the referring or prescribing physician; cognitive and speech ability that does not enable instruction, intervention and evaluation; lack of ability and willingness to travel to the place of evaluation; unable to perform sit-to-stand and stand-to-sit transfers independently or with assistance, without assistive technology; not having participated in an intense comprehensive rehabilitation program other than post-stroke acute and subacute rehabilitation within the previous 6 months; and not scheduled for other treatment with focus on intensive training during the study period. All 41 individuals who completed the intervention were included in the present study.

Procedures

Ethics approval was granted by the Regional Ethical Review Board in Gothenburg, Sweden (Ref number: 549–12). The 41 individuals enrolled in the previous exploratory intervention study underwent a baseline phase with similar length as the intervention (i.e. enriched task-specific therapy) to determine the stability of outcome measures (Vive, Af Geijerstam Jl, Kuhn, and Bunketorp-Käll, 2020). The first baseline assessment (test 1) was followed by a second baseline assessment 3 weeks later (test 2). Data from the first and second baseline assessments was used for test–retest analysis in the present study. An independent physical therapist performed all clinical assessments. Efforts were made to provide stability of the test conditions across test occasions, such as using identical instructions, data collection protocols, and equal examining table, desk, chairs, and other equipment.

Outcome measure

For the purpose of measuring functional motor performance, the M-MAS UAS-99 (Andersson, 1999) was used, which is a Swedish modification of the original MAS (Carr, Shepherd, Nordholm, and Lynne, 1985) and the original M-MAS (Carr and Shepherd, 1989; Loewen and Anderson, 1988). The eight motor components included in the M-MAS UAS-99 are hierarchically scored from 0 to 5, and the observer assesses the quality of movement and the speed of performance of each task. The maximum total score of 55 indicates optimal functional motor performance. The eight items can be divided into three domains: 1) bed mobility (two items, maximum score 10 points); 2) lower limb functional tasks (three items, maximum score 15 points); and 3) upper limb function (three items) (Nyström and Hellström, 2013). Upper limb function is tested at each side at a time (maximum 15 + 15 = 30 points) (Andersson, 1999). The M-MAS UAS-99 (translated to English) can be found in Supplemental Material A.

Data analyses

Estimation of intraclass correlation coefficient

The intraclass correlation coefficient (ICC_[2,1]) was used as a measure of relative intrarater reliability, reflecting the variation of data measured by one rater across two or more measurements (Fisher, 1992). The strength of correlation was interpreted according to Koo and Li (2016) where a correlation below 0.50 = poor; between 0.50 and 0.75 = moderate; between 0.75 and 0.90 = good; and above 0.90 = excellent. The definition by Shrout and Fleiss presented as two numbers in parentheses was used in the present study (e.g. ICC_[2,1]; two-way random single measures, absolute agreement). The first digit refers to the model (1, 2, or 3), and the second digit refers to the type, which is either a single rater/measurement (1) or the mean of raters/measurements (k).

Estimation of standard error of measurement

The variability of two measurements on the same individual was used to calculate the absolute measurement error called the standard error of measurement (SEM). The SEM assesses how precisely a test measures a subject’s true score and accounts for within-subject variability (intra-individual standard deviation). The SEM was estimated by taking the square root of the within-subject variance in the participants’ test and retest scores. SEM = pooled SD x √(1-r), where r = ICC_[2,1]. The lower the SEM, the more reliable the measurement is (Atkinson and Nevill, 1998).

Estimation of minimal detectable change, absolute reliability

The Minimal Detectable Change with 95% confidence (MDC₉₅), absolute reliability, or repeatability represents the magnitude of change necessary to exceed the measurement error of two repeated measures at a specified confidence interval (CI). The MDC with 95% CI (MDC-95) was calculated as MDC = 1.96 * √2 * SEM, where 1.96 is the two-sided tabled z value for the 95% CI and √2 is used to account for the variance of two measurements. The MDC₉₅ is the smallest change in score that likely reflects true change (not measurement error alone). Wilcoxon signed test was used for establishing systematic deviations between measurement timepoints. Analyses were done using SPSS v.22.0 (IBM, Armonk, NY, USA).

Floor and ceiling effect

Floor and ceiling effects are used to describe the variance of a test if a subject’s scores are either near the bottom score of the test (floor), or near the top score of the test (ceiling). The floor and ceiling effects were calculated as the percentage of participants who achieved the minimum and maximum possible scores for the total score and for all domains, respectively.

Results

Forty-one subjects participated in the study, 8 (20%) females and 33 (80%) men, with a mean age of 59.6 (±13.9) years. The demographics and clinical characteristics of the study participants are presented in Table 1. The M-MAS UAS-99 data from the two assessment points (test-retest) are presented in Table 2. The subdomains bed mobility and lower limb functional tasks showed excellent internal consistency; Cronbach’s Alpha = 0.83 and 0.86.

Table 1. Demographics and clinical characteristics of the study participants.

Characteristics
Participants	41
Women	8 (20%)
Men	33 (80%)
Mean age, years (SD) median [min;max]	59.6 (13.9) 64 [22;84]
Months since stroke onset, mean (SD) median [min;max]	35.5 (29.5) 26 [6;130]
Modified Rankin Scale
Mean grade, mean (SD) median [min;max]	3.4 (0.7) 3 [2;4]
Grade 2	5 (12%)
Grade 3	16 (39%)
Grade 4	20 (49%)

Mean (SD)/median [min;max] is given for continuous variables and n (%) for categorical variables.Modified Rankin Scale: an ordinal disability rating scale scored 0–6. 2 – Slight disability. Able to look after own affairs without assistance, but unable to carry out all previous activities. 3 – Moderate disability. Requires some help, but able to walk unassisted. 4 – Moderately severe disability. Unable to attend to own bodily needs without assistance and unable to walk unassisted. SD: Standard Deviation.

Table 2. Data from test 1, and test 2, change scores for test-retest (test 1 and 2), proportion of observations differing ≤ 1, 2 or 3 points, intraclass correlation, standard error of measurement, minimal detectable change with 95% confidence interval and coefficient of variation.

Measure	Test 1 mean (SD) median; (min-max)	Test 2 mean (SD) median; (min-max)	Change mean (SD)95% CI for mean median (min;max) Sign	% of observations differing 1 pts or less between test 1 & 2	% of observations differing 2 pts or less between test 1 & 2	% of observations differing 3 pts or less between test 1 & 2	ICC[2,1] (95% CI)	SEM (Intra ID SD)	MDC95 integer (decimal)	CV
MMAS UAS-99 total score (0–55)	37.8 (8.9) 36; (20–55)	38.1 (8.8) 37; (22–55)	0.3 (1.6) -0.2; 0.8 0 (−4;5) p = .21	68%	93%	95%	0.99 (0.98;0.99)	0.44	2 (1.2)	0.03
MMAS UAS-99 bed mobility (0–10)	8.5 (2.4) 10; (2–10)	8.4 (2.3) 10; (3–10)	−0.1 (0.8) -0.3; 0.2 0 (−2; 2) p = .59	90%	100%	100%	0.97 (0.94;0.98)	0.14	1 (0.4)	0.05
MMAS UAS-99 Lower limb functional tasks (0–15)	10.9 (3.6) 12; (4–15)	11.1 (3.5) 12; (4–15)	0.3 (0.9) -0.0; 0.6 0 (−2; 3) p = .07	90%	98%	100%	0.98 (0.97;0.99)	0.31	1 (0.9)	0.05
MMAS UAS-99 Upper limb function (0–30)	18.5 (4.8) 16 (14–30)	18.6 (4.6) 17 (13–30)	0.1 (0.7) -0.1; 0.3 0 (−2; 2) p = .35	98%	100%	100%	0.99 (0.99;1.00)	0.23	1 (0.7)	0.03

CI: Confidence interval; CV: Coefficient of variation; ICC_[2,1]: Intraclass Correlation; Intra ID SD: Intra Individual Standard Deviation; MDC95: Minimal Detectable Change with 95% Confidence Interval; M-MAS UAS −99: Modified Motor Assessment Scale Uppsala Academic Hospital version 99; SEM: Standard Error of Measurement; SD: standard deviation.

Intrarater reliability

Complete data for the test–retest analyses were available for all participants. The ICC_[2,1] for the total score of the test was 0.99. The ICC_[2,1] for bed mobility was 0.97, lower limb functional tasks 0.98, and upper limb function 0.99 (Table 2 and Figures 1 – 4). The MDC₉₅ for the M-MAS UAS-99 total score was 2 rounded to the nearest integer. For the domains bed mobility, lower limb functional tasks, and upper limb function, the MDC₉₅ was shown to be 1 rounded to the nearest integer, respectively. No systematic deviations were shown between tests 1 and 2 (total score p = .21, bed mobility p = .59, lower limb functional tasks p = .07 and upper limb function p = .35 [Wilcoxon signed test]).

Figure 1. Scatterplot and Bland–Altman plot for Total Score.

Figure 2. Scatterplot and Bland–Altman plot for Bed Mobility.

Figure 3. Scatterplot and Bland–Altman plot for Lower Limb Functional Tasks.

Figure 4. Scatterplot and Bland–Altman plot for Upper Limb Function.

The change in the M-MAS UAS-99 total and domain scores from test-retest versus the mean score for tests 1 and 2 are presented in Figures 1 – 4 using Bland-Altman plots (Bland and Altman, 1999). The Bland-Altman plots visualize the differences in measurements between test 1 and 2, and the upper and the lower limit of the 95% confidence interval for the average difference between test 1 and 2, and as showed in Figure 1 – 4, a majority of the differences between the two test occasions fell in the range of the 95% confidence intervals for the total score, bed mobility, lower limb functional tasks and upper limb function domains.

Table 2 shows the proportion of individuals differing less than or equal as 1, 2 or 3 points for the total score and subdomains. For the total score 95% of observations differed less than or equal as 3 points between test 1 and test 2, and for the subdomains 100% of observations differed less than or equal as 3 points between test 1 and 2.

Floor and ceiling effects

Of the total study population of 41 individuals, two subjects (4.9%) reached the maximum M-MAS UAS-99 score at test 1 and one subject (2.4%) at test 2. For the bed mobility domain, 25 individuals (61.0%) reached the maximum score at test 1, and 23 (56.1%) at test 2. For the lower limb functional tasks, eight individuals (19.5%) reached the maximum score at test 1, and 7 (17.1%) at test 2. For the upper limb function domain, two individuals (4.9%) had the maximum score at test 1, and one (2.4%) at test 2. No one received zero points on the domains at any assessment time points, suggesting no floor-effect of the measure.

Discussion

The results of the present study indicate that the repeatability of the M-MAS UAS-99 is excellent. The ICC_[2,1] represents the degree of repeatability between two successive assessments, that is, relative intrarater reliability. The ICC_[2,1] for the M-MAS UAS-99 total score and subdomains varied between 0.97 and 0.99, suggesting that the intrarater reliability reflecting the variation of data measured by one rater across two or more measurements was excellent for the M-MAS UAS-99 (Koo and Li, 2016). Data from earlier studies on the original versions of the MAS have shown a test–retest reliability of 0.87 to 1.0 (mean r 0.98) (Carr, Shepherd, Nordholm, and Lynne, 1985). The Bland–Altman plots showed that the change in scores did not increase with increasing mean for neither the total score, nor the domains, suggesting no bias between the mean differences.

To the best of our knowledge, this is the first study to determine the absolute reliability, and the MDC₉₅ for the M-MAS UAS-99. The calculated MDC₉₅ for the total M-MAS UAS-99 score (1.2) implies that in clinical settings, an increase on individual basis must reach 2 points to be interpreted as a change that exceeds the statistical error of the test. For the domains bed mobility, lower limb functional tasks, and upper limb function, the change in scores had to be at least 0.41, 0.88, and 0.65, respectively, to be considered true changes. Translated to clinical practice, this entails that any gain in functional motor performance as measured by the individual domains must reach 1 point to represent a real change. This is to be compared with other ordered categorical stroke measures such as the Fugl-Meyer Assessment of motor recovery after stroke where the MDC₉₀ is suggested to be 3.2 points out of 66, for the upper extremity section (See et al., 2013) or the Berg Balance scale, where the MDC₉₅ is suggested to be between 2.7 (Alghadir, Al-Eisa, Anwer, and Sarkar, 2018) and 4.1 (Flansbjer, Blom, and Brogårdh, 2012) out of 56 points.

A strength of the current study is that all participants were in a stable chronic phase post-stroke. For an individual in a chronic phase after a stroke, no change in functional motor performance deficits would be expected within 3 weeks, without any offered intervention. Nevertheless, to determine whether a change between two measurement time points is true, one must consider daily fluctuations in the conditions of individuals with chronic stroke. To compare the performance of an individual in a chronic phase after stroke between two successive sessions, one must accurately arrange consistent circumstances with regard to the time during the day and measure implementation, which we ensured to do in this study.

We have previously demonstrated that individuals with chronic stroke who underwent a 3-week-long enriched task-specific therapy had in average a 2.3-point increase in the total M-MAS UAS-99 score (Vive, Af Geijerstam Jl, Kuhn, and Bunketorp-Käll, 2020). According to the findings in the present study the participants’ gains in functional motor performance achieved by therapy could be considered clinically relevant. However, as many as 25 participants (61%) reached the maximum score for bed mobility (10 points) already at baseline (test 1); thus a ceiling effect was present. English, Hillier, Stiller, and Warden-Flood (2006) showed ceiling effects between 9% and 56% for the different items included in the original version of the test. Other studies have shown presence of large ceiling effect for the M-MAS UAS-99 for individuals 1–3 years after stroke (Vahlberg et al., 2013). Nevertheless, the study cohort in the current study represented a group of individuals presented with a wide spectrum of post-stroke disability equivalent to scores between 2 and 4 on the mRS, and half the cohort (49%) suffered from moderately severe disability equivalent to an mRS score of 4. Accordingly, the study cohort did represent a group of individuals unable to walk or attend to bodily functions without assistance from another person (Rankin, 1957).

The absence of statistically significant changes between the M-MAS UAS-99 scores at test 1 and test 2 in the present study indicates that there was no systematic bias between test sessions. However, there was a tendency for the lower limb functional tasks domain to indicate a change in score (p = .07). Yet, the median value for both test 1 and 2 were similar (median 12), indicating that a few number of individuals affected the calculations of significance in change.

Limitations

Using parametric statistics for summed scores from Likert scales are widely debated (Jamieson, 2004; Lubke and Muthén, 2004). However, a large number of publications within the physical therapy field have thoughtfully used parametric calculations of MDC of well-known and largely used scales both using summation of items as the Berg Balance scale (Donoghue and Stokes, 2009) and the Mini-Best test (Godi et al., 2013) and using subdomains of scores such as the Fugl-Meyer Upper Extremity subdomain (Wagner, Rhodes, and Patten, 2008) and the ARAT test with subdomains (Yozbatiran, Der-Yeghiaian, and Cramer, 2007). The choice of using the three domains: 1) bed mobility; 2) lower limb functional tasks; and 3) upper limb function in the present study was based on the logical subdivision of the scale and the study of Nyström and Hellström (2013). The subdomain upper limb function has earlier been subject to psychometric evaluation (Hsueh and Hsieh, 2002; Lannin, 2004), whereas psychometric evaluation has not yet been performed for the subdomains lower limb functional tasks and bed mobility. We therefore performed a validity calculation of internal consistency for these subdomains showing a relatively high internal consistency with Cronbach’s Alpha of 0.83 and 0.86, respectively. The MDC results from these two subdomains might therefore be interpreted with cation.

A combination of distribution and anchor-based methods has been described in clinical reports to describe statistically and clinically meaningful changes of outcomes (Cella et al., 2002; Eton et al., 2004). In this study we used a distribution-based approach to calculate SEM and MDC₉₅. Distribution-based calculations have the advantage of simplicity of use since they do not require an external criterion (Salas Apaza et al., 2021). However, they produce similar MDC results for both deterioration and improvement (Haley and Fragala-Pinkham, 2006). In this study, there were no anchors measuring similar functional aspects as the M-MAS UAS −99, which was why the use of an anchor-based approach to establish the MDC or the MCID was not possible. Further, intrarater reliability was assessed in the present study using only one single rater, and the results could not be applied to the reliability across two raters. Future research should include intrarater reliability across multiple raters as well as well as assessment of interrater reliability to confirm its reliability. Additional studies should also focus on the clinically meaningful changes of this and other motor outcomes.

Conclusion

The M-MAS UAS-99 has excellent intrarater reliability, with ICC_[2,1] varying between 0.970 and 0.995 for the total score and the different domains. Any individual increase in total test score must reach 2.0 to be considered a true change, and for the different domains, an increase of 1.0 is needed.

Acknowledgments

The authors thank the following funding agencies for supporting the study: Aina Wallström’s and Mary-Ann Sjöblom’s Foundation; Peter Eriksson Foundation; Swedish state under the agreement between the Swedish government and the county councils; ALF-agreement (725241); Promobilia Foundation; Swedish Stroke Association; Rune and Ulla Almlöv’s Foundation; and the Foundation for Rehabilitation and Medical Science.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/09593985.2022.2122913

Additional information

Funding

This work was supported by the Swedish state under the agreement between the Swedish government and the county councils [725241];The Swedish Stroke Association;Aina Wallström’s and Mary Ann Sjöblom’s Foundation;Foundation for Rehabilitation and Medical science;Peter Eriksson Foundation;Rune and Ulla Almlöv’s Foundation;Stiftelsen Promobilia