Validity and reliability of the Selective Control of the Upper Extremity Scale in children with upper motor neuron lesions

Abstract

Purpose

We evaluated the validity and intra-, inter-, and test–retest reliability of the Selective Control of the Upper Extremity Scale (SCUES) sum and item scores in patients with upper motor neuron lesions.

Methods

Thirty-one boys and 15 girls (mean age ± SD: 11 years 1 month ± 3 years 9 month) with upper motor neuron lesions participated. We correlated SCUES scores with the range of motion items of the Melbourne Assessment 2 (MA2) and Box and Block Test (BBT) to establish concurrent validity and compared scores between the more and less affected side for discriminative validity. Intra-class correlation coefficients (ICC) and smallest detectable changes (SDC) indicated relative and absolute reliability.

Results

For the more affected side, SCUES sum scores correlated well with MA2 (ρ = 0.83) and BBT (ρ = 0.77), and reliability proved high for intra-rater (ICC = 0.93, SDC = 2.55), inter-rater (ICC = 0.86, SDC = 3.58), and test–retest (ICC = 0.98, SDC = 1.41) reliability. Reliability of single items varied from 0.64 (inter-rater elbow) to 0.98 (intra-rater elbow). Limb and item scores were lower for the more affected side.

Conclusion

The SCUES limb and item scores seem valid and reliable in children with upper motor neuron lesions. While future studies should evaluate the responsiveness of the SCUES, we recommend that the same rater should score a patient twice.

Implications for rehabilitation

• The SCUES assesses selective voluntary motor control and appears valid and reliable in patients with upper motor neuron lesions.

• Test–retest reliability of the SCUES seems excellent.

• SCUES single item scores show concurrent validity and acceptable reliability.

• Limb and item scores are significantly lower for the more affected side.

Keywords:

• Cerebral palsy
• children with disability
• psychometrics
• reliability and validity
• selective motor control
• stroke

Introduction

An upper motor neuron lesion (UMNL) can be the cause of various neurological symptoms in children and adolescents with, for example, cerebral palsy (CP), stroke, or traumatic brain injury. Classically, the symptoms were divided into positive signs, such as hyperreflexia or increased muscle tone, and negative signs, such as reduced strength or selective voluntary motor control (SVMC). During recent years, reduced selective motor control, which is defined as the “reduced ability to isolate the activation of muscles in a selected pattern in response to demands of a voluntary posture or movement” [1], received increasing attention because it seems to affect particularly bimanual performance [2–4].

Few assessments quantify SVMC, especially of the upper extremity [5]. Some included sub-scores or items that assess (aspects of) SVMC, such as the Quality of Upper Extremity Skills Test [6,7], the range of motion (ROM) subscale of the Melbourne Assessment 2 (MA2) [8–10], the Fugl-Meyer [11,12], or the Shriners Hospital Upper Extremity Examination (SHUEE) [13]. Only recently, two clinical assessments were developed specifically for children with SVMC impairments. The Test of Arm Selective Control (TASC) assesses flexion/extension of the shoulder, elbow, wrist, and fingers, shoulder abduction/adduction, forearm supination/pronation, thumb opposition (pincer grip), and metacarpophalangeal extension (key grip) [5]. Its 3-point ordinal scoring (normal, impaired, or absent SVMC) resembles that of the Selective Control Assessment of the Lower Extremity (SCALE) [14,15] and includes descriptors like reduced ROM, slow, extra movement, or mirror movement. Some years before, the Selective Control of the Upper Extremity Scale (SCUES) was published [16]. It rates SVMC of single-joint movements of shoulder abduction/adduction, elbow flexion/extension, forearm supination/pronation, wrist flexion/extension, and fingers/thumb (grasp/release) on a 4-point ordinal scale and includes similar descriptors. The validity and reliability of these tools have been investigated, for the SCUES in children with unilateral spastic CP [16,17], and for the TASC in children with spastic CP without significant ataxia or dystonia [5]. For example, concurrent validity of the SCUES has been investigated by correlating it with the spontaneous functional analysis section of the SHUEE (Spearman correlation coefficient ρ= −0.69), MACS (ρ= −0.24), and BBT (ρ= −0.47) [16] or with the Jebsen Taylor Hand Function Test (ρ= −0.80), and MACS (ρ= −0.78) [17]. The latter study [17] also showed that SCUES scores were smaller for the affected than the non-affected side (i.e., discriminative validity). Intra-rater and inter-rater reliability has been investigated in both studies [16,17], showing acceptable to excellent intra-class correlation coefficient (ICC) values for joints and sum scores. The SCUES has further served to validate a novel SVMC assessgame [18,19].

Various patient groups with an UMNL like children with spastic CP, mixed patterns of CP such as spastic-dystonic or -ataxic, or children with acquired brain lesions due to stroke or a traumatic injury might profit from therapy focusing on improving SVMC by practicing tasks that include accurate motor control while suppressing involuntary movements. When we started with this project, only the SCUES was available. Indicators of absolute reliability (i.e., measurement error), test-retest reliability, and validity of individual items had not been investigated. So, we aimed to determine the concurrent and discriminative validity and relative and absolute intra-rater, inter-rater, and test–retest reliability of the SCUES limb and item scores in children and adolescents with an UMNL. For the validity, we formulated the following hypotheses for the more affected side a priori: (i) Correlations between the SCUES limb score and the ROM subscale of the MA2 are higher (similar construct) than between the SCUES and Box and Block Test (BBT), (ii) SCUES item scores correlate at least moderately with congruent MA2 item scores. (iii) Concerning discriminative validity, limb, and item scores are lower for the more affected side.

Methods

Translation procedure

Two persons (including TG) translated the implementation guidelines, evaluation criteria, and evaluation forms independently from English into German [20]. Both were native German speakers with several years of clinical experience. The translations were compared and synthesized by two occupational therapists (J. L. and T. G.) and a physiotherapist with long-term experience in translations (HvH). Any ambiguities regarding language, technical or cultural aspects were discussed and summarized in a consensus version. During the translation process, questions regarding the interpretation and evaluation of the items were clarified with the authors of the original version. The back-translation, made by a professional translation company and subsequently slightly adapted by the team, was sent to the authors of the original version. They provided feedback, which was incorporated into the German version before legitimizing the final version (https://www.kispi.uzh.ch/rza/de/forschende/publikationen/downloads/Seiten/default.aspx).

Participants and setting

Participants were recruited by convenience sampling. The measurements for this prospective, cross-sectional study (with repeated assessments for test-retest reliability) were performed from April 2017 until October 2018 in the Swiss Children's Rehab of the University Children's Hospital Zurich, a private practice and a school for children with special needs in Switzerland. Inclusion criteria were age three to 18 years old; UMNL, which affected upper limb function; able to sit safely upright on a chair or in a wheelchair for 60 min, and able to understand and implement simple verbal instructions, such as “please extend and bend your elbow three times in a row covering the full range of motion as I have shown you.” We included children with a “Manual Ability Classification System” (MACS) level I to IV [21]. Level I is defined as “handles objects effortlessly and successfully” while IV indicates “handles a limited selection of easily managed objects in adapted situations.” The MACS shows good validity and is reliable [21].

We excluded children who had received Botox or upper extremity surgery within the last six months. The Human Research Ethical Committee of the Canton of Zurich approved the study (PB_2016-01843).

Selective control of the upper extremity scale

The SCUES was developed for children and adolescents aged 3–18 years with unilateral spastic CP to quantify SVMC [16]. This video-based assessment takes about 15 min. A camera, a table, a chair, and a 10.2 cm (4 in.) high block supporting the lower arm are required. Five movements are examined (see Introduction). First, the joint movement to be tested is demonstrated by the therapist to the patient. The therapist performs the joint movement passively three times over the entire ROM. Then, the patient performs the movement actively three times at the same speed, first, with the joint movement on the neurologically less affected side, followed by the more affected side. This is repeated for all movements and filmed from the front.

Each joint is evaluated based on the video recording with a score: 3. normal, 2. mildly diminished, 1. moderately diminished, and 0. no SVMC. The descriptors “mirror movements,” “movements of other joints,” “movement of the trunk,” and “movement of less than 85% of passive ROM” are also noted on the evaluation sheet.

Comparator measures

The MA2 is a valid and reliable measure to evaluate the quality of upper extremity movements in children with neurological impairments aged 2.5–15 years [8–10]. The MA2 measures ROM, accuracy, dexterity, and fluency. It includes 14 test items that evaluate reaching, grasping, releasing, and manipulating objects. For the current study, we considered the sub-scale ROM. It contains nine items evaluating the ROM and corresponding joint position, as well as compensatory movements. The reliability of this subscale is excellent (ICC ≥ 0.94) [8]. We compared the SCUES items with the congruent MA2 items: SCUES shoulder ab/adduction with MA2 item 2 (reach sideways to elevated position), elbow flexion/extension with item 14 (hand to mouth and down), pro/supination with item 12 (pro/supination), wrist flexion/extension with item 7 (release of pellet), and finger and thumb flexion/extension with item 5 (release of crayon). These MA2 items have almost perfect intra-rater reliability (0.81 ≤ kappa ≤ 0.97) and moderate to almost perfect inter-rater and test-retest reliability values (0.52 ≤ kappa ≤ 0.97) [8].

The BBT is a valid and reliable assessment testing unilateral gross manual dexterity in typically developing children [22] and children with CP [23]. The participant is asked to transfer as many blocks as possible from one side to the other side of a box within 60 s.

Procedures

Four occupational therapists with 16, 9, 4, and 2 years of experience working with children participated. The training consisted of an introduction to the application and scoring of the SCUES (60 min) and reading the manual (60 min) with a subsequent focus on how to score each item (30 min). Then, each occupational therapist had to score three videos provided by the SCUES developer team [16]. Scorings were extensively compared and discussed between the OTs (180 min), resulting in 450 min of training per therapist.

At the first appointment (duration 45 min), the SCUES, MA2 ROM subscale, and the BBT were assessed. At the second measurement, only the SCUES was performed (15 min). All tests were videotaped. To obtain independent yet stable ratings, the time between ratings by the same therapist was at least 2 weeks (intra-rater reliability), three different raters rated the same video independently from each other (inter-rater reliability), and participants performed the SCUES twice within 1–7 days (test–retest reliability).

Statistical analyses

Concurrent validity: we calculated non-parametric Spearman’s rank correlations (ρ) between SCUES sum and item scores and the comparator measures with 95% confidence intervals (95%CI) using bootstrapping (1000 samples, biased corrected accelerated procedure). Differences between the more and less affected side were evaluated with Wilcoxon-signed-rank tests (because of 6 comparisons, we set α at 0.05/6 = 0.008). Relative reliability was quantified with ICCs, according to Rouson et al. [24]. We also calculated the reliability of item scores with these formulas, as results are similar to those obtained with quadratic weighted Kappa analyses. Simultaneously, it allows to include data of three raters in the inter-rater reliability analyses and provides the SEM value, which multiplied with 1.96 x √2 results in the smallest detectable change (SDC). The SDC quantifies the absolute measurement error and determines the value that should be exceeded for an individual patient to be considered a true change. We interpreted ICC values similarly as Wagner et al. [16]: ICC ≥ 0.75 excellent agreement and 0.40 ≤ ICC ≤ 0.75 fair to good agreement [25].

Results

Thirty participants with an UMNL participated. Three children could not perform a second assessment (test–retest reliability) due to early discharge from the rehab center. For intra-rater and inter-rater reliability, we could evaluate SCUES video-recordings of 16 additional patients (derived from a previous project [18,19]). The age (mean ± SD) of the forty-six children and adolescents was 11 years 1 month ± 3 years 9 months and ranged from 5 years 9 months to 18 years 10 month. The participants’ characteristics are displayed in Table 1.

Table 1. Characteristics of the participants.

	Validity (n = 30)	Intra-, inter-rater reliability (n = 46)	Test–retest reliability (n = 27)
Sex
Males	22	31	18
Females	8	15	9
Diagnosis
Unilateral spastic CP	3	5	2
Bilateral spastic CP	9	16	11
CP other^a	4	6	4
Encephalitis	6	6	1
Stroke ischemic	4	7	5
Stroke hemorrhagic	1	2	1
Traumatic brain injury	2	3	3
Epilepsy	1	1	–
MACS
I	10	11	6
II	16	24	14
III	3	9	6
IV	1	2	1
More affected side
Left	23	32	17
Right	7	14	10
Dominant side
Left	8	15	11
Right	22	31	16

MACS: Manual Ability Classification System.

^aCP other were children with a mixed spastic-dystonic (n = 1) or spastic-ataxic (n = 3) cerebral palsy.

The SCUES limb score of the more affected side correlated well with the MA2 sum scores (Table 2, Figure 1(A)). Also, the correlations between the SCUES and MA2 item scores were higher for the more affected side. Participants transferred fewer blocks with the more affected arm (BBT: median 36.5, IQR: 26.5–45.0) compared to the less affected arm (median: 43.5 blocks, IQR: 33.75–52.50; p < 0.001). Correlations between the SCUES limb scores and the BBT were 0.77 (95%CI: 0.55–0.89, p < 0.001, Figure 1(B)) for the more affected side and 0.59 (95%CI: 0.25–0.82, p = 0.001) for the less affected side. SCUES scores were significantly lower for the more affected side (Table 2, SCUES Diff.*).

Figure 1. Validity and reliability of SCUES limb scores of the more affected side. Scatterplots showing the relationship between the SCUES and (A) the sum of the Melbourne Assessment 2 range of motion items (MA2) and (B) the Box and Block Test (BBT). Scatterplots C–E show the reliability findings for the (C) intra-rater, (D) inter-rater, and (E) test–retest reliability. In (D), we presented the results for comparing raters 1 and 2 in black and raters 1 and 3 in grey. Note that this was done for visual interpretation only (statistics were performed as described in the methods section). Different signs reflect different diagnoses. “CP other” were children with a mixed spastic-dystonic or spastic-ataxic cerebral palsy. SCUES: Selective Control of the Upper Extremity Scale; CP: cerebral palsy; TBI: traumatic brain injury.

Table 2. Median (IQR) SCUES and MA2 scores and correlations for the more and less affected side.

Items SCUES	Items MA2	More affected side			Less Affected side			SCUES Diff.^a
		SCUES	MA2	ρ (95%CI)^a p-value	SCUES	MA2	ρ (95%CI)^a p-value	Z-value p-value
Shoulder	2. Reach sideways to elevated position	2 (2–3)	3 (2–3)	0.65 (0.46 to 0.80) p < 0.001	3 (2–3)	3 (3–3)	0.45 (−0.11 to 0.76) p = 0.01	−4.00 p < 0.001
Elbow	14. Hand to mouth and down	2 (2–3)	3 (2–3)	0.58 (0.26 to 0.82) p = 0.001	3 (2–3)	3 (2.75–3)	0.30 (−0.18 to 0.67) p = 0.11	−3.69 p < 0.001
Forearm	12. Pronation / Supination	2 (2–3)	4 (4–4)	0.39 (−0.07 to 0.70) p = 0.03	2.5 (2–3)	4 (4–4)	0.19 (–) p = 0.33)	−3.65 p < 0.001
Wrist	7. Release of pellet	2 (2–2)	3 (2–3)	0.62 (0.41 to 0.77) p < 0.001	2.5 (2–3)	3 (3–3)	0.53 (0.29 to 0.70) p = 0.003	−3.46 p = 0.001
Fingers/Thumb	5. Release of crayon	2 (2–3)	3 (2–3)	0.51 (0.19 to 0.74) p = 0.004	3 (2–3)	3 (3–3)	0.23 (−0.22 to 0.58) p = 0.22	Z= −3.16 p = 0.002
Sum	Sum items 2, 14, 12, 7, and 5	11 (9.75–13.25)	15 (11.75–16.00)	0.77 (0.56 to 0.88) p < 0.001	13 (11–15)	16 (15–16)	0.46 (0.15 to 0.69) p = 0.01	−4.87 p < 0.001
Sum	Total MA2 ROM subscale score		24 (18.25–26.25)	0.83 (0.69 to 0.91) p < 0.001		25.5 (24-27)	0.70 (0.42 to 0.88) p < 0.001

The SCUES and the MA2 were tested on the same day, except for one participant where the MA2 was performed one day later. Unlike the SCUES limb and BBT scores, the MA2 data were not normally distributed. SCUES: Selective Control of the Upper Extremity Scale; MA2: Melbourne Assessment 2; ROM: range of motion; ρ: non-parametric Spearman’s correlation.

^aSCUES Diff. shows the Z-values of the Wilcoxon signed-rank tests and the p-values calculated for the differences between the SCUES scores of the more and less affected side.

For the intra-rater reliability (46 participants rated twice), the median time between the independent evaluations of the video recordings amounted to 54 days with a minimum of 18 days. For the inter-rater reliability, 46 participants were rated by three raters independently from each other. For the test–retest reliability (27 participants assessed twice), the median time between assessments was 7 days, and the minimal interval at least 1 day.

The relative reliability of limb and single item scores was high, especially for the more affected side, and for intra-rater and test–retest reliability compared to inter-rater reliability (Table 3, Figure 1(C,D,E)). The SDC values for the more affected side indicated that for the limb scores, a change of 2, 3, or 4 points could be considered a true change for test–retest, intra-rater, and inter-rater reliability, respectively. For the individual joints, a change of 1 or 2 points should be considered a true change (Table 3).

Table 3. Reliability of SCUES limb and item scores.

	Intra-rater				Inter-rater				Test–retest
	More affected		Less affected		More affected		Less affected		More affected		Less affected
SCUES	ICC	SDC	ICC	SDC	ICC	SDC	ICC	SDC	ICC	SDC	ICC	SDC
Shoulder	0.91	0.58	0.81	0.82	0.66	1.13	0.68	1.03	0.82	0.85	0.84	0.71
Elbow	0.98	0.29	0.79	0.76	0.64	1.11	0.60	0.97	0.82	0.85	0.80	0.75
Forearm	0.87	0.96	0.93	0.41	0.82	1.16	0.60	1.00	0.89	0.99	0.70	1.01
Wrist	0.81	1.01	0.97	0.29	0.80	1.16	0.67	1.00	0.84	1.12	0.94	0.44
Fingers/ Thumb	0.90	0.71	0.85	0.65	0.77	1.13	0.64	1.03	0.89	0.82	0.67	0.96
Limb score	0.93	2.55	0.97	1.13	0.86	3.58	0.78	3.05	0.98	1.41	0.95	1.62

SCUES: Selective Control of the Upper Extremity Scale; ICC: intra-class correlation coefficient; SDC: smallest detectable change.

Discussion

We aimed to determine the concurrent and discriminative validity and intra-rater, inter-rater, and test–retest reliability of the SCUES limb and item scores in children and adolescents with an UMNL. To investigate the validity, we had formulated three hypotheses: (i) correlations between the SCUES limb score and the ROM subscale of the MA2 are higher (similar construct) than between the SCUES and Box and Block Test (BBT), (ii) SCUES item scores correlate at least moderately with congruent MA2 item scores. (iii) Limb and item scores are lower for the more affected side.

We can confirm our first hypothesis, as for the more affected side, the SCUES limb scores correlated higher with the MA2 subscale than with the BBT. The difference was smaller as expected, as Wagner et al. [16] found a correlation of 0.47 with the BBT. We cannot explain the difference because, besides the diagnoses, most patient characteristics were comparable between their and our study. We assume that the true value lies somewhere in the overlapping confidence intervals. Still, correlations with the MA2 were high, indicating that the SCUES can assess SVMC also in a more heterogeneous group of children with an UMNL. The correlations were slightly higher than those observed between the SCUES and the SHUEE (ρ = 0.69) [16] or the TASC and the ABILHAND-Kids logit scores (r = 0.596) [5] and comparable to the correlation between the SCUES and the Jebsen Taylor Hand Function Test (ρ= −0.80) [17].

To our knowledge, we are the first to provide information on the concurrent validity of single SCUES items. For the more affected arm, correlations with MA2 items were moderate to high, as hypothesized. Interestingly, the item with the lowest correlation was the pro-/supination movement, which appears identical in both tests. However, raters should only score the ROM of the pro-/supination movement in the MA2 (results see Table 2) and not involuntary movements, explaining the poorer correlation.

Finally, limb and item scores were significantly lower for the more compared to the less affected side, which was previously also observed for the SCUES [17] and TASC [5].

A previous study [17] reported SDC values of 2.11 for intra-rater reliability and 1.16 for inter-rater reliability. For the intra-rater reliability, our SDC value was comparable, but we had a higher value for the inter-rater reliability. While test–retest reliability was not investigated before, it appears excellent. Therefore, we recommend that the same rater should assess pre to post therapy-induced improvements in SCUES scores.

On the item level, the inter-rater SDC values exceeded 1 for each joint of the more affected side, indicating that an individual joint score should improve by at least 2 points to be considered a true change. To us, this seems clinically a true challenge. For most of the joints, the intra-rater and test-retest SDC values were below 1, indicating that an improvement of 1 point would be beyond regular variability. Only for the wrist joint, we found SDC values larger than 1. We assume that scoring the wrist was more difficult due to two factors: First, the frontal video recording makes it difficult to estimate the ROM of the wrist movement and determine whether the participants might have slightly lifted the lower arm from the foam block. Second, some raters found it difficult to decide whether co-occurring finger movements should be considered involuntary or physiological.

Methodological considerations

Regarding the poorer inter-rater reliability, we wondered whether the training was insufficient. Comparing our inter-rater ICC values for the more affected side with those reported by Wagner et al. [16] (shoulder 0.66 versus 0.72, elbow 0.64 versus 0.88, pro/supination 0.82 versus 0.84, wrist 0.80 versus 0.84, and finger/thumb 0.77 versus 0.83, respectively), the values seem comparable, despite that they used a different study design where six raters scored 10 videos. This might indicate that training might have been sufficient. However, our ICC values and those reported by Wagner et al. [16] are lower compared to the excellent inter-rater ICC values (0.92–0.98) reported by Yildiz et al. [17]. They reported even better inter-rater compared to intra-rater reliability results.

We discussed merely validity and reliability for the more affected side, however, the assessment of SVMC of the less affected side could also be relevant, particularly for patients who are bilaterally affected.

Subgroups based on the diagnoses were small, limiting statistical comparisons. Yet, based on the visual interpretation of Figure 1, we could not detect differences in psychometric test results between subgroups.

Unlike previous studies, we did not plan to compare the SCUES with the MACS for validity purposes because the MACS includes both hands, allows compensatory strategies in handling objects in daily life, and is developed for and psychometrically investigated in children with CP. In our study, MACS scores for participants with diagnoses other than CP should be considered a gross indicator of how our participants could handle objects in daily life. When correlating the SCUES with the MACS (ρ= −0.77, p < 0.001), we found a much stronger relationship as previously reported for the SCUES (ρ= −0.24, p = 0.37) [16] or TASC (ρ= −0.53, p < 0.001) [5], but similar to the correlation found by Yildiz et al. (ρ= −0.78, p < 0.001) [17]. Perhaps, this indicates that SVMC might be a relevant prerequisite for handling objects in daily life in patients with an UMNL.

As in the original study [16], the number of participants with MACS level III or IV was relatively small, so further psychometric evaluations are needed to determine whether the SCUES is valid and reliable for patients who are more severely affected. Finally, compared to the intra- and inter-rater reliability analyses, the numbers for validity and test–retest reliability were smaller.

Conclusion

The SCUES seems valid and shows acceptable reliability in patients with an UMNL who would in daily life be candidates for therapies aiming to ameliorate SVMC. Based on our results, we recommend that the same rater should perform pre-post assessments. Future studies should investigate the responsiveness to determine whether the SCUES can assess therapy-induced improvements in SVMC.

Acknowledgments

The authors greatly appreciate the participation of the children and their parents, who provided approval. This work could not have been done without the help of the occupational therapists Annina Herzog and Esther Zamarrón of the Swiss Children’s Rehab, as well as Annina Fahr and Julia Balzer of the Research Department, who assisted in some of the measurements. The authors are grateful to Lisa Wagner and Jon Davids, who developed the SCUES, for their feedback during the translation procedure. The authors are also grateful to Mrs Beate Gartmann, who performed one of the translations. The authors thank the reviewers for their constructive feedback.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was financially supported by the Swiss National Science Foundation (Projects 32003B_156646 and 32003B_179471), the J&K Wonderland Foundation (Steinhausen, Switzerland), and the Mäxi Foundation (Zurich, Switzerland).