Validity and reliability measures of the Swedish Karolinska version of the Edinburgh Cognitive and Behavioral ALS Screen (SK-ECAS)

Abstract

Objective

Cognitive and behavioral impairment is observed in up to 50% of patients with amyotrophic lateral sclerosis (ALS). The Edinburgh Cognitive and Behavioral ALS Screen (ECAS) is a 5-domain screening tool customized for quick cognitive screening in patients with ALS. Although the ECAS is available in Swedish at the Karolinska University Hospital (SK-ECAS), it has not yet been validated in Sweden stressing the need to assess validity and reliability of the SK-ECAS Version A.

Methods

The study included 176 patients with ALS or other motor neuron disease diagnosed between September 2017 and October 2021 at the Karolinska ALS Clinical Research Center in Stockholm, Sweden, and 35 age-matched healthy control subjects. SK-ECAS was validated against the Montreal Cognitive Assessment (MoCA) and optimal cutoffs, receiver operating characteristic (ROC) curve and area under the curve (AUC) were calculated.

Results

We identified an optimal cutoff of 108 for the SK-ECAS total score and 82 for the SK-ECAS ALS-specific score to detect cognitive impairment. The SK-ECAS showed good performance in indicating abnormal cognition with an AUC of 0.73 for SK-ECAS ALS-specific score and 0.77 for SK-ECAS total score. There was good internal consistency with a Cronbach’s alpha of 0.79.

Conclusions

This study demonstrates good validity and reliability indices for SK-ECAS Version A for the detection of cognitive impairment in newly diagnosed ALS patients.

Keywords:

• Amyotrophic lateral sclerosis
• cognition
• neuropsychology
• Sweden
• validation

Introduction

Amyotrophic lateral sclerosis (ALS) is a heterogenous neurodegenerative disease in which loss of motor neurons results in progressive paresis of skeletal muscles and ultimately death from respiratory failure (1). Although ALS primarily affects the motor system, other areas of the brain may also be affected (2,3). Approximately 50% of patients with ALS experience some degree of cognitive impairment during the disease course and up to 15% develop severe cognitive and behavioral symptoms fulfilling the criteria for frontotemporal dementia (FTD) (4,5).

The Montreal Cognitive Assessment (MoCA) is a commonly used and rapid screening test developed to assess mild stages of cognitive impairment (6). It enables to test executive function, learning, visuo-spatial skills and attention and is successfully administered in 79.9% of ALS patients. Clinically, the MoCA is mainly used to rule out mild cognitive impairment (MCI) (7). However, the progressive impairment and loss of specific cognitive functions commonly attributed to ALS are not well captured by MoCA or other similar instruments and may be confounded by ALS associated motor impairments, underscoring the need for an alternative method specific to ALS (8). Thus, the Edinburgh Cognitive and Behavioral ALS Screen (ECAS) was developed to assess both cognition and behavioral changes in patients with ALS, including a spectrum of cognitive domains (9). These include language, verbal fluency, executive function, memory, and visuospatial skills. It is validated for both verbal and written versions, which makes it a good screening instrument during the course of ALS. In addition to the first version of the ECAS (Version A), two additional versions were developed (Versions B and C) to avoid practice effects associated with repetition (10). ECAS has been translated to more than 24 languages and, although ECAS is available in Swedish, the validity of the Swedish version of ECAS used at the Karolinska University Hospital (SK-ECAS) has not been published. This version presents minor differences to the official Swedish ECAS version available on the ECAS website and recently used in publications (11,12).

The aim of this study was to assess the validity, reliability and predictive performance of the SK-ECAS in newly diagnosed ALS patients from Stockholm, Sweden using the MoCA as a reference.

Methods

Study population

The ALS Clinical Research Center (ALS CRC) at the Karolinska University Hospital in Stockholm, Sweden provides multidisciplinary ALS care and is a tertiary center for ALS patients in the Stockholm region with a population of over two million inhabitants. Between September 2017 and October 2021, 478 patients were diagnosed with ALS at the ALS CRC (adhering to the revised El Escorial criteria for clinically definite, probable, or possible ALS) and 98 patients with other motor neuron disease (MND) including primary lateral sclerosis and progressive spinal muscular atrophy. Of these, we included 145 ALS patients and 31 MND patients who had at least one MoCA assessment and one SK-ECAS Version A assessment. We decided to include patients with a diagnosis of other MND to maximize statistical power since sensitivity analyses excluding these patients showed comparable results (see results section and Supplementary material 1 and 2). We also included 35 age-matched healthy controls who were either siblings or spouses of the ALS patients. Both patients and healthy controls spoke Swedish as a native language. The study was approved by the Swedish Ethical Review Authority (2021-06397-02), completed in accordance with Helsinki Declaration and in respect of Good Clinical Practice, with all participants giving written informed consent.

Data collection

Data on patient characteristics and results on cognitive testing (ECAS and MoCA) were extracted by the first author from the Swedish Motor Neuron Disease Quality Registry (SMNDR) which included 99% of ALS/MND patients in the Stockholm region as of 2017 (13). ECAS and MoCA were assessed by the same research assistant nurse for all patients.

ECAS assessment

The ECAS takes about 20 min to administer and is divided into two sections: an ALS-specific section consisting of 100 points and a non-ALS-specific section consisting of 36 points. Both sections are then combined to reach an ECAS total score with a maximum of 136. The ALS-specific section includes three domains: executive functions (reverse digit span, social cognition, alternation, and inhibitory sentence completion), fluency (free fluency and restricted fluency), and language (naming, comprehension, and spelling). The non-ALS-specific section includes memory (immediate recall, delayed recall, and delayed recognition) and visuospatial functions (dot counting, cube counting, and number location). We used ECAS Version A as this was the only version translated to Swedish at the time of the first assessment (September 2017). Five potential cutoffs indicating abnormality were investigated, including a total score of 105, 107, 108, 110, and 115 and an ALS-specific score of 77, 78, 80, 82, and 83, as described previously for ECAS version A (14).

MoCA assessment

The MoCA takes around 10 min to administer and consists of 30 points reflecting visuospatial abilities (4 points), executive function tasks (4 points), a short-term memory recall task (5 points), an attention task (1 point), a concentration task (3 points), a working memory tasks (2 points), an orientation task (6 points) and language tasks (5 points) (6). As visuospatial abilities are evaluated through drawing, patients who could not hold a pen and draw did not complete the MoCA and were therefore not included in this study. We used previously established cutoffs to define abnormalities in MoCA (<27) (6).

Statistical analyses

Categorical variables were summarized as proportions (percentages) and the chi-square test was performed to assess differences between groups. Continuous variables were reported as mean with standard deviation (SD). Normality assumption was met, and the independent samples t-test was used for groups with unequal variances to assess for differences between groups. We used Spearman’s rank correlation to assess the monotonous relationship between continuous variables. To assess internal consistency we used Cronbach’s alpha, considering a coefficient of 0.70–0.80 to be satisfactory (15). Logistic regression and receiver operating characteristics (ROC) curves were used to assess the sensitivity and specificity of the SK-ECAS to detect cognitive impairment, and the area under the curve (AUC) was calculated. A MoCA score of less than 27 was used as gold standard to define cognitive impairment (6). To identify cutoffs for ECAS abnormalities, the ROC curves were fit against MoCA. Similar cutoffs as in Niven et al. were investigated. For the total score we looked at cutoffs of 105, 107, 108, 110 and 115. For the ALS-specific score we looked at cutoffs of 77, 78, 80, 82, 83. We also performed a sensitivity analysis excluding the 31 patients with other MND.

Results

Participant characteristics

Among the ALS/MND patients there were slightly more males (56.2%), with a mean age of 64.2 years (SD: 11.5) and a mean ALSFRS-R of 37.2 (SD: 8.3). These variables did not differ between patients with ALS and those with other MND (Table 1). The ALS/MND patients did not differ substantially from healthy controls in age or sex distribution.

Table 1 Participant characteristics and scores in cognitive testing (ECAS and MoCA).

	ALS/MND patients (n = 176)	ALS patients (n = 145)	MND patients other than ALS (n = 31)	Healthy controls (n = 35)
Sex, number (%)
Female	77 (43.8)	68 (46.9)	9 (29.0)	21 (60.0)
Male	99 (56.2)	77 (53.1)	22 (71.0)	14 (40.0)
Age in years, mean (SD)	64.2 (11.5)	64.3 (11.4)	63.8 (11.8)	61.7 (11.6)
MoCA score, mean (SD) (max 30)**	26.4 (2.7)	25.9 (2.9)	26.5 (2.3)	28.0 (1.8)
ALSFRS-R score at time of ECAS measure, mean (SD)	37.2 (8.3)	36.7 (8.5)	39.7 (6.9)	–
ECAS total score, mean (SD) (max 136)**	107.6 (14.1)	106.6 (14.6)	112.3 (10.5)	120.8 (6.4)
ECAS ALS-specific score, mean (SD) (max /100)**	79.9 (11.1)	79.1 (11.5)	83.8 (8.1)	90.1 (4.7)
ECAS non-ALS-specific score, mean (SD) (max 36)*	27.7 (5.5)	27.6 (5.7)	28.5 (4.4)	30.7 (3.0)

ALS: amyotrophic lateral sclerosis; MND: motor neuron disease; ECAS: Edinburgh Cognitive and Behavioral ALS Screen; MoCA: Montreal Cognitive Assessment; ALSFRS-R: ALS functional rating scale revised.

* p < 0.01 comparing ALS/MND patients to healthy controls.

** p < 0.001 comparing ALS/MND patients to healthy controls.

ECAS score

Evaluating cognitive function was significantly different between ALS/MND patients and healthy controls, e.g., an SK-ECAS total score of 107.6 versus 120.8 (p < 0.001) and an SK-ECAS ALS-specific score of 79.9 versus 90.1 (p < 0.001).

The mean SK-ECAS total score was 107.6 (SD: 14.1) in ALS/MND patients, compared to 120.8 (SD: 6.4) in healthy controls (p < 0.001). This difference was primarily explained by a lower ALS-specific score (mean: 79.9; SD: 11.1 in patients vs. mean: 90.1; SD: 4.7 in controls) (p < 0.001) while the difference was smaller in the non-ALS-specific score (mean: 27.7; SD: 5.5 versus mean: 30.7; SD: 3.0) (p < 0.01). The difference in MoCA score was statistically significant with a mean score of 26.4 (SD: 2.7) in ALS/MND patients and a mean score of 28.0 (SD: 1.81) in healthy controls (p < 0.001).

Internal consistency

In ALS/MND patients, there was a very strong correlation between MoCA score and SK-ECAS total score ($\varrho$=0.92, p < 0.0001) highlighting convergent validity as well as between MoCA score and ALS-specific score (p < 0.0001). There was a lower correlation between ALS nonspecific scores ($\varrho$=0.92 and 0.62, p < 0.0001) highlighting discriminant validity (Table 2). A similar pattern was observed for the healthy controls.

Table 2 Spearman correlation coefficients and p values between different cognitive measures among 176 ALS/MND patients and 35 healthy controls.

	SK-ECAS total	SK-ECAS ALS-specific	SK-ECAS non-ALS-specific	MoCA
SK-ECAS total
ALS/MND patients	1	0.92, p < 0.0001	0.62, p < 0.0001	0.54, p < 0.0001
Healthy controls	1	0.94, p < 0.0001	0.69, p < 0.0001	0.53, p = 0.0012
SK-ECAS ALS-specific
ALS/MND patients	0.92, p < 0.0001	1	0.30, p < 0.0001	0.44, p < 0.0001
Healthy controls	0.94, p < 0.0001	1	0.43, p = 0.0091	0.46, p = 0.0059
SK-ECAS non-ALS-specific
ALS/MND patients	0.62, p < 0.0001	0.30, p < 0.0001	1	0.47, p < 0.0001
Healthy controls	0.69, p < 0.0001	0.43, p = 0.0091	1	0.41, p = 0.0138
MoCA
ALS/MND patients	0.54, p < 0.0001	0.44, p < 0.0001	0.47, p < 0.0001	1
Healthy controls	0.53, p = 0.0012	0.46, p = 0.0059	0.41, p = 0.0138	1

ALS: amyotrophic lateral sclerosis; MND: motor neuron disease; SK-ECAS: Swedish Edinburgh Cognitive and Behavioral ALS Screen at the Karolinska University Hospital; MoCA: Montreal Cognitive Assessment.

As for internal reliability, the Cronbach’s alpha was 0.79 (raw value: 0.74) for SK-ECAS among ALS/MND patients.

Predictive performance

When regressing SK-ECAS on MoCA using a cutoff value of 27 on MoCA, the AUC was 0.73 for SK-ECAS ALS-specific score and 0.77 for SK-ECAS total score (Figure 1).

Figure 1 ECAS ROC detecting cognitive impairment defined as a pathologic MoCA score. (A) ECAS total score (best cutoff = 108). (B) ECAS ALS-specific score (best cutoff = 82). ECAS: Edinburgh Cognitive and Behavioral ALS Screen; ROC: Receiver operating characteristic; MoCA: Montreal Cognitive Assessment.

The optimal cutoff was 108 for SK-ECAS total score and 82 for SK-ECAS ALS-specific score (Table 3).

Table 3 Sensitivity, specificity, and predictive values of cutoff values for SK-ECAS ALS-specific and total scores.

Score	Cutoff	Sensitivity	Specificity	Positive predictive value	Negative predictive value	Point Estimate	95% Wald confidence limits
SK-ECAS total score	105	0.58	0.84	0.73	0.73	0.134	0.066	0.272
	107	0.61	0.80	0.69	0.74	0.157	0.080	0.309
	108	0.69	0.78	0.70	0.78	0.124	0.063	0.245
	110	0.72	0.73	0.65	0.78	0.150	0.077	0.292
	115	0.81	0.47	0.53	0.77	0.263	0.130	0.528
SK-ECAS ALS-specific score	77	0.51	0.80	0.65	0.69	0.231	0.118	0.451
	78	0.51	0.80	0.65	0.69	0.231	0.118	0.451
	80	0.59	0.75	0.63	0.72	0.233	0.123	0.444
	82	0.66	0.72	0.63	0.74	0.203	0.106	0.387
	83	0.73	0.62	0.58	0.76	0.229	0.120	0.439

ALS: amyotrophic lateral sclerosis; ECAS: Edinburgh Cognitive and Behavioral ALS Screen.

Sensitivity analyses

Excluding the 31 patients with other MND from the analyses did not show any substantial changes in the results. The SK-ECAS total score and the scores of the individual items were not notably different and there was an identical Cronbach’s alpha (Supplementary Table 1). The correlation coefficients between SK-ECAS and MoCA were also similar, e.g., 0.54 and 0.52 for SK-ECAS total score including all ALS/MND patients and ALS patients only, respectively (Supplementary Table 2).

Discussion

The ECAS has been translated into more than 24 languages and is currently standard of care in many ALS care teams across Europe (16). ECAS is also increasingly being used in ALS clinical drug trials both for screening purposes and as an outcome measure, stressing the importance of validations of the various ECAS versions. This study shows good validity and performance indices for SK-ECAS for detecting cognitive dysfunction in early-stage ALS patients in Sweden.

Using a MoCA score below 27 as reference for cognitive impairment, the AUC was 0.73 for SK-ECAS ALS-specific score and 0.77 for SK-ECAS total score. Our results were similar to previous ECAS validation studies with comparable methodology (Supplementary material 3). Two out of the five previous validation studies reported AUC values of 0.87 and 0.75 for the ECAS total score (17,18). The Cronbach’s alpha of 0.79 observed in our study was higher or equal to values reported in four of the previous studies (0.74, 0.77, 0.78 and 0.79, respectively) (17–20) although lower compared to one of them (0.86) (21). The optimal cutoff in our study was 108 for SK-ECAS total score and 82 for SK-ECAS ALS-specific score, i.e., a 3- and 5-point difference as compared to the previously published cutoff of 105 and 77 for the English version of the ECAS (14). This might be explained by the fact that our study only used MoCA to validate SK-ECAS, while Niven et al. performed a more extensive neuropsychological assessment. For this reason, additional confirmation of the cutoffs presented in this study is needed.

Among the 145 patients in the present study, 41.4% showed abnormal total and ALS-specific scores using the optimal cutoff observed, compared to 29.2% in the original ECAS publication using the English version cutoff (9). Further, 47.8% of patients in our study showed abnormal ECAS total score, with a higher proportion than any of the five existing validation studies (ranging from 34.4% to 43.3%). These findings might indicate that our study sample was more cognitively impaired but may also be due to differences in sample composition (e.g., population-based sample in the present study) and other factors (e.g., different reference used for defining cognitive impairment).

Including 145 patients, our study was larger than the five previous validation studies which included between 30 and 107 ALS patients. Our study however has several limitations. First, our cohort was composed of only patients from the Stockholm region and thus may not be representative of the overall ALS population in Sweden. Second, we used the MoCA as reference for cognitive impairment as opposed to more extensive neuropsychological evaluation. Third, our study only considered SK-ECAS Version A, as S-ECAS Versions B and C only became publicly available in 2021 and further studies are needed to validate these. Fourth, our data collection did not include the behavioral interview composing the ECAS, as ECAS behavioral data started to be collected in the Spring 2023 at our site. Last, more research is needed to explore the cognitive profile of the Swedish ALS population, both at diagnosis and as the disease progresses.

Conclusion

This study shows good validity and performance indices for SK-ECAS Version A for the detection of cognitive impairment in newly diagnosed ALS patients in Stockholm, Sweden. The proportion of abnormal ECAS total score was higher in our study compared to previously published validation studies. As this study was conducted in a moderately sized sample from a restricted geographical area, further research is needed to explore the cognitive profile of the entire Swedish ALS population.

Acknowledgment

The authors report no competing interests. This project was funded thanks to the Ulla-Carin Lindquists foundation for ALS research, grant number 2021:1. We would like to thank Jenny Hellqvist for her precious help during the data collection process. We would also like to thank the Umeå research team for their Swedish translation of the ECAS. And most important, we are grateful to the many patients and healthy control individuals who contributed to this study.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.