ECAS A-B-C: alternate forms of the Edinburgh Cognitive and Behavioural ALS Screen

Abstract

Background: The Edinburgh Cognitive and Behavioural ALS Screen (ECAS) is a short assessment by which neuropsychological symptoms can be detected and quantified in people with ALS. To avoid potential practice effects with repeated administration, here we present alternative versions of the ECAS suitable for measuring change over time. Objective: To develop two alternate versions of the ECAS: ECAS-B and ECAS-C. Method: One hundred and forty-nine healthy adult participants were recruited. Thirty participants completed a pilot study in developing the alternate versions. Two groups of 40 participants were administered the ECAS-B or ECAS-C and compared to published data of the original ECAS (ECAS-A) to determine equivalence. An additional 39 participants were administered the ECAS consecutively, either repeating the original version (ECAS-A-A-A) serially or the different versions (ECAS-A-B-C) to determine potential practice effects. Recordings of assessments were scored by a second researcher to determine inter-rater reliability. Results: No significant differences were found between versions (A, B, C) of the composite performance measures of ALS Specific, ALS Non-Specific, and ECAS Total scores. Repeated serial administration of ECAS-A (A-A-A) produced some practice effects for composite scores, whereas no such effects were found when alternate versions were administered serially (A-B-C). Exceptionally high intra-class correlations were found for all three versions of the ECAS suggesting a high degree of rater agreement. Conclusion: The newly developed alternate forms of the ECAS are both highly equitable to the original ECAS-A and enable avoidance of practice effects, thus supporting their use in measuring cognition and behaviour over time.

Keywords:

• Cognition
• behaviour
• screen
• ECAS
• alternate forms
• reliability

Introduction

Up to 50% of patients with ALS will experience changes in cognition and/or behaviour. Considerable clinical (1,2), genetic (3), pathological (4), and neuropsychological data (2–5) have demonstrated that ALS and frontotemporal dementia (FTD) significantly overlap. The observed cognitive and behavioural changes in ALS parallel those observed in frontotemporal dementia, namely, deficits in executive functions, language functions, verbal fluency, and social cognition (6–9). Similarly, behavioural features of ALS include apathy, perseveration, and disinhibition (10–12). Despite this overlap, cognitive and behavioural symptoms in ALS do not always fall neatly into the three recognized FTD subtypes: behavioural variant FTD, non-fluent progressive aphasia and semantic dementia, raising the question whether ALS/FTD might be more than a simple juxtaposition on ALS and FTD (13). This underlines the importance of an ALS-appropriate cognitive and behavioural assessment.

However, the assessment of cognition in ALS has been historically difficult due to the ubiquitous requirement for intact motor functioning in neuropsychological assessment. The Edinburgh Cognitive and Behavioural ALS Screen (ECAS) has been recently developed to overcome this issue (14). The ECAS has been designed to measure cognitive functions, unrestricted by physical disability (15), that are commonly affected in ALS (executive functioning, language functioning, and verbal fluency) in addition to functions less commonly affected (memory and visuospatial functions). Additionally, the ECAS includes a clinical caregiver behaviour interview based on diagnostic criteria for FTD (16). Although ECAS has been primarily designed for use in ALS, it may be useful in all patients in whom motor dysfunction might influence their performance on cognitive tests, e.g. Parkinsonism or paraplegia. The ECAS is a short screening tool designed with high clinical utility and is administrable by non-neuropsychological health care professionals. It has been validated against a comprehensive neuropsychological battery in Scottish (17), German/Swiss-German (15,18), Italian (19), Chinese (20), and Irish populations (21).

Given the brevity of the ECAS and its accommodation for physical disability, it may be suitable for measurement of changes in symptoms over the course of the disease. Cognitive dysfunction may have important implications for patient management, treatment fidelity, power of attorney, and end-of-life decision making (22–24). Behaviour change has been linked to increased carer burden (25,26) and shortened survival (27,28). As such, the accurate assessment of cognition and behaviour over time is of vital importance to meeting the needs of patients and their families. However, it has been well documented that the repeated administration of the same neuropsychological test can result in an improvement in performance (29). This improvement, termed practice effects, may result from 1) learning the content of test items, e.g., remembering the content of a prose story to be remembered; and 2) development of test-taking strategies (30,31). With regard to ALS, practice effects may mask subtle deteriorations in cognition, or exaggerate improvements due to intervention. Recently, Burkhardt et al. (2016) demonstrated the presence of practice effects with the ECAS whereby participants’ performance significantly improved over serial assessments of six months (32).

A common method for overcoming practice effects is the development of alternate versions of a test in which elements of the test are changed while retaining characteristic features and level of difficulty (33). The aim of this study was to develop alternate forms of the ECAS to permit repeated assessment of cognitive functions in ALS over time, and for the accurate monitoring of cognitive and behavioural progression during the disease course. Specifically, this study aimed to: (1) present two alternate versions of the ECAS (ECAS-B and ECAS-C); (2) investigate the equivalency of the ECAS alternate forms to the original ECAS (ECAS-A); (3) investigate whether alternate forms of the ECAS reduce practice effects during serial administration compared to repeated administration of the original ECAS; and (4) investigate the inter-rater reliability of all three versions of the ECAS.

Methodology

Participants

One hundred and forty-nine healthy adults were recruited prospectively and matched by age, gender, and education to that of the original publication of the ECAS (14). Participants were representative of the demographic profile of ALS patients. Additionally, the previously published (retrospective) data on the ECAS (n = 40) were included in this study (14), resulting in a total sample size of 189 participants. Participants were free of current or past neurological or psychiatric conditions, reading/writing disabilities, and were not a blood relative of a person with ALS. Participants were recruited from a volunteer panel held by Edinburgh University, in addition to local charitable organizations and community noticeboards.

Development of the ECAS-B and ECAS-C

The ECAS cognitive screen consists of 15 subtests measuring five cognitive domains, namely: Language, Verbal Fluency, Executive (ALS-Specific) and Memory, Visuospatial (ALS Non-Specific) functions. To develop alternate versions of the ECAS cognitive screen, a pool of alternate stimuli was generated for each subtest and piloted on a sample of healthy adults. Stimuli selection and development is described in supplementary materials. Arrays of stimuli were carefully selected and formed into two alternate ECAS versions, the ECAS-B and the ECAS-C. Selection of stimuli was based on an item-by-item and group-level exploration of response accuracy, in addition to retaining semantic and linguistic characteristics present in ECAS-A. The ECAS A, B, and C and guidelines for usage are available on http://ecas.psy.ed.ac.uk.

Procedure

Participants were recruited into six consecutive groups across three study phases (Table 1). In phase 1, a pool of alternate stimuli was generated to produce two alternate forms of the ECAS (ECAS-B and ECAS-C) and were administered to a sample of 30 participants to broadly determine equivalence in performance between corresponding sets of items in these two versions. In phase 2, the ECAS-B and ECAS-C were administered to two prospectively recruited groups of 40 participants matched by age, gender, and education to the data of 40 healthy controls whose data were previously used to establish normative data for the ECAS-A (14).

Table 1. Description and function of participant groups.

Phase	Function	Procedure	n
Phase 1: Pilot study	ECAS-B and ECAS-C formation	Administered array of possible stimuli	30
Phase 2: Establishing normative data	Establish normative data and equivalence of alternate forms	Administered ECAS-A	40
		Administered ECAS-B	40
			Administered ECAS-C	40
Phase 3: Exploring practice effects	Measure relative practice effects of administering same versus different ECAS forms	Administered ECAS-A serially	20
		Administered ECAS-A, ECAS-B, and ECAS-C sequentially	19

In phase 3, an additional 39 participants were randomly assigned to one of two conditions. Participants were either administered the ECAS-A three times consecutively (A-A-A), or administered alternate forms of the ECAS (A-B-C). As practice effects have shown susceptibility to short retest intervals (e.g. see Calamia, Markon, & Tranel, 2012) and to maximize the possibility of detecting such effects, participants were administered the ECAS repeatedly during the same sitting. Phase 3 testing for each participant lasted approximately 50 min, limiting the possibility of fatigue. Between each ECAS administration for both groups, participants completed a 5-min visual-search distractor task or a 5-min rest to further reduce the possibility of fatigue. Additionally, all prospective participants were administered the Test of Premorbid Functioning (TOPF) as an estimate of Full-Scale IQ (FSIQ) (34).

The inter-rater reliability of all forms (A, B, and C) of the ECAS was additionally explored. A subset of participants consented to having their assessment session audio-recorded (n = 94). These audio recordings were then scored by a second rater, trained to administer and score the ECAS by the scale’s authors (14). Both raters (RR and CC) were experienced in the administration and scoring of the ECAS. When audio recordings were unclear or given in written format, raw unscored paper forms were provided.

All participants provided informed written consent and this research was approved by the Psychology Research Ethics Committee of the University of Edinburgh.

Statistical analysis

Demographic data and estimated FSIQ were compared across groups using a χ² test for categorical data and one-way analysis of variance (ANOVA) for continuous data. For all analyses, when distributions or residuals violated statistical assumptions, power- or log-transformations were applied. When transformations failed to correct violations of test assumptions, non-parametric alternatives were used. Analyses were conducted using R 3.3.2. In all cases, alpha was set to 0.05.

To explore the equivalence of the ECAS-A, ECAS-B and ECAS-C forms, three analysis methods were employed on the scales’ targeted domains (language, executive functioning, fluency, memory, visuospatial), as well as ALS-Specific, ALS Non-Specific, and ECAS Total scores. A one-way ANOVA or Kruskal-Wallis test was used to compare the alternate forms’ means or medians (as appropriate). Kolmogorov-Smirnov tests were employed to compare the shape and spread of the distribution for ECAS-B and ECAS-C compared to ECAS-A. Standard null hypothesis significance testing does not directly assess the equivalence of data, but rather tests the evidence against the null. As such, the one-way ANOVA and Kolmogorov-Smirnov tests were employed to assess whether means and distribution of scores on the ECAS alternate forms significantly differ. Consequently, a Bayesian ANOVA was employed to directly test the null hypothesis and examine the probability that the ECAS alternate forms are the same. Bayes factors for the null hypothesis were calculated using medium prior of 0.7. Due to significant rates of ceiling effects in the Language and Visuospatial domains of the ECAS, Fisher’s exact test for count data was used.

Possible practice effects of using ECAS A-A-A versus ECAS A-B-C were explored using a mixed effects model with Time and Group (A-A-A versus A-B-C) and a random intercept and slope fitted for each participant. To explore the differential impact of Group the interaction term (Time*Group) was added to the model. p values were obtained for the mixed effect model by likelihood ratio tests of the full model (Time*Group) against a reduced model without the interaction term.

Finally, inter-rater reliability of all three forms of the ECAS was explored using intra-class correlation (ICC) to determine the degree of agreement between two independent raters. ICCs and their 95% confidence intervals were calculated based on mean-rating, absolute-agreement, two-way random-effects models (35).

Results

ECAS B and ECAS C: Normative data and equivalency

Prospectively recruited participants (n = 80) were randomly assigned to one of two groups and matched by age, gender, and education to a third retrospectively collected group (n = 40). No significant differences were observed for background demographic data, nor for estimated FSIQ between the two prospectively recruited groups (Table 2).

Table 2. Demographic characteristics of independent ECAS A, B, and C groups.

	ECAS-A (n = 40)	ECAS-B (n = 40)	ECAS-C (n = 40)	p
Age	59.20 ± 12.58	60.20 ± 15.32	58.52 ± 14.28	0.856
Gender  (Male)	45%	45%	43%	0.967
Education  (Years)	12.28 ± 2.52	13.84 ± 3.25	13.38 ± 3.25	0.086
TOPF*	–	106.51 ± 11.90	105.46 ± 9.81	0.670

* TOPF (Test of Premorbid Function) unavailable for retrospective data. Welch t-test applied.

Mean performance for each ECAS cognitive domain across alternate forms was similar (Table 3). Results of one-way ANOVAs, Kruskal-Wallis, and Kolmogorov Smirnov tests demonstrated no significant differences between forms in the domains of Fluency, Executive Functions, and Memory. Additionally, no significant differences were observed for the ALS Non-Specific, ALS Specific, and ECAS Total composite scores. Fisher’s exact test for Language revealed no significant difference in the frequency of scores obtained (p = 0.147). Conversely, the Visuospatial domain was significantly different across ECAS versions (p = 0.013). This difference was, however, entirely driven by a larger proportion of participants for ECAS-B and ECAS-C making a single error (i.e. scoring 11 out of 12).

Table 3. Comparison of performance across independent groups for the ECAS A, B, and C.

Domain	ECAS-A	ECAS-B	ECAS-C	ANOVA	KS A-B	KS A-C	Bayes BF01
Language	27.62 ± 0.70	27.18 ± 1.15	27.12 ± 0.99	–	–	–	–
Fluency	19.85 ± 2.50	19.70 ± 2.99	20.45 ± 3.09	0.190*	0.999	0.914	6.72
Executive Functions	40.48 ± 3.54	40.23 ± 4.05	39.77 ± 3.70	0.703	0.999	0.759	9.43
Memory	18.68 ± 2.73	18.62 ± 2.17	18.30 ± 3.21	0.906	0.573	0.914	10.52
Visuospatial	11.85 ± 0.48	11.45 ± 0.81	11.43 ± 0.90	–	–	–	–
ALS-Specific	87.95 ± 4.98	87.10 ± 5.77	87.35 ± 5.34	0.775	0.914	0.914	10.14
ALS Non-Specific	30.52 ± 2.96	30.07 ± 2.39	29.73 ± 3.62	0.519	0.164	0.759	7.09
ECAS Total	118.47 ± 6.64	117.17 ± 7.15	117.08 ± 7.12	0.610	0.164	0.573	8.34

* Kruskalskal-Wallis test. KS = Kolmogorov-Smirnov test. BF₀₁ = Bayes factor for the null hypothesis. Due to ceiling effects, statistical analysis here was not appropriate for language and visuospatial functions.

Thresholds for impairment for the alternate versions (ECAS-B and ECAS-C) demonstrate parity across all versions using both 2 standard deviations and the 95th percentile. Cut-offs for impairment are retained from ECAS-A for the ECAS-B and ECAS-C (Supplementary Table 2).

Practice effects

Participants were randomly assigned to one of two conditions; the ‘same’ group received ECAS-A three times serially (A-A-A), while the ‘different’ group was administered ECAS-A followed by ECAS-B and ECAS-C (A-B-C). No significant differences were observed between groups in age, gender, and education (Table 4). One-way repeated analysis of variance for the ECAS A-A-A group demonstrated a significant improvement over time for ALS Specific (F(2,38) = 5.68, p = 0.007), ALS Non-Specific (F(2,38) = 100.42, p < 0.001), and ECAS Total Scores (F(2,38) = 25.88, p < 0.001) as displayed in Figure 1. Additionally, the executive and memory subdomains and the majority of their subtests demonstrated a significant improvement over time (See Supplementary Table 3). No significant differences were observed in ALS Specific, ALS Non-Specific, or ECAS Total Scores for participant in the ECAS A-B-C group, nor any cognitive subdomains or subtests.

Figure 1. Comparison of practice effects using the same (A-A-A) versus different (A-B-C) versions of the ECAS.

Table 4. Demographic characteristics of ECAS A-B-C (different) and ECAS A-A-A (same) groups.

	Different (n = 19)	Same (n = 20)	p
Age	55.00 ± 10.32	57.25 ± 12.67	0.546
Gender (Male)	57.89%	50%	0.863
Education (Years)	16.21 ± 2.52	15.53 ± 2.54	0.401
TOPF	113.19 ± 10.20	107.30 ± 10.63	0.099

TOPF = Test of Premorbid Functioning.

A significant group difference was observed in baseline ECAS-A Total score (t(36.72) = 3.03, p = 0.005) with those in the ECAS A-B-C group performing better than the ECAS A-A-A. However, a six-point difference was observed between groups for estimated FSIQ. While this did not reach statistical significance, a linear regression model demonstrated a significant positive effect of IQ on ECAS Total Score (F(1,34) = 13.67, p < 0.001, β = 0.449) explaining 28.67% of the variance.

Due to this small but significant difference in baseline performance between groups, a linear mixed effect model was fit. The addition of an interaction term (Group*Time; b = 3.44, SE = 0.859) significantly contributed to the fit of the ECAS Total model (χ²(1) = 13.43, p < 0.001). A similar significant Time*Group interaction was observed for ALS Non-Specific functions (b = 2.25, SE = 0.341, p < 0.001), but not for ALS Specific functions (b = 1.20, SE = 0.77, p = 0.127).

The effect of Time*Group was significant for ECAS Total and ALS Non-Specific functions, even accounting for a random intercept and slope for each participant (i.e. individual variation in baseline performance and rate of change), suggesting that the rate of improvement of using the ECAS-A serially is significantly greater than using the ECAS alternate forms.

Inter-rater reliability

Mean-rating, absolute-agreement, two-way random effects ICC models were generated for each cognitive domain and version of the ECAS. Across all versions of the ECAS and for all cognitive domains, inter-rater reliability was excellent ranging from 0.930 to 0.998. Supplementary Table 1 displays the respective ICC, 95% confidence intervals, and model statistics for each comparison; in all cases p < 0.001 indicated significant agreement between independent raters.

Discussion

The ECAS was developed to accurately assess cognitive functions in patients with ALS while controlling for motor disability. Not only has the ECAS shown high sensitivity and specificity to cognitive impairment against a full neuropsychological battery (17–21), it has high clinical utility in describing the nature of these impairments. Monitoring progression of cognitive and behavioural symptoms may have important implications for patient management, treatment, prognosis, end-of-life decision making, and caregiver burden (22–28). The purpose of this study was to develop alternate forms of the ECAS to facilitate repeated assessment and longitudinal monitoring of cognition and behaviour in patients with ALS. Particularly, the aims were to present and determine equivalency of two alternate forms of the ECAS (ECAS-B and ECAS-C) to the original ECAS-A, to investigate whether alternate forms of the ECAS reduce practice effects relative to the ECAS-A, and to investigate the inter-rater reliability of all three forms of the ECAS.

The findings in this study provide strong evidence that the newly developed ECAS-B and ECAS-C are equivalent to that of the original ECAS-A. Results of independent group analysis suggest that (1) performance on the alternate forms does not significantly differ from the original ECAS; and (2) there is strong evidence that the alternate forms come from the same distribution of scores. While a single significant difference was observed for the visuospatial domains of the ECAS, examination of the score distribution revealed that this is due to ceiling effects and driven by a one- point difference in the alternate versions, therefore not affecting the equivalence of the alternate forms.

To establish the utility of the alternate forms in reducing practice effects, the ECAS-A was administered serially to a group of participants and compared to a separate group who were administered the alternate versions of the ECAS. Results of this study suggest significant practice effects exist for the ECAS-A when administered serially. This finding is in agreement with recent research demonstrating that the ECAS-A is susceptible to practice effects with repeated administration (32). The present study was designed to maximize the possible detection of practice effects as short intervals have been shown to exacerbate such an effect (29). However, no significant change in performance was detected over time when alternate versions of the ECAS were administered. Additionally, a significant Time by Group (i.e. time representing repeated assessment and group representing participants who received the same or different versions of the ECAS) interaction was observed when the ECAS A-A-A group was compared to the ECAS A-B-C group. The mixed effects model used in the analyses considered individual variability over time and baseline performance for each participant, suggesting that differences in practice effects were not due to individual variation. Rather, evidence herein suggests that the use of alternate versions of the ECAS is successful in reducing practice effects present in the repeated administration of the ECAS-A.

Cut-off scores, based on 2 standard deviations (SD) below the mean, for abnormality have previously been reported for the ECAS-A (14) and validated against a full neuropsychological battery (17). The present study demonstrated that the newly presented alternate versions are highly equivalent to the original ECAS. Examination of the cut-off scores for the alternate versions (ECAS-B and ECAS-C) demonstrate equality across all versions using two common methods (i.e. 2 standard deviations below the mean and the 95th percentile). For example, using a threshold of 2 SDs, the cut-offs for ALS-Specific functions is 77, 75, and 76 for versions A, B, and C, respectively. Similarly, cut-offs using the 95th percentile for ECAS Total scores are 105 for both ECAS-B and ECAS-C, where the published cut-off for ECAS-A is also 105. Given the lack of clinically meaningful differences between versions, the lack of observable practice effects, and similar cut-offs using two different methods, the cut-offs for the ECAS-A have been retained for the alternate versions and are displayed in Supplementary Table 2.

An additional goal of this study was to explore the inter-rater reliability of the ECAS and its alternate form. The administrations of the ECAS in this study were audio-recorded and scored by a second independent rater. Agreement for all cognitive domains and versions of the ECAS ranged between 0.930 and 0.998, providing evidence of exceptionally high agreement. While these findings are promising, one caveat here is that both raters had a background in psychology and were trained in the use of the ECAS by the scale’s authors. Care should be taken in inferring generalisability in rater agreement between health care professionals with different professional backgrounds (e.g. nurses, neurologists). However, the two raters in this study (CC and RR) were highly experienced in administering the ECAS resulting in an excellent level of agreement. This highlights the benefit of appropriate training in the standardization of assessment and, as such, training is recommended for all health professionals using the ECAS.

The findings of this study provide strong evidence that the alternate versions of the ECAS are equitable to the original ECAS and allow for the longitudinal monitoring of cognitive function in individuals with ALS. However, some further research is required to explore how the alternate versions function over time. In this study, the alternate forms were presented in a fixed order (A-B-C) and for practical purposes this order is therefore recommended. While no evidence of order effects was found herein, future research may explore order effects using randomized presentation. Furthermore, reliable measures of change are needed to determine what change in performance is over and above normal variation and constitutes a significant improvement or decline in function. Methods such as the Reliable Change Index or regression based methods will in the future allow for this.

Ceiling effects were observed in all three versions of the ECAS for the language and visuospatial domains. While ceiling effects are common in neuropsychological tests, they limit the certainty with which equivalency can be assumed. It would be beneficial to explore the relative practice effects of using the same versus different ECAS versions in an ALS sample whom are less likely to approach ceiling.

Finally, future research may explore the effect of different testing intervals on repeated assessment. Testing intervals of 4, 6, and 12 months may be common within research and clinical practice and the effect of interval length should be explored in relation to reliability statistics of the ECAS alternate versions.

In conclusion, our findings suggest that the ECAS-B and ECAS-C are demonstrably equivalent to the original ECAS and provide the opportunity to monitor the longitudinal cognitive and behavioural profile of people with ALS longitudinally while controlling for practice effects both clinically and in research settings. Therefore, the neuropsychological profile may be monitored over the course of the disease allowing clinicians to provide time-appropriate, accurate, and person-centred care services.

Declaration of interest

Orla Hardiman has received fees for consultation work from Biogen Idec, Cytokinetics and Novartis. She serves as Editor-in-Chief of Amyotrophic Lateral Sclerosis. Ammar Al-Chalabi has consulted for Biogen Idec, Cytokinetics Inc, OrionPharma, Mistubishi-Tanabe Pharma and Chronos Therapeutics. The remaining authors declare no conflict of interest.

Supplementary material available online

Acknowledgements

The authors would like to thank those who participated in this study.

Additional information

Funding

This work was supported by a grant from the Amyotrophic Lateral Sclerosis Association [ALSA; ID: 179]. The project is supported through the following funding organizations under the aegis of JPND - http://www.jpnd.eu (United Kingdom, Medical Research Council [MR/L501529/1], Economic and Social Research Council [ES/L008238/1], and Irish Health Research Board [HRB-JPND/2013/1]). CC receives support from the Euan MacDonald Centre for Motor Neurone Disease Research. AAC receives salary support from the National Institute for Health Research Maudsley Biomedical Research Centre.