https://orcid.org/0000-0002-4217-3941
Abstract
Background
ALS clinical care and research has changed dramatically since the COVID-19 pandemic, accelerating the need for cognitive assessments to be adapted for remote use.
Objectives
To develop the remote administration method of the Edinburgh Cognitive and Behavioural ALS Screen (ECAS), and determine its reliability and validity. Methods: The validation process consisted of: (1) Two versions of the ECAS (A and B) were administered, one in-person and one remotely via video call in a randomized order to 27 people without ALS; (2) The ECAS was administered remotely to 24 pwALS, with a second rater independently scoring performance; and (3) Acceptability was assessed by gathering feedback from 17 pwALS and 19 clinicians and researchers about their experience of using the ECAS remotely.
Results
In the group without ALS, the remote and in-person ECAS total scores were found to be equivalent, and a Bland-Altman plot showed good agreement between the two administration methods. In pwALS, there was excellent agreement between two raters (ICC = 0.99). Positive feedback was gained from pwALS, researchers and clinicians with regards to ease of process, convenience, time, and the environment.
Conclusions
These findings provide evidence of the reliability and validity of the remote administration of the ECAS for pwALS, with clinicians, researchers and pwALS viewing it as a good alternative to face-to-face administration.
Introduction
The COVID-19 pandemic created unprecedented challenges to ALS clinical care and research, leading to a reduction in face-to-face visits. This accelerated the need to adopt remote ways of conducting clinical assessments and research, which would also benefit those living in rural areas and those with travel and mobility issues. Given that MND Care Centers are regional and mobility is frequently affected by motor neurone disease (MND), these issues affect a large number of people with ALS (pwALS).
A fundamental component in the provision of appropriate care for pwALS is the assessment of cognition and behavior (Citation1). This is also increasingly being used as an outcome measure in clinical trials (Citation2). This is important given that up to 50% of patients develop cognitive and/or behavioral impairment with 15% suffering from frontotemporal dementia (FTD; Citation3). These percentages can be even higher in later stages of the disease (Citation4).
A well-established screening instrument for detecting the cognitive impairments specific to ALS is the Edinburgh Cognitive and Behavioral ALS Screen (ECAS; Citation5). It is a brief multi-domain assessment including ALS-specific domains (language, verbal fluency, and executive functions), and ALS nonspecific domains (memory and visuospatial functions). Higher total scores of the ECAS indicate higher levels of cognitive functioning. It also includes a separate caregiver/informant interview to detect behavioral abnormalities common in behavior variant FTD, based on diagnostic criteria (Citation6,Citation7). Several studies validating the administration of the ECAS in ALS have been published (Citation8,Citation9). However, to date, it has been validated for face-to-face administration only (Citation10).
Assessing patients’ cognition is most appropriately undertaken in-person for several factors, including the wider availability of validated tests which have normative data for face-to-face assessments, the multiple influences on cognitive functioning, and the complexity of presentation in people with cognitive difficulties often requiring face-to-face observation. In addition, the added cognitive demands that remote assessments can place on patients (e.g. environmental distractions or poor connection), as well as issues related to dysarthria, motor decline, co-morbid FTD, and advanced disease stages can make virtual assessments more challenging (Citation11). Despite these limitations, there are notable advantages to remote interviews. Such assessments eliminate the need for patients to travel, therefore they are less fatigued during the assessment. As well as being convenient, it can save patients both time and money. Patients no longer have to face the cognitive demands associated with planning and arranging a clinic visit, which can be particularly challenging for individuals with cognitive and physical difficulties. Additionally, the anxiety often experienced when trying to arrive on time for an appointment or being within a clinical setting is alleviated with remote assessments. Several studies have demonstrated good agreement between remote cognitive assessments and traditional in-person assessments, particularly for tasks that rely on verbal responses (Citation12). Research in patients with neurodegenerative diseases has shown promising results in terms of the reliability and validity of remote cognitive assessments (e.g. Citation13,Citation14). However, it should be noted that there are currently no cognitive assessments specifically tested and validated for remote administration via videoconferencing in ALS, highlighting the need for further research in this area.
The aims of the study were to develop and validate a remote method of administering the ECAS. Reliability and validity were explored using three approaches: (1) a comparison of remote vs original in-person ECAS administration; (2) inter-rater reliability for the remote administration; and (3) evaluating the experiences of pwALS, clinicians and researchers who have administered the ECAS remotely.
Methods
Documents and materials suitable for the remote administration of the ECAS via teleconferencing were developed. This included: guidance notes for the clinician/researcher on administration; an instruction sheet for the pwALS on how to set up their room and device (larger devices such as computers, laptops, and tablets were recommended); written response booklets (to be sent to participants beforehand if required); and visual stimuli for ECAS versions A-B-C which can be screen shared. To ensure participants could see the stimuli, two preliminary images were presented at the outset, and participants were asked to describe them. Note that this is for the cognition part of the ECAS. The behavioral component consists of an oral interview, and there should be no substantive difference in its administration, regardless of whether it is done in-person or remotely. We aimed to make the procedure of the remote version as similar to the in-person version as possible. Given that the original in-person ECAS was designed to accommodate speech and motor impairments, we maintained awareness of this in the development of the remote version. Thus, for tasks like verbal fluency, participants could respond by either speaking, writing, or by typing using the ‘chat’ feature of the teleconferencing platform.
Ethical approval was obtained from the National Health Service (NHS) Scotland A Research Ethics Committee and the Psychology Research Ethics Committee at the University of Edinburgh. Data were also collected as part of a registered service evaluation through Sheffield Teaching Hospitals NHS Foundations Trust. All participants gave their informed consent under the Declaration of Helsinki.
Remote vs in-person
Participants
Twenty-seven people without ALS were recruited. Participants were recruited via social media through the University of Edinburgh School of Psychology, Philosophy and Language Sciences marketing department. Inclusion criteria: aged between 18-80 years; fluent in English; and no major medical or neurological condition.
Procedure
Participants undertook two versions of the ECAS (A and B), one administered remotely on Microsoft Teams, and one administered in-person. ECAS versions and administration methods were randomly assigned and counterbalanced to prevent order effects. Alternative versions of the ECAS (versions A, B and C) have been found to be equivalent with no repetition or practice effects (Citation15). Assessments took place within a 2-week period.
Inter-rater reliability
Participants
Twenty-four pwALS were recruited through the CARE-MND platform – a Scotland wide register for pwMND to take part in research (Citation16). Potential participants who met the eligibility criteria were provided with information packs regarding the study then contacted the researcher if they wanted to participate. Participants were included if they: were aged between 18-80 years; were fluent in English; had clinical and electrophysiological evidence of combined upper and lower motor neurone involvement; and met the revised El Escorial criteria for possible, probable, or definite ALS (Citation9). Participants were excluded if they had any major comorbid medical or neurological disorder.
Procedure
Participants were administered the ECAS remotely via Microsoft Teams. All participants were able to carry out the assessment independently without any help, and responded by either speaking or typing into the ‘chat’ function. A second rater (author SA) independently scored the ECAS from transcribed video recordings.
Experiences
Participants
All clinicians and researchers who had been given access to the remote ECAS were invited by email to participate in a feedback survey and 19 took part. Seventeen pwALS who had been administered the ECAS remotely within the Sheffield MND Care Center & Clinical Neuropsychology Services were also invited to feedback. These 17 were separate from those described in B, the inter-rater reliability sub-study.
Procedure
Two online surveys were developed (one for clinicians and researchers and one for pwALS) and were administered to participants to gain feedback and explore their experiences of completing or administering the ECAS remotely. Participants were given a series of statements regarding the remote ECAS whereby they could respond using a 5-point Likert scale from strongly disagree to strongly agree. Statements were related to whether: the process was straightforward; the instructions were clear and understandable; it was convenient for them and saved time; they felt uncomfortable; completing the assessment with someone over video call was not as good as doing it in-person; and if they enjoyed it. Open questions were related to whether they experienced any technical problems and what they liked and disliked about the assessment.
Statistical analysis
IBM SPSS Statistical v27 was used for the statistical analysis. Distribution of the data was assessed using the Shapiro-Wilk test, which determined the use of parametric analyses. (A) Remote vs in person: Cronbach’s alpha was used to examine internal consistency reliability. To explore the equivalence of in-person and remote administration methods in people without ALS, a two one-sided t-test (TOST) was used. A Bland-Altman plot was also used to assess the agreement between the two measures. To explore order effects, a repeated measures t-test was used. (B) Reliability: Cronbach’s alpha was used to examine internal consistency reliability. Intra-class correlation (ICC) was used to determine inter-rater reliability of administering the remote ECAS to pwALS, with two raters. (C) Experiences: To explore experiences of pwALS, clinicians and researchers who had completed or administered the ECAS, descriptive statistics of quantitative survey data and a thematic analysis of qualitative data was completed.
Results
Remote vs in-person
Participant characteristics for the group without ALS are shown in .
Table 1. Participant characteristics of participants with and without ALS.
The Cronbach’s alpha of the remote ECAS in this group (N = 27) was found to be 0.57, based on the 5 subscales of the ECAS (language, verbal fluency, executive, memory, and visuospatial). Higher values (max 1) indicate greater internal consistency reliability.
The TOST procedure indicated that the observed effect size (dz = 0.23) was significantly within the equivalent bounds of −9 and 9 scale points, t(26) = 6.64, p = 0.00 (see ). A change of ≥9 for the ECAS total score is required for reliable change (Citation17). There was not a significant difference between session 1 and 2 ECAS scores, suggesting no order effect, t(26) = −1.61, p = 0.119.
Figure 1. ECAS total scores (remote and in-person) for the group of people without ALS. X indicates the mean total scores (116.2 for remote and 114.4 for in-person). Standard errors were 1.6 and 1.8 respectively. Whiskers indicate range and the dot is an outlier.
The Bland-Altman plot (see ) shows that there is good agreement between the two methods of administration and that they are largely interchangeable, producing equivalent scores.
Figure 2. Bland-Altman plot. The x-axis of the plot represents the average of the two measurements, while the y-axis represents the difference between the two measurements. The dark horizontal line is drawn at the mean difference (1.74) to indicate the bias between the two methods. The grey lines are the upper and lower limits of agreement which represent the range within which 95% of the differences between the two methods of measurement are expected to fall.
Inter-rater reliability
Participant characteristics for pwALS are shown in .
The Cronbach’s alpha of the remote ECAS in this group (N = 24) was found to be 0.78, indicating good internal consistency. This was based on the 5 subscales of the ECAS (language, verbal fluency, executive, memory, visuospatial).
Two raters (authors DG and SA) independently scored the remote ECAS total scores in pwALS. The ICC was 0.99, showing excellent agreement between the two raters.
Experiences
Nearly a third of participants were Clinical Psychologists, over a quarter were PhD students, and the remainder were nurses or assistants within psychology or research. Most respondents (94.7%) were working in the United Kingdom (UK) and Ireland. Participants had completed an average of 13.3 (SD = 18.1) remote ECAS assessments, taking an average of 31.6 minutes (SD = 9.6) to complete.
For the clinicians and researchers, all participants (100%) agreed that the administration of the remote ECAS was straightforward and that the guidelines were clear and understandable (see ). Nearly all participants (94.7%) agreed that administering the ECAS remotely was convenient for them and saved them time. Over three quarters (78.9%) disagreed that they felt uncomfortable during the administration process. Nearly one third (31.6%) agreed that doing the ECAS remotely is not as good as doing it in person, 26.3% disagreed, and 42.1% neither agreed nor disagreed. Over three quarters of participants (84.2%) agreed that administering the ECAS remotely is an appropriate way of assessing cognition and 89.5% felt satisfied administering the ECAS in this way.
Figure 3. Feedback from clinicians and researchers on their experiences of administering the ECAS remotely.
For the group of pwALS, the majority of respondents agreed that the remote ECAS was straightforward (88.2%) and that the instructions were clear and understandable (88.3%), however a small proportion (5.9% respectively) disagreed with this (see ). Every respondent (100%) agreed that doing the ECAS remotely was convenient for them and saved them time. Nearly three quarters (70.6%) felt comfortable during the assessment process, whilst just under a quarter (23.5%) did not feel comfortable. No additonal information was given for why this was the case. When questioned if performing the remote ECAS was not as good as performing it in-person, the results were varied (35.3% strongly disagreed, 23.6% neither agreed nor disagreed, and 41.2% somewhat agreed); however over 76.5% agreed that they enjoyed completing the ECAS remotely.
Figure 4. Feedback regarding the remote administration of the ECAS from pwALS.
When presented with the statement, “Overall, I was satisfied with the cognitive screening process”, 70.6% of respondents strongly agreed with this. With regards to any technical issues experienced, 70.6% of pwALS did not encounter any.
All participants were given the opportunity to provide open-ended responses to questions regarding what they liked and disliked about doing the ECAS remotely. While the qualitative data set was small, a thematic analysis, conducted independently for both groups (pwALS and researchers/clinicians), revealed consistent themes and therefore, are presented in a unified manner. The participant responses revealed that the remote ECAS has an impact on pwALS, researchers and clinicians in several ways, which can be categorized under the following overarching themes:
Ease of process
Clinicians and researchers administering the assessment found that it was ‘easy to do, easy to complete’ (Advanced Nurse Specialist in MND), whilst those completing the assessment found that ‘every exercise was clear and precise’ (pwALS 1).
Convenience
Administering the ECAS remotely was found to be ‘convenient for many people living with MND and their family members, it is often less tiring for them’ (Clinical Psychologist), with many pwALS agreeing that ‘it was very convenient’.
Time
Remote assessment saved both pwALS and clinicians/researchers significant amounts of time. One participant stated that they ‘didn’t have to travel as it’s a 3 hour journey one way’ (pwALS 2). Whilst it was noted by clinicians that ‘there are fewer cancellations…it saves me a lot of time’ (Clinical Psychologist).
Environment
Clinicians and researchers thought that it was useful for patients to complete screening in their home environment. Given that pwALS were able to complete the assessment in the comfort of their own home, it made them ‘feel more relaxed’ (pwALS) during the assessment.
Whilst the majority of responses were positive, and many responding with “nothing” when questioned what they disliked about the remote ECAS, one respondent mentioned that they ‘felt embarrassed’ whilst another felt ‘detached from real human interaction’.
Discussion
We have provided evidence of the reliability and construct validity of the remote administration version of the ECAS for assessing cognition. There were no significant differences in performance scores between the remote and original in-person versions of the ECAS in participants without ALS, and good agreement was found between the two measures. This was as expected as the materials and procedure were designed to be as close to the original in-person administration as possible. It is psychometrically robust with good internal consistency and excellent inter-rater reliability. Clinicians, researchers and pwALS also provided positive feedback regarding their experience of completing the remote ECAS.
The feedback indicated that the remote ECAS is an acceptable method for assessing cognition and has additional benefits compared to face-to-face administration. Clinicians and researchers agreed that the process was straightforward, convenient, saved time, and that the administration guidelines were clear and understandable. Similarly, for pwALS, over three quarters enjoyed completing the ECAS remotely and agreed that it was straightforward, convenient, saved time, and that the instructions were clear and understandable. Given the relative ease and lack of negative feedback, it may be more likely that people agree to repeat the test later on in their disease course to monitor progression.
The thematic analysis further emphasized the positive feedback gained from the surveys and suggested that the remote administration of the ECAS could improve access to cognitive assessments for pwALS. pwALS reported that they did not have to travel to the clinic, it was convenient for them, saved time, and were less tired whilst completing the assessment. Not only was this method of administration useful during the COVID-19 pandemic when travel restrictions were in place, it is useful for patients who live in more remote areas and cannot travel to clinic for any number of reasons. Given that pwALS were able to complete the assessment in a familiar environment (i.e., within the comfort of their own home), clinicians and researchers reported that patients appeared more relaxed. These benefits are similar to those found by Geronimo et al. (Citation18) when they assessed the feasibility and acceptability of telehealth for ALS care. Given that anxiety and fatigue is common in pwALS (Citation19), reducing any sort of stress-related performance may provide a more accurate reflection of peoples cognitive abilities. The remote administration of the ECAS could therefore improve access to cognitive assessments for pwALS.
The ECAS has been validated in other disorders such as Parkinson’s disease, Progressive Supranuclear Palsy and Huntington’s disease (Citation20), and the remote administration method may be useful for assessment of these populations. The ECAS has also been validated in behavioral variant FTD and Alzheimer’s disease (Citation21,Citation22), and we found no issues with our remote procedure in our one ALS participant who also had FTD. However, the procedure may need to be explored with other types of dementias where people may struggle with some technical equipment, although this may be helped by support from a carer/family member.
This study is not without its limitations. The COVID-19 restrictions prevented us from conducting in-person assessments for pwALS, although they were still included in the interrater reliability and feedback components of the study. It is therefore important that future research seeks to further validate the findings in an ALS patient cohort. The recruitment of these participants through the CARE-MND register, wherein individuals had already expressed a willingness to be contacted for research purposes, may have introduced selection bias. This approach potentially led to an underrepresentation of individuals with cognitive impairment, who might have been less likely to register or respond to research invitations, thereby affecting the overall representativeness of our sample. The suitability and effectiveness of remote ECAS administration with people with cognitive impairment remains to be demonstrated. Hence, further investigations should be conducted to explore the feasibility and acceptability of remote administration specifically in pwALS who have cognitive impairment. Validating the process with young and healthy participants, who differed significantly in age from pwALS, presents a potential limitation as age-related factors may affect performance. This difference in terms of ages raises concerns about the applicability of our findings to older individuals who may have less familiarity with technology and online interactions, potentially leading to different experiences and preferences regarding remote administration. Consequently, further research is needed to evaluate in-person vs remote administration within an older population. The high level of education observed among participants may limit the generalizability of our findings, potentially influencing proficiency in navigating the remote administration version of the ECAS. While future research should aim to include a more educationally heterogenous participant pool, disentangling the impact of education level on ECAS scores presents a nuanced challenge for both remote and in-person administrations of the assessment (Citation23).
In conclusion, we have provided evidence of the reliability and validity of the remote version of the ECAS. Clinicians, researchers and pwALS view it as a good alternative to face-to-face administration. The method may be useful for clinical use and research, in addition to clinical trials.
Acknowledgements
We would like to thank the participants for taking part.
Declaration of interests
Author SA is a co-author of the ECAS and is an Associate Editor with the journal.
Additional information
Funding
References
- NICE. Motor neurone disease: assessment and management 2016. Available from: https://www.nice.org.uk/guidance/ng42.
- Beswick E, Forbes D, Hassan Z, Wong C, Newton J, Carson A, et al. A systematic review of non-motor symptom evaluation in clinical trials for amyotrophic lateral sclerosis. J Neurol. 2022;269:411–26.
- Goldstein LH, Abrahams S. Changes in cognition and behaviour in amyotrophic lateral sclerosis: nature of impairment and implications for assessment. Lancet Neurol. 2013;12:368–80.
- Crockford C, Newton J, Lonergan K, Chiwera T, Booth T, Chandran S, et al. ALS-specific cognitive and behavior changes associated with advancing disease stage in ALS. Neurology 2018;91:e1370–e1380.
- Abrahams S, Newton J, Niven E, Foley J, Bak TH. Screening for cognition and behaviour changes in ALS. Amyotroph Lateral Scler Frontotemporal Degener. 2014;15:9–14.
- Strong MJ, Abrahams S, Goldstein LH, Woolley S, McLaughlin P, Snowden J, et al. Amyotrophic lateral sclerosis – frontotemporal spectrum disorder (ALS-FTSD): revised diagnostic criteria. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18:153–74.
- Rascovsky K, Hodges JR, Knopman D, Mendez MF, Kramer JH, Neuhaus J, et al. Sensitivity of revised diagnostic criteria for the behavioural variant of frontotemporal dementia. Brain. 2011;134:2456–77.
- Pinto-Grau M, Burke T, Lonergan K, McHugh C, Mays I, Madden C, et al. Screening for cognitive dysfunction in ALS: validation of the Edinburgh Cognitive and Behavioural ALS Screen (ECAS) using age and education adjusted normative data. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18:99–106.
- Lulé D, Burkhardt C, Abdulla S, Böhm S, Kollewe K, Uttner I, et al. The Edinburgh Cognitive and Behavioural Amyotrophic Lateral Sclerosis Screen: a cross-sectional comparison of established screening tools in a German-Swiss population. Amyotroph Lateral Scler Frontotemporal Degener. 2015;16:16–23.
- Niven E, Newton J, Foley J, Colville S, Swingler R, Chandran S, et al. Validation of the Edinburgh Cognitive and Behavioural Amyotrophic Lateral Sclerosis Screen (ECAS): a cognitive tool for motor disorders. Amyotroph Lateral Scler Frontotemporal Degener. 2015;16:172–9.
- Fox-Fuller JT, Rizer S, Andersen SL, Sunderaraman P. Survey findings about the experiences, challenges, and practical advice/solutions regarding teleneuropsychological assessment in adults. Arch Clin Neuropsychol. 2022;37:274–91.
- Brearly TW, Shura RD, Martindale SL, Lazowski RA, Luxton DD, Shenal BV, et al. Neuropsychological test administration by videoconference: a systematic review and meta-analysis. Neuropsychol Rev. 2017;27:174–86.
- Requena-Komuro M-C, Jiang J, Dobson L, Benhamou E, Russell L, Bond RL, et al. Remote versus face-to-face neuropsychological testing for dementia research: a comparative study in people with Alzheimer’s disease, frontotemporal dementia and healthy older individuals. BMJ Open. 2022;12:e064576.
- Yoshida K, Yamaoka Y, Eguchi Y, Sato D, Iiboshi K, Kishimoto M, et al. Remote neuropsychological assessment of elderly Japanese population using the Alzheimer’s Disease Assessment Scale: a validation study. J Telemed Telecare. 2020;26:482–7.
- Crockford CJ, Kleynhans M, Wilton E, Radakovic R, Newton J, Niven EH, et al. ECAS ABC: alternate forms of the Edinburgh cognitive and behavioural ALS screen. Amyotroph Lateral Scler Frontotemporal Degener. 2018;19:57–64.
- Leighton D, Newton J, Colville S, Bethell A, Craig G, Cunningham L, et al. Clinical audit research and evaluation of motor neuron disease (CARE-MND): a national electronic platform for prospective, longitudinal monitoring of MND in Scotland. Amyotroph Lateral Scler Frontotemporal Degener. 2019;20:242–50.
- Crockford C, Newton J, Lonergan K, Madden C, Mays I, O'Sullivan M, et al. Measuring reliable change in cognition using the Edinburgh Cognitive and Behavioural ALS Screen (ECAS). Amyotroph Lateral Scler Frontotemporal Degener. 2018;19:65–73.
- Geronimo A, Wright C, Morris A, Walsh S, Snyder B, Simmons Z. Incorporation of telehealth into a multidisciplinary ALS Clinic: feasibility and acceptability. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18:555–61.
- Sandstedt P, Littorin S, Johansson S, Gottberg K, Ytterberg C, Kierkegaard M. Disability and contextual factors in patients with amyotrophic lateral sclerosis-a three-year observational study. J Neuromuscul Dis. 2018;5:439–49.
- Foley JA, Niven EH, Paget A, Bhatia KP, Farmer SF, Jarman PR, et al. Sensitivity and specificity of the ECAS in Parkinson’s disease and progressive supranuclear palsy. Parkinson’s Disease. 2018;2018:1–8.
- De Icaza Valenzuela MM, Bak TH, Thompson HE, Colville S, Pal S, Abrahams S. Validation of the Edinburgh cognitive and behavioural ALS screen (ECAS) in behavioural variant frontotemporal dementia and Alzheimer’s disease. Int J Geriatr Psychiatry. 2021;36:1576–87.
- Carelli L, Solca F, Migliore S, Torre S, Brugnera A, Mancini F, et al. Compensating for verbal-motor deficits in neuropsychological assessment in movement disorders: sensitivity and specificity of the ECAS in Parkinson’s and Huntington’s diseases. Neurol Sci. 2021;42:4997–5006.
- De Icaza Valenzuela MM, Bak TH, Pal S, Abrahams S. The Edinburgh Cognitive and Behavioral ALS screen: relationship to age, education, IQ and the Addenbrooke’s Cognitive Examination-III. Amyotroph Lateral Scler Frontotemporal Degener. 2018;19:585–90.