Sensitivity and specificity of the ECAS in identifying executive function and social cognition deficits in MND

Abstract

Objective: Motor neurone disease [MND] encompasses broad cognitive impairments, which are not fully captured by most screening tools. This study evaluated the specificity and sensitivity of the Edinburgh Cognitive and Behavioral ALS Screen [ECAS] in detecting impairments in executive function and social cognition. Methods: Participants (MND = 64; Healthy Controls = 45) completed the ECAS and standard neuropsychology tests of executive function and social cognition. Sensitivity and specificity of the ECAS were assessed at three levels (ALS-Specific score, executive function domain score, individual subtests: social cognition, inhibition, working memory, alternation). Results: MND patients were impaired on standard social cognition, initiation, visuomotor alternation, and verbal learning tests but not on inhibition or working memory tests, relative to controls. ECAS results revealed that the ALS-Specific score was high in specificity but low-to-moderately sensitive in identifying social cognition, inhibition, and working memory deficits, and that both sensitivity and specificity were high for identifying alternation deficits. The ECAS executive function domain score was high in specificity but poor in sensitivity for all four executive function domain subtests. The individual ECAS subtests were highly specific with good sensitivity, but the social cognition subtest lacked sensitivity. Conclusions: Impairments in social cognition may go undetected when using the ECAS as a screening tool. Thus, social cognition may need to be considered as a standalone component, distinct from the other executive functions. In addition, the test itself may need to be adjusted to encompass other aspects of social cognition that are affected in MND.

Key messages

• Cognitive screening tools are key to detect cognitive changes in MND, with the domains most affected being executive functions, language, and social cognition.

• The ECAS measure, developed for MND, has good specificity but lacks sensitivity to impairments in social cognition.

• Clinical implications are that cognitive impairments in social cognition may not be identified in MND patients by the ECAS.

• Adjustment to the ECAS cognitive screening tool widely-used in MND is suggested.

Key words:

• Motor neurone disease
• social cognition
• inhibition
• screening tool

A significant number of patients with motor neurone disease (MND) experience cognitive and behavioral changes as well as motor changes (1). Cognitive impairments in MND extend beyond executive functions to language and social cognition, and general screening tools do not accurately capture the full spectrum of dysfunction. For instance, the Mini-Mental State Examination (2) and the Addenbrooke’s Cognitive Examination (3) do not adequately assess executive functions (4,5). The ALS Cognitive Behavioral Screen (6), Frontal Assessment Battery (7) and Dementia Rating Scale (8) exclude useful domains, such as language and social cognition (9). Moreover, most of these screening tools, including the Montreal Cognitive Assessment (10), are not well-adjusted to motor impairments, thus hindering performance or erroneously amplifying cognitive impairment scores (11–13).

To overcome these limitations, Abrahams and colleagues (14) devised a clinical screening tool specifically for amyotrophic lateral sclerosis (ALS), the most common form of MND. The Edinburgh Cognitive and Behavioral ALS Screen (ECAS) assesses executive functions, word fluency, language, memory, and visuospatial functions, while accounting for speech and motor limitations (14). The current study examines which aspects of executive function are measured by the ECAS, and how effective it is in doing so.

Executive functions are a set of top-down cognitive processes that enable goal-directed behavior, including inhibition, working memory, cognitive flexibility, reasoning and problem-solving (15,16). The ECAS assesses executive functions using four tests (reverse digit span, alternation, sentence completion, and social cognition – Part A and B) to tap working memory, flexibility, inhibition, and social cognition.

The ECAS includes social cognition under the executive function domain. However, social cognition itself comprises interrelated components that are associated with distinct brain regions, including: theory of mind, the ability to perceive another person’s mental state distinctly from one’s own; affective empathy, one’s emotional response to another person’s situation; social perception, the ability to interpret and respond to basic social and emotional cues; and social behavior (17).

Using the ECAS total score, cognitive deficits were detected in nearly one-third of ALS patients (14). Subsequent validation studies of the ECAS in different languages and cultures reported cognitive impairments in 22%–37% of patients (German-Swiss (11); Chinese (18); Italian (12); Greek (19)). The ECAS total score comprises an ALS-Specific and a non-ALS-specific score, which reflect cognitive domains characteristically affected or not typically affected in ALS, respectively. Approximately 21% to 43% of patients are impaired on the ALS-specific score. However, only 6%–21% of patients were impaired on the non-ALS-specific score (11,12,14,18).

Validity of the ECAS as a screening tool for MND has been assessed by calculating sensitivity and specificity values. Sensitivity is the proportion of patients that are truly impaired, who score as impaired on the ECAS (i.e. true positive). Specificity is the proportion of patients that are truly unimpaired, who score as unimpaired on the ECAS (i.e. true negative). Ideally, screening tools should place more emphasis on sensitivity than specificity, as false positives can be detected upon further follow-up tests.

The ECAS has been validated against “gold standard” neuropsychological assessment (20,21). These studies reveal relatively high specificity for the ECAS in detecting cognitive impairment, with the total and ALS-specific scores yielding 80%–85% specificity. The executive function domain was 82%–85% specific. An assessment of the German-Swiss ECAS against comparative cognitive domain measurements (22) yielded 100% specificity (11). The social cognition subtest was discretely assessed against the Reading the Mind in the Eyes Test (23) and the Judgment of Preference task (24), which all involve gaze interpretation, and yielded 90%–100% specificity (20,21). Only one study has assessed the sentence completion task independently, against a Stroop task, and found 76% specificity (21). Thus, specificity of the ECAS total and domains is consistently high.

In contrast, there have been mixed findings regarding the sensitivity of the ECAS. While both studies reported relatively high sensitivity (85%–100%) for the total and ALS-specific scores, the sensitivity of individual tests and domains was not consistently high. For the executive function domain, sensitivity rates reported range from 57% to 100% (20,21) where the German-Swiss version yielded 43% sensitivity (11). When assessed against comparable neuropsychological tests, the social cognition subtest only yielded 25%–38% sensitivity (20,21). However, 100% sensitivity was found for the sentence completion task (21). Thus, sensitivity of the ECAS executive function domain is equivocal, which may indicate that patients who are truly impaired are not being detected by the ECAS.

In summary, these validation studies suggest that the ECAS may lack sensitivity in picking up social cognitive and executive function deficits. One of the main purposes of developing the ECAS was to address the newfound area of social cognitive impairment in MND. The ECAS has shown good sensitivity and specificity in discriminating social cognition performance between Parkinson’s disease, progressive supranuclear palsy and healthy controls (25). This evokes the question of which social cognition component the test is measuring, and whether that component aligns with the impaired component in MND.

ALS is thought to be on the same clinical spectrum as behavioral variant frontotemporal dementia (bvFTD), thus explaining shared characteristics including social cognitive impairments (1). The Social Cognition and Emotional Assessment (SEA), and shortened Mini-SEA, assess orbital and ventromedial prefrontal functions typically affected in FTD (26,27). The Mini-SEA includes the faux-pas test and emotions recognition test, which assess theory of mind and social perception, respectively, two of the most frequently used parameters of social cognition (17). Links have been identified between the ventromedial prefrontal cortex and social reasoning (including faux pas detection); amygdala and social judgment of faces (including eye gaze direction); and somatosensory cortices and empathy (28). The ECAS social cognition subtest could be validated against the Mini-SEA, to examine whether it taps social cognitive components relevant to FTD/MND (i.e. theory of mind, social perception).

In addition, we also aimed to assess the inhibitory sentence completion task from the executive domain against the Hayling Sentence Completion Test [HSCT] (29) and the Stroop Test (21). Recent evidence suggests that the inhibitory component of the HSCT and the Stroop are distinct and separable processes (30). The HSCT comprises a semantic context and requires participants to inhibit a dominant response and potentially use strategies to generate an alternative response (31,32) whereas the Stroop requires modulation of response conflict and speed-accuracy tradeoffs (33,34). We also aimed to assess the other two subtests from the executive function domain (working memory, alternation) against Digit Span and Trail Making Test B.

Methods

64 MND patients (M_age = 59.64, SD = 10.01) and 45 healthy controls (M_age = 60.07, SD = 8.68), matched for age, education and premorbid intelligence (IQ) (p > 0.05), were recruited from the Royal Brisbane and Women’s Hospital (RBWH) MND Clinic and The University of Queensland (UQ) as part of a larger neuropsychological study. Approval for the study was obtained from the RBWH and UQ Human Research Ethics Committees. All participants provided informed written consent.

Consistent with the literature, our MND cohort had more male than female participants (N_male = 50, N_female = 14), with 45 patients having limb-onset disease and 19 with bulbar-onset disease. Forty-seven MND patients had a diagnosis of ALS, seven patients had a diagnosis of predominate upper motor neuron disease (UMN), three patients had a diagnosis of predominate lower motor neuron disease (LMN), and seven were not yet classified. The average time between disease onset and neuropsychological assessment was 19.75 months (SD = 14.00, range = 5 to 96 months).

Cognitive impairment group analysis

We first compared patients’ performance on neuropsychological assessments with controls, in order to establish cognitive impairment in the MND cohort. Standard tests measured memory (Rey Auditory Verbal Learning Test [RAVLT]; Topographical Recognition Memory Test), attention and working memory (Digit Span), language (SydBat Naming), executive function (HSCT, Stroop Test¹, Trail Making Test B), visuomotor speed (Trail Making Test A), and social cognition (Mini-SEA) (see Table 1). In addition to the four HSCT clinical scaled scores (Initiation RT, Suppression RT, Suppression Errors, HSCT Overall Scaled Score), we calculated response times (RT), number correct, and errors. The Mini-SEA consists of the emotion recognition task and faux pas stories, which were combined to generate a composite score, (26,27). Tasks on the Mini-SEA are untimed, with visual aids provided to alleviate working memory demands. Impairment was >2SD below the mean, based on published normative data unless stated otherwise.

Table 1 Descriptive characteristics, neuropsychology tests, and the Hayling and Mini-SEA test scores for healthy controls and patient groups: mean and standard deviation.

	Controls (n = 45)	MND (n = 64)
	M (SD)	M (SD)	% <2 SDs
Sex (M:F)	24:20	50:14	–
Age	60.07 (8.68)	59.64 (10.01)	–
Education	13.29 (3.22)	12.35 (2.80)	–
Premorbid IQ^a	106.63 (10.28)	104.89 (12.27)	–
RAVLT^b T1-5	50.04 (9.94)	42.96 (9.14)**	6%
RAVLT^b T6	9.65 (3.30)	8.51 (3.17)	8%
RAVLT^b T7	9.70 (3.56)	7.98 (3.50)	8%
RAVLT^b recognition	13.26 (1.54)	12.38 (2.43)	18%
RMT topography^c	25.15 (2.79)	24.98 (3.38)	0%
SydBat Naming^g	27.32 (2.15)	26.14 (2.57)	9%
Digit Span Forwards^d	11.78 (2.62)	10.83 (2.11)	2%
Digit Span Backwards^d	8.56 (2.31)	8.35 (2.21)	3%
Trail Making Test A^e	26.37 (8.81)	30.29 (9.99)	0%
Trail Making Test B^e	60.15 (24.74)	75.98 (34.27)*	8%
Stroop color naming^f	234.14 (49.03)	212.63 (36.38)	0%
Stroop interference^f	106.50 (19.26)	103.09 (23.20)	0%
Stroop errors^f	1.20 (1.67)	1.79 (1.69)	9%
Stroop difference^f	130.50 (43.34)	109.39 (38.29)	0%
Verbal fluency ‘F’^h	13.55 (5.02)	11.12 (4.06)	5%
Hayling Test^i			–
Scaled Score#			–
Overall	5.37 (1.57)	5.02 (1.82)	8%
Initiation RT	5.67 (1.04)	5.39 (1.03)*	0%
Inhibition RT	5.42 (1.24)	5.25 (1.39)	5%
Inhibition Errors	5.67 (2.02)	5.27 (2.22)	8%
Response Time (seconds)			–
Initiation	8.54 (13.06)	10.57 (11.38)*	–
Inhibition	43.61 (41.57)	47.27 (40.21)	–
Inhibition – Initiation	35.07 (34.18)	35.70 (36.60)	–
Response categories			–
Initiation correct/15	14.75 (.51)	14.70 (0.56)	–
Inhibition			–
A Errors	1.02 (1.21)	1.10 (1.35)	–
B Errors	3.00 (2.26)	3.34 (2.64)	–
Total correct/15	10.88 (3.12)	10.51 (3.44)	–
Correct w strategy	4.66 (3.44)	6.07 (4.46)	–
Mini-SEA^j			–
Emotion recognition/35	29.73 (2.64)	27.24 (3.16)**	13%
Faux pas stories/40	36.26 (2.49)	32.46 (6.05)*	25%
Total/30	26.14 (1.21)	23.78 (3.32)**	29%
ECAS	–	n = 64	%<2 SDs
Overall/136	–	110.64 (10.69)	28%
ALS-specific/100	–	83.31 (8.63)	23%
Language/28	–	26.56 (1.63)	34%
Naming/8	–	7.64 (0.65)	–
Comprehension/8	–	7.98 (0.13)	–
Spelling/12	–	10.94 (1.55)	–
Verbal fluency/24	–	16.25 (5.24)	27%
S/12	–	8.19 (2.51)	–
T/12	–	8.03 (3.50)	–
Executive (48)	–	40.66 (4.17)	8%
Sentence completion/12	–	10.59 (1.70)	–
Social cognition/12	–	11.58 (1.59)	–
Digit span/12	–	7.22 (2.02)	–
Alternation/12	–	11.27 (1.76)	–
ALS nonspecific/36	–	27.17 (3.84)	25%
Memory/24	–	15.20 (3.95)	36%
Immediate recall/10	–	5.31 (1.70)	–
Delayed recall/10	–	8.08 (2.13)	–
Delayed recognition/4	–	1.81 (1.47)	–
Visuospatial/12	–	11.84 (0.41)	2%
Dot/4	–	3.94 (0.24)	–
Cube/4	–	3.94 (0.24)	–
Number/4	–	3.97 (0.18)	–

Cutoffs for Stroop based on healthy mean and standard deviation; ∼ Scaled Score is 1–10, 6 is average; * = p < 0.05, ** = p < 0.01, compared to Healthy Controls (HC); Not all participants completed all tests (N: HC/MND): ^aNelson and Willison (35) (N: 40/35); ^bIvnik et al. (43) (N: 23/53); ^cWarrington (44) (N: 41/59); ^dWechsler (45) (N: 41/60); ^eArmy Individual Test Battery (46) (N: 23/56); ^fTrenerry et al. (47) (N: 21/24); ^gSavage et al. (48) (N: 22/57); ^hBenton et al. (49) (N: 22/60); ⁱBurgess and Shallice (29) (N: 45/59); ^jBertoux et al. (27) (N: 22/38).

Independent-samples t-tests were used to analyze outcome variables. Mann–Whitney U tests were used for variables that were not normally distributed. Only significant results are reported. Kendall’s Tau correlations were also performed to investigate convergent validity of ECAS subtests with standard neuropsychology tests.

ECAS specificity and sensitivity analysis

We investigated the specificity and sensitivity of the ECAS ALS-Specific score, executive function subscore, and the four executive function subtest scores compared to standard neuropsychology tests (ECAS Social Cognition with Mini-Sea; ECAS Sentence Completion with HSCT Suppression and Stroop Interference; ECAS Digit Span and Alternation with WAIS-III Digit Span Backwards Scaled Score and Trail Making Test B response time).

Specificity was calculated as [True Negatives]/[True Negatives + False Positives] * 100 (36); where a true negative is a patient who is intact on the neuropsychological assessment and scored above the clinical cutoff on the ECAS, and a false positive is an intact patient who scored below the cutoff. Sensitivity was calculated as [True Positives]/[True Positives + False Negatives] * 100 (36); where a true positive is a patient who is impaired on the neuropsychological assessment and scored below the cutoff on the ECAS, and a false negative is an impaired patient who scored above the cutoff. The cutoff scores we used were: 105/136 for ECAS overall score, 77/100 for ALS-specific score, 24/36 for non-ALS specific score (14). The ECAS executive function domain comprises four tests, each worth 12 points. As the ECAS does not evaluate clinical impairment at the level of each subtest, we set the impairment cutoff at below a full score on the Social Cognition and Alternation subtests (i.e. ≤11). We compared three different cutoffs for Sentence Completion (i.e. 9, 10, and 11) and two cutoffs for Alternation (i.e. 10 and 11). For ECAS Digit Span, a cutoff for impairment was set at ≤ 6 to best reflect WAIS-III Digit Span norms, and we compared three possible cutoff scores (i.e. 4, 5, and 6).

Results

Cognitive impairment in MND

A summary of the results is presented in Table 1. On neuropsychological tests, MND patients performed poorer than controls for verbal learning (i.e. RAVLT T1-5; U = 858.00, p = 0.005) and visuomotor alternation (i.e. Trail Making Test B: U = 274.00, p = 0.021), but equivalent on other measures of executive function (Stroop Test), visual memory and visuomotor speed. On the HSCT, MND patients scored lower on the Initiation RT SS, compared with healthy controls (U = 1596.50, p < 0.05), but not on the other scaled scores. With respect to the total response time, MND patients were slower to complete the Initiation Section 1, compared with controls (U = 901.50, p < 0.05). There were no group differences in the Suppression-Initiation RT difference, nor on error or correct response category analyses. Apart from verbal recognition memory, <10% of MND patients were impaired on standard neuropsychology tests including executive function tasks. MND patients performed worse than controls across all Mini-SEA measures: emotion recognition (U = 616.50, p < 0.01), faux pas stories (U = 371.50, p < 0.05), and total combined score (U = 398.00, p < 0.01). Thirteen percent of MND patients scored as impaired on emotion recognition, 25% scored as impaired on faux pas stories, and 29% scored as impaired on the total combined score.

Convergent validity of the ECAS

Most ECAS subtests were significantly correlated with comparable standard neuropsychology tests and therefore demonstrated convergent validity (see Table 2). Exceptions were ECAS Comprehension (due to ceiling effects), Immediate Recall, Delayed Recognition, Dot Counting, and Number Location.

Table 2 Correlations between ECAS subtests and standard neuropsychology tests.

ECAS subtest	Standard neuropsychology test	N	τ	p
Language
Naming/8	Sydbat naming/30	57	0.40	<0.001
Comprehension/8	Sydbat comprehension/30	55	n/a^	n/a^
Spelling/12	NART IQ	35	0.41	0.002
Fluency
S/12	F wpm	60	0.45	<0.001
T/12	F wpm	60	0.39	<0.001
Executive
Sentence completion/12	Hayling inhibition correct/15	59	0.51	<0.001
	Hayling inhibition Err SS	59	0.50	<0.001
	Hayling inhibition RT	59	−0.42	<0.001
	Stroop interference	23	0.26	0.113
Social cognition/12	Mini-SEA emotion recognition/35	38	0.34	0.014
	Mini-SEA faux pas stories/40	28	0.02	0.922
	Mini-SEA total/30	28	0.14	0.360
Digit Span/12	WAIS III Digit Span Backwards /14	60	0.74	<0.001
	TMT B RT	37	−0.22	0.079
Alternation/12	TMT B RT	37	−0.18	0.163
	WAIS III Digit Span Backwards /14	60	0.22	0.045
Memory
Immediate recall/10	RAVLT T1-5	53	0.16	0.125
	RAVLT T6/15	53	0.14	0.179
Delayed recall/10	RAVLT T7/15	53	0.36	<0.001
Delayed recognition/4	RAVLT Recognition /15	50	−0.03	0.800
Visuospatial
Dot/4	Judgment of Line Orientation	33	−0.18	0.250
Cube/4	Judgment of Line Orientation	33	−0.31	0.049
Number/4	Judgment of Line Orientation	33	−0.18	0.250

Note. ^ - correlation not appropriate for comprehension due to ceiling effects on both tasks.

Specificity and sensitivity of the ECAS

The percentage of patients impaired on the ECAS were as follows: 28% on ECAS overall score, 23% on ALS-specific score, 25% on non-ALS specific. Only eight percent of patients were impaired on the ECAS executive function domain. Seven of 51 patients (14%) were impaired on the ECAS social cognition subtest. Table 3 provides a detailed breakdown of the specificity and sensitivity of: the ECAS ALS-Specific Score; the Executive Function Subscore; the Social Cognition subtest, in identifying social cognitive deficits as measured by the Mini-SEA; the Sentence Completion subtest in identifying inhibitory deficits as indexed by the HSCT Suppression Error and Stroop Interference measures; the Reverse Digit Span subtest in identifying working memory deficits as indexed by the WAIS Digit Span Backwards SS; and the Alternation subtest in identifying visual attention and task-switching deficits as measured by Trails Making Test B RT and WAIS Digit Span Backwards SS.

Table 3 Sensitivity and specificity of ECAS ALS-specific score, and executive function total and subtest scores in detecting impairments on standard neuropsychology assessments.

	ECAS ALS-specific		ECAS exec. functions		ECAS social cognition
Mini-SEA	Specificity	Sensitivity	Specificity	Sensitivity	Specificity			Sensitivity
Emotion recognition	85%	80%	91%	20%	91%			60%
Faux pas stories	95%	43%	90%	14%	86%			14%
Mini-SEA overall	95%	38%	90%	13%	90%			25%
	ECAS ALS-specific		ECAS exec. functions		ECAS sentence completion
Hayling	Specificity	Sensitivity	Specificity	Sensitivity	Cutoff	Specificity		Sensitivity
Inhibition Error SS	80%	38%	91%	8%	≤11	54%		85%
	–	–	–	–	≤10	76%		85%
	–	–	–	–	≤9	95%		69%
Stroop Interference	91%	–	96%	–	≤11	35%		–
	–	–	–	–	≤10	43%		–
	–	–	–	–	≤9	65%		–
	ECAS ALS-specific		ECAS exec. functions		ECAS digit span
	Specificity	Sensitivity	Specificity	Sensitivity	Cutoff		Specificity	Sensitivity
WAIS-III Digit Span Backwards SS	76%	50%	93%	50%	≤6		66%	100%
	–	–	–	–	≤5		85%	100%
	–	–	–	–	≤4		93%	100%
	ECAS ALS-specific		ECAS exec. functions		ECAS alternation
	Specificity	Sensitivity	Specificity	Sensitivity	Cutoff	Specificity			Sensitivity
WAIS-III Digit Span Backwards	76%	50%	93%	50%	≤11	64%			50%
	–	–	–	–	≤10	88%			50%
Trail Making Test B	82%	100%	91%	33%	≤11	71%			100%
	–	–	–	–	≤10	94%			100%

Note. Standard neuropsychology cutoffs based on 2SD below normative mean.

The ECAS ALS-specific score was moderate-to-high in specificity and had low-to-moderate sensitivity for identifying social cognition, inhibition, and working memory deficits. Sensitivity and specificity were high for identifying alternation impairments as indexed by the Trail Making Test B, but sensitivity was poor when indexed by WAIS Digit Span Backwards. The Executive Function subscore is high in specificity but very poor in sensitivity for social cognition, inhibition, working memory, and alternation.

The Social Cognition subtest had high specificity but lacked sensitivity for detecting social cognitive deficits. The Sentence Completion subtest had moderate specificity and high sensitivity against the HSCT Inhibition Error scaled score when a cutoff of ≤10 was used; however, poor specificity against the Stroop Interference score. The Digit Span and Alternation subtests had high specificity and 100% sensitivity when cutoffs of ≤4 and ≤10, respectively, were used.

Discussion

The ECAS has become the most widely-used screening tool for identifying cognitive and behavioral deficits in MND; however, the effectiveness of its executive function domain has been equivocal. The present study assessed the validity of the ECAS as a measure of executive function and social cognitive impairments, indexed by its ability to correctly dismiss unimpaired patients (i.e. specificity) and correctly detect impaired patients (i.e. sensitivity) on comparable neuropsychological measures.

Cognitive impairment

As a group, MND patients were largely intact on neuropsychology tests except for verbal initiation on the HSCT, verbal learning, visuomotor alternation and social cognition (both emotion recognition and faux pas detection). Poorer initiation than inhibition on the HSCT is like that observed in dynamic aphasia in FTD (37). Verbal output may be vulnerable in MND, which is a consideration in screening tools. The social cognitive impairment appears to be more prominent than their executive function impairments. Thus, it is important for screening tools to include an effective measure of social cognition.

ECAS validation

Similar to previous validation studies (12,14), approximately one third of MND patients were cognitively impaired as measured by the ECAS overall score and about a quarter were impaired on both the ALS-Specific and non-ALS-specific scores. Apart from the visuospatial domain, the lowest number of patients impaired was found for the ECAS executive function domain (8%).

The ECAS sentence completion subtest was highly sensitive and moderately specific in detecting impaired ability to inhibit a prepotent response, particularly when the cutoff was set at 10. In our sample, the ECAS executive function domain as a whole is highly specific (91%), but has very poor sensitivity (8%), as indexed by the HSCT Suppression Error SS. This means that the four tests combined are poor at detecting inhibitory impairments, and a substantial proportion of patients impaired on the HSCT Suppression Error SS were identified as unimpaired on the ECAS executive function score. The ECAS ALS-Specific score had good specificity in detecting an inhibition impairment, but did not detect two-thirds of truly impaired patients.

The social cognition subtest has very good specificity (86–91%), in terms of not picking up as impaired patients whose performance on the Mini-SEA was unimpaired. However, it has poorer sensitivity, particularly when identifying patients who were impaired on faux pas detection (14%). Sensitivity in detecting emotion recognition deficits was comparatively better (60%), but almost half of impaired patients were not detected. The ECAS social cognition subtest was adapted from Baron-Cohen and colleagues (38), intended to measure theory of mind. However, it can also be argued to assess social perception -i.e. interpret social cues from eye gaze. Within Adolphs’ framework (28), this is social judgment of faces. The Mini-SEA measures emotion recognition and faux pas recognition. The emotion recognition test overlaps with the ECAS social cognition test, in that it also taps social perception and social judgment of faces. However, faux pas stories assess theory of mind and social reasoning (28,29). It may also assess social behavior as it requires knowledge of whether a behavior is socially acceptable or appropriate. Judging from the sensitivity scores, the ECAS social cognition subtest primarily measures emotion recognition at the level of facial perception.

Similarly, the ECAS executive function subscore was highly specific in detecting social cognitive deficits (>90%), but had poor sensitivity across all Mini-SEA outcome measures (≤20%). Thus, the four components of the ECAS executive function domain may be too diverse, and impairments may nullify each other. For instance, someone impaired on inhibition may not be impaired on social cognition, and therefore their total score would not fall below the executive function domain cutoff. It is a strong possibility that social cognition is distinct from executive functions and should be assessed as a separate domain. Executive skills and social cognition recruit non-overlapping brain regions, and clinical cohorts can be impaired on one while intact on the other (39,40). The ECAS ALS-Specific score has good specificity, but again poorer sensitivity, although it was generally better at detecting patients who were impaired on the Mini-SEA, compared with the social cognition subtest and the executive function subscore. One possibility is that patients who are impaired on the social cognition subtest are generally more impaired across domains.

Conclusion

The present study assessed how well the ECAS detects cognitive difficulties in areas only recently highlighted to be problematic in MND; namely, executive function and social cognition. Overall, the ECAS ALS-Specific score has good specificity but lower sensitivity for identifying inhibition, working memory and social cognitive impairments. This means that impairments may go undetected and potentially result in patients not receiving appropriate information and subsequent management of difficulties. Additionally, the executive function score has reduced sensitivity to all four components (inhibitory control, working memory, alternation and social cognition). This reinforces the idea that executive functions comprise diverse processes, albeit with some commonalities, associated with distinct brain regions (15,16). Further, social cognition may be dissociable from executive functions. This points to an adjustment of the ECAS executive function domain, with a suggestion of removing the social cognition subtest to be a separate domain. Furthermore, it seems to only overlap with certain aspects of emotion recognition, particularly the direction of eye gaze in social judgment of faces. Social cognition itself is an emerging field, with varying conceptualisations of its components. Future studies should evaluate which aspects of social cognition are impaired in MND, and ensure that screening tools tap these specifically.

Acknowledgements

We wish to thank all participants for their time, especially MND patients and their families.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Funding

This work was supported by an Australian National Health and Medical Research Council (NHMRC) Boosting Dementia Research Leadership Fellowship (APP1135769) awarded to GR, and the Brazil Family Program for Neurology.

Notes

1 The Stroop Test consists of two sections. First, participants read aloud a list of colour words (tan, red, blue, green). The Stroop Colour Naming score reflects the number achieved in 2 min. Second, the words are then presented in an incongruent ink colour (e.g. “blue” printed in red ink) and participants name the colour of the ink. The Stroop Interference score reflects the number named in 2 minutes and Stroop Errors reflects the number of inhibition errors. The Stroop Difference score is calculated as the proportional difference between second interference section minus the first reading section (as detailed in Martin et al. (41,42)).