The Executive Checklist (EC-10) – a new rating instrument for clinicians assessing dysexecutive behavior

Abstract

Assessment of executive functions (EF) is often criticized for its lack of ecological validity. As a consequence, several self- and partner rating scales for EF have been developed, while rating scales designed for clinicians are lacking. We therefore developed the Executive Checklist (EC-10), a new rating scale for clinicians assessing dysexecutive behavior during neuropsychological assessment and examined its psychometric properties. Consecutive referrals from a memory clinic with subjective cognitive impairment (SCI; n = 27), mild cognitive impairment (MCI; n = 29), dementia (DEM; n = 16), as well as 11 healthy controls were assessed with the EC-10 while performing common neuropsychological tests. Results showed that the EC-10 had excellent inter-rater reliability, good internal consistency and modest relations to cognitive laboratory measures. The EC-10 increased the classification rate above and beyond the influence of neuropsychological tests when comparing patients with SCI and MCI or between cognitively impaired and non-impaired patients. Conclusively, the present study demonstrates that clinical observations can be quantified using the EC-10 and that this rating provides valuable information. As executive deficits are common in many neurological and neuropsychiatric disorders, validating the EC-10 in broader patient groups should be an important avenue for future research.

Keywords:

• Alzheimer’s disease
• ecological validity
• executive function deficits
• mild cognitive impairment
• rating instrument

Performance-based tests used to assess executive functioning (EF) deficits (i.e., impairment in working memory, inhibitory control, cognitive flexibility and/or planning) often display poor ecological validity as they are only weakly related to patients’ difficulties in daily life functioning (Barkley & Murphy, 2010; Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Chaytor & Schmitter-Edgecombe, 2003). Consequently, several self- and partner rating scales for EF have been developed, but the area still lacks a rating scale designed to enable clinicians to rate the patient’s behavior during neuropsychological assessment. To address this limitation, we developed a new behavioral rating instrument, the Executive Checklist (EC-10), which can be used by clinicians to quantify dysexecutive behavior during neuropsychological testing to obtain a more comprehensive assessment of the patient’s executive deficits. The present study aimed to examine the psychometric properties of the EC-10, including to what extent this instrument can be used to differentiate between clinical groups of older adults with different degree of cognitive decline and a healthy control group.

The continuum of cognitive decline

It is increasingly recognized that progression along the trajectory from Subjective Cognitive Impairment (SCI) to Mild Cognitive Impairment (MCI) to dementia such as Alzheimer’s disease (AD), may begin years before clinical onset with very subtle cognitive alterations (Dubois et al., 2014; Han, Nguyen, Stricker, & Nation, 2017; Nordlund et al., 2010; Petersen et al., 1999; Reisberg, Shulman, Torossian, Leng, & Zhu, 2010; Sperling et al., 2011; Winblad et al., 2004). Notably, previous studies have shown that memory impairment in combination with EF deficits predict conversion to dementia better than memory impairment alone (Albert, Moss, Tanzi, & Jones, 2001; Dickerson, Wolk, & Alzheimer Disease Neuroimaging Initiative, 2011; Espinosa et al., 2009; Hessen et al., 2014; Martín et al., 2016; Nordlund et al., 2010; Rapp & Reischies, 2005; Tabert et al., 2006). Other studies have shown a higher frequency of EF deficits in patients with MCI compared to healthy controls (Rabin et al., 2006). In addition, EF deficits have been shown to be related to impaired functional ability in instrumental activities of daily living (e.g., shopping, using the telephone, managing medication and finances) among patients with MCI (Martín et al., 2016; Pereira, Yassuda, Oliveira, & Forlenza, 2008) or dementia (Martyr & Clare, 2012; Royall, Palmer, Chiodo, & Polk, 2004). It has therefore been considered important to use wide-ranging cognitive assessment methods to further elucidate and identify subgroups of patients with MCI who may be at especially high risk of progression to dementia (Han et al., 2017; Mitchell & Shiri‐Feshki, 2009; Petersen et al., 2001; Winblad et al., 2004).

Assessment of executive deficits

As described above, assessing EF deficits in patients with cognitive decline is important. However, the assessment of EF deficits is an area of controversy. Remarkably, neuropsychological tests and rating scales have been shown to share less than 10% of the variance across studies among, for example, patients diagnosed with ADHD (Barkley & Murphy, 2010). It has therefore been argued that tests and ratings capture at least partly different constructs and that tests and ratings have different strengths and limitations (see review by Toplak, West, & Stanovich, 2013). However, it should also be noted that ratings provided by the patient him-/herself or a significant other also have limitations, the most important one probably being rater bias (Toplak et al., 2013). We therefore did not intend to create the EC-10 as a replacement for neuropsychological tests, but rather as an instrument that can be used in combination with neuropsychological tests to capture the patient’s behavior while performing the tests. Most clinicians are already making these observations on a qualitative basis, but the EC-10 has the advantage of providing a more standardized, quantitative way of collecting observational data.

Aims of the study

The aim of the present study was to address the poor ecological validity of EF tests and to bridge the gap between neuropsychological tests and rating scales by introducing the EC-10, a newly developed behavioral rating instrument, designed to enable clinicians to rate dysexecutive behavior in patients undergoing neuropsychological assessment. More specifically, the following research objectives were addressed:

1. To examine the reliability of the EC-10 by investigating internal consistency and inter-rater reliability.

2. To examine the construct validity of the EC-10 by examining relations to neuropsychological tests assessing general intellectual ability, visual memory, verbal episodic memory and working memory.

3. To examine the discriminative validity of the EC-10 by examining group differences between healthy elderly controls, patients with SCI, patients with MCI, and patients with dementia.

4. To determine whether the EC-10 increases the number of correctly classified individuals among patients with different levels of cognitive decline, above and beyond the influence of neuropsychological tests.

Methods

Participants and procedure

The study included four different groups: (1) patients with Subjective Cognitive Impairment (SCI; n = 27), (2) patients with Mild Cognitive Impairment (MCI; n = 29), (3) patients with dementia (n = 16; Alzheimer’s dementia = 13 and subcortical vascular dementia = 3) and (4) healthy controls without any psychiatric disorder (n = 11). All clinical patients were recruited from the Memory Clinic, Karolinska University Hospital, Sweden. In accordance with clinical praxis at the Memory Clinic, participants underwent extensive testing before receiving a diagnosis. This included the following: (1) cognitive screening using the Mini Mental State Examination (MMSE; Folstein, Folstein, & McHugh, 1975); (2) neurological examination; (3) clinical interview with the patient and a significant other by a physician; (4) laboratory measures, including analysis of blood and cerebral spinal fluid for biomarkers; (5) magnetic resonance imaging; (6) neuropsychological assessment; (7) diagnostic conference where all steps were evaluated according to clinical guidelines. With regard to MCI, F06.7 criteria presented in the International Statistical Classification of Diseases and Related Health Problems – Tenth Revision (ICD-10; WHO, 2004) and Winblad et al. (2004) consensus criteria (i.e., 1.5 SD below adjusted norms in conjunction with clinical judgment) were used. With regard to dementia, the clinical criteria according to ICD-10 (WHO, 2004) were used. SCI was diagnosed in accordance with ICD-10 criteria (R41.8A). Exclusion criteria in the clinical groups were kept to a minimum in order to obtain a representative clinical sample. However, according to clinical praxis, patients were not eligible for assessment if they reported any ongoing substance abuse during the past six months. In addition, depressive symptoms were assessed using either the Cornell Scale for Depression in Dementia (CSDD; Alexopoulos, Abrams, Young, & Shamoian, 1988) or the Geriatric Depression Scale (GDS; Sheikh & Yesavage, 1986) to conclude that depressive symptoms were not the primary cause of the patients’ memory problems.

The healthy controls were recruited through advertisements at hospitals, schools, libraries, churches, cafés, and community centers. Inclusion criteria for the healthy controls were: (1) age between 50 and 75 years (in order to obtain a similar age range for all groups); (2) no self-perceived memory impairment; (3) no present major somatic condition; (4) no history of mental illness; (5) no ongoing substance abuse during the past 6 months. Descriptive data for all four groups are presented in Table 1.

Table 1. Mean, standard deviations and results of group comparisons for both the background variables and the EC-10 measures.

	Healthy controls (HC) M (SD)	Subjective Cognitive Impairment (SCI) M (SD)	Mild Cognitive Impairment (MCI) M (SD)	Dementia (DEM) M (SD)	F/χ^2	Post hoc analyses
Background variables
Age (years)	63.14 (6.37)	59.37 (10.03)	65.17 (10.24)	72.12 (9.83)	5.94***	HC < DEM; SCI < MCI < DEM
Education (years)	17.95 (3.02)	12.50 (4.25)	11.69 (2.48)	10.00 (2.13)	14.68***	HC > SCI, MCI > DEM
Gender (% female)	63.6%	77.8%	48.3%	62.5%	5.21, ns
MMSE	NA	28.26 (1.68)	27.54 (2.16)	25.00 (2.14)	12.74***	SCI, MCI > DEM
EC-10 variables
Total score	3.18 (7.53)	6.96 (6.19)	19.52 (10.28)	30.00 (12.24)	18.73***	HC, SCI < MCI < DEM
Frequency	1.27 (2.28)	3.70 (3.24)	9.96 (5.31)	14.75 (6.37)	18.25***	HC, SCI < MCI < DEM
Relevance	1.90 (5.41)	3.24 (3.03)	9.55 (5.07)	15.25 (6.14)	17.61***	HC, SCI < MCI < DEM

^*** p < .001, ns: non-significant; MMSE: Mini Mental State Examination; NA: Not available.

There were no significant differences between the four groups with regard to male/female ratio. However, as can be seen in Table 1, there were significant group differences with regard to age and educational level. With regard to age, significant group differences were expected as there is an increasing cognitive decline with age and SCI patients were therefore younger than MCI patients, who were in turn younger than the patients with dementia. The controls only differed from patients with dementia. Age and educational level were therefore used as covariates in all analyses. The study was carried out in accordance with the Declaration of Helsinki and was approved by the regional ethics committee at the Karolinska Institutet, Stockholm. All participants gave their written consent for participation in the study.

EC-10 rating scale

The EC-10 was developed to assess executive deficits when performing neuropsychological tests. The fundamental idea behind the EC-10 is that almost all neuropsychological tests involve a certain degree of EF and that the patients’ degree of dysexecutive behaviors can therefore be observed during the entirety of the neuropsychological assessment. The EC-10 includes items such as “The patient has difficulty following instructions and test rules” and “The patient has difficulty sorting out relevant information”. Several examples are provided for each item. Altogether, seven clinicians with formal education and experience using neuropsychological tests were included in the present study. The EC-10 is reliant on the competent observations of the clinician, and the clinician is advised to fill in the form as soon as possible after the neuropsychological assessment. The raters did not have knowledge of the diagnosis of the participants when completing the ratings. For each item, the clinician is asked to assess both the frequency (i.e., the occurrence of the behavior) and the relevance (i.e., the clinical significance of impairment) of the observed dysexecutive behavior. The clinicians did not receive any formal training regarding the EC-10 beyond the information presented on the scoring form (see Supplementary Appendix A). Completing the checklist takes about 5–10 min. The EC-10 is freely available for use in both research and clinical work and all 10 items included in the English version of the EC-10 are presented in Supplementary Appendix A. This version was created from the original Swedish version using translation and back-translation with the assistance of two bilingual researchers. Please contact the last author to obtain the original Swedish version of the EC-10.

Neuropsychological assessment

Although the EC-10 can possibly be used in many different patient groups, the aim of the study was to assess the usefulness of the EC-10 in patients at a Memory Clinic. In order for the study to be clinically relevant, we therefore chose to include tests that are part of the standard neuropsychological assessment for this patient group. The neuropsychological tests described in detail below, which were all selected from well-established international test batteries, were therefore included in the study. In line with common neuropsychological test standards, the clinician conducting the neuropsychological testing assessed the validity of the assessment and, when in doubt, used the Test of Memory Malingering (TOMM; Tombaugh, 1997) to verify that the patient was engaging appropriately in the tasks. No signs of malingering were observed within the present study.

Verbal episodic memory was assessed using the Rey Auditory Verbal Learning Test (RAVLT; Schmidt (1996). Here, participants are asked to listen to a list of 15 words and to repeat as many of the words as possible (i.e. immediate recall). This procedure is repeated five times. In the present study, the participants were also asked to recall all words they could remember 30 min after the initial learning procedure (i.e. delayed recall). The number of words recalled after 30 min was used to measure verbal episodic memory.

Verbal working memory was assessed using the subtests Digit span from the fourth edition of the Wechsler Adult Intelligence Scales (WAIS-IV; Wechsler, 2008). In the digit span test, participants are asked to listen to series of numbers and then repeat them in the same order as the administrator (i.e. Digit Span Forward), in backward order (i.e., Digit Span Backwards), or in sequential order from lowest to highest (i.e., Digit Span Sequencing). The sum score of all three subtests was used to measure working memory (Wechsler, 2008).

Visual memory was assessed using the Rey-Osterrieth Complex Figure Test (Osterrieth, 1944; Rey, 1941). In this task, participants are first asked to copy a complex figure as correctly as possible on a blank piece of paper while the figure is in front of them. In the second part of the task, they are asked to replicate the figure from memory on another piece of blank paper right after finishing the copying. Scoring was done according to Osterrieth’s (1944) 18-point criteria, and only immediate recall (i.e., second part of the task) was included as a measure of visual memory.

General intellectual abilities were investigated using one test from the verbal intelligence index (i.e., Information) and one test from the performance intelligence index (i.e., Block Design) taken from the WAIS-IV (Wechsler, 2008). The information subtest measures semantic memory and includes 25 a wide range of general knowledge questions with cumulative difficulty. The Block Design subtest measures visuospatial abilities and the participants are instructed to rearrange blocks that have various color patterns on different sides to match a pattern presented in a booklet in front of them. Both the Information and Block Design subtests have been shown to correlate highly with full scale IQ (Wechsler, 2008).

Covariates. Age and years of education were included as covariates in the analyses. As previously mentioned, age is the single strongest predictor of dementia, with 1-2% of 60-year olds and sometimes as many as 50% of 90-year olds being affected (Prince et al., 2013). While education does not show similarly strong effects on prevalence estimates, there are a number of studies showing increased risk estimates for developing AD for individuals with low education level compared to those with high education level (e.g., Meng & D’Arcy, 2012). As mentioned above under the heading “participants and procedure,” there were also significant group differences in both age and education, which also points to the importance of including these variables as covariates.

Statistical analysis

First, outliers were handled using the outlier labeling rule, but no outliers were identified (Hoaglin & Iglewicz, 1987). In all analyses, we used raw scores and included age and educational level as covariates. As a preliminary analysis, we first used Pearson product-moment correlations to investigate the relation between the two IC-10 subscales, frequency and relevance. As the first main analysis, internal consistency was calculated using Chronbach’s alpha. In line with recommendations (e.g., Nunnally, 1978), alpha values below .70 are considered unacceptable, .70–.79 fair, .80–.89 good and .90 and above excellent. Second, intra-class correlations (ICC) were used to assess inter-rater agreement between the clinician conducting the neuropsychological assessment and one of the authors (HV) using a random subsample of 19 participants. Separate measures of reliability were calculated for the EC-10 total score, the EC-10 frequency subscale and the EC-10 relevance subscale. In line with recommendations, ICC values less than .40 are considered poor, .40–.59 fair, .60–.74 good and .75 and above excellent (Cicchetti, 1994). Next, different types of validity were examined. First, the convergent validity of the EC-10 was examined by studying bivariate correlations between the three EC-10 measures and performance-based tests of memory and general intellectual ability. These analyses were performed separately for cognitively non-impaired (i.e., healthy controls and patients with SCI) and cognitively impaired (i.e., patients with either MCI or dementia) individuals. Second, the discriminative validity of the EC-10 was studied using Univariate Analysis of Variance (ANCOVA) to investigate differences between the four groups. Post hoc analyses (i.e., planned comparisons) were used to make the following comparisons: (1) healthy controls versus SCI, (2) SCI versus MCI, and (3) MCI versus dementia. Hedges’s variation of Cohen’s d, with pooled standard deviations, was used to estimate the size of the effects for the planned comparisons. According to Cohen (1988), .30 is considered a small effect, .50 a medium effect and .80 a large effect. Last, hierarchical logistic regression analyses were used to determine whether the EC-10 can increase the number of correctly classified individuals above and beyond the influence of cognitive tasks. The cognitive tasks were entered in the first step and the total score for the EC-10 in the second step. There were no indications of multicollinearity as the variance inflation factor (VIF) was below 5 for all independent variables. Altogether, three different logistic regression analyses were conducted: (1) SCI versus MCI, (2) MCI versus dementia, and 3) cognitively non-impaired (i.e., healthy controls and patients with SCI) versus cognitively impaired (i.e., patients with either MCI or dementia) individuals.

Results

When studying correlations between EC-10 frequency and EC-10 relevance, the results showed that these two subscales were very highly correlated in all four groups, with correlations ranging from .90 to .96, all ps < .001. However, we also studied the exact agreement between frequency and relevance by calculating a difference score (i.e., frequency minus relevance) between frequency and relevance using the scores for each item included in the EC-10. The results showed that whereas about 60% of participants in the cognitively non-impaired groups (i.e., healthy controls and the SCI group) had the same sum score for frequency and relevance, this was only the case for a minority of the patients with MCI (38%) or dementia (12%). Thus, the frequency and relevance subscales of the EC-10 capture different aspects, at least among cognitively impaired individuals.

Reliability

Internal consistency as measured with Cronbach’s alpha was good for both the frequency (α = .90) and relevance subscale (α = .89). Intra-class correlations showed excellent reliability between the two independent raters for the EC-10 total score (ICC = .91), as well as for the two subscales frequency (ICC = .90) and relevance (ICC = .88). The highest mean value (i.e., most problems) was found for item #9 (“The patient has difficulties creating strategies for solving tasks with multiple steps”) for both the frequency and relevance subscale. The lowest mean values were found for item #2 (“The patient digresses from the main thread of a conversation”) and item #5 (“The patient repeats solutions or has difficulties finishing tests”) for both subscales.

Validity

To assess the convergent validity of the EC-10, bivariate correlations between the EC-10 and the neuropsychological test measures were examined. As the correlations were very similar for the two subscales, only the correlation for the total scale is presented (see Table 2). The results showed that the EC-10 total score was significantly related to all cognitive variables when including all participants. However, a different pattern of results emerged when conducting separate analyses for non-impaired (i.e., healthy controls and patients with SCI) and cognitively impaired (i.e., patients with either MCI or dementia) individuals. Among cognitively non-impaired individuals, the EC-10 total score was only significantly associated with the immediate recall condition from the ROCFT measuring visual memory. However, among cognitively impaired individuals, the EC-10 total score was significantly related to visual memory as well as to both measures of general cognitive ability.

Table 2. Partial Correlations (using age & education as covariates) examining associations between the EC-10 total score and the Cognitive Measures.

	Cognitively non- impaired (HC & SCI)	Cognitively impaired (MCI & DEM)	Total sample
Visual memory
Rey-Osterrieth Complex Figure Test	−.43*	−.54***	−.60***
Verbal episodic memory
Rey Auditory Verbal Learning Test	.05	−.12	−.56***
Working memory
WAIS-IV Digit Span subtest	−.21	−.20	−.35**
General intellectual ability
WAIS-IV Information	−.18	−.39*	−.38**
WAIS-IV Block design	−.01	−.41*	−.34*

Next, the discriminative validity of the EC-10 was examined by investigating group differences between the four groups. The results of the ANCOVA using age and education as covariates (see Table 1) showed a significant main effect for EC-10 frequency, relevance as well as for the EC-10 total score. As also shown in Table 1, post hoc analyses showed that the healthy controls and patients with SCI did not differ significantly, but both two groups had significantly lower scores than patients with MCI, who in turn had significantly lower scores than patients with dementia. Effects sizes for the post hoc comparisons ranged from small (g = .34) for relevance to large (g = .80) for frequency for the comparison between healthy controls and SCI. All planned comparisons between patients with SCI, MCI or dementia showed large effect sizes (gs ranging from .84 to 1.50).

Last, logistic regression analyses were used to investigate to what extent EC-10 total score can increase the ability to differentiate between groups, beyond the influence of the neuropsychological measures (see Table 3). In the first model (Model 1), a comparison was made between patients with SCI and patients with MCI. The results showed that Step 1 (i.e., including only the cognitive variables) was significant χ² = 16.07, p < .01. Step 2 (i.e., when adding EC-10 total score) was also significant, χ² = 16.28, p < .01. As can be seen in Table 3, the results showed an increase in the number of correctly classified individuals for both patients with SCI and patients with MCI when the EC-10 total score was added to the model. When trying to differentiate between patients with MCI and patients with dementia (Model 2), the results showed that Step 1 of the model was significant, χ² = 16.88, p < .01, but not Step 2 when adding EC-10, χ² = 0.09, ns. Finally, when comparing cognitively non-impaired (i.e., healthy controls and patients with SCI) and cognitively impaired (patients with either MCI or dementia) individuals, the results showed significant effects of both Step 1, χ² = 35.27, p < .001, and Step 2, χ² = 18.61, p < .001. As shown in Table 3, the increase in the correctly classified cognitively non-impaired individuals was relatively small when adding EC-10 in Step 2, whereas a substantial increase in the correctly classified cognitively impaired individuals was found.

Table 3. Results of the logistic regression analysis.

	Proportion correctly classified
Model 1: SCI vs. MCI	SCI	MCI	Total
Step 1. Cognitive tasks & covariates	81.3%	81.3%	81.3%
Step 2. Cognitive tasks & covariates & EC-10	93.8%	87.5%	90.6%
Model 2: MCI vs. DEM	MCI	DEM	Total
Step 1. Cognitive tasks & covariates	87.5%	66.7%	80.0%
Step 2. Cognitive tasks & covariates & EC-10	87.5%	66.7%	80.0%
Model 3: HC + SCI vs. MCI + DEM	HC + SCI	MCI + DEM	Total
Step 1. Cognitive tasks* & covariates	88.9%	80.0%	84.6%
Step 2. Cognitive tasks* & covariates & EC-10	92.6%	92.0%	92.3%

Discussion

The aim of the present study was to introduce the EC-10, a new systematic behavioral rating instrument designed to enable clinicians to assess dysexecutive behavior in patients undergoing neuropsychological assessment. The results showed that the internal consistency and inter-rater reliability of the EC-10 ranged from good to excellent. With regard to convergent validity, the EC-10 was found to be modestly related to the neuropsychological tests, at least among cognitively impaired individuals. Concerning the instrument’s ability to differentiate between groups, the results showed that the patients with SCI had a significantly lower mean EC-10 total score than patients with MCI, who in turn had a significantly lower mean EC-10 total score than patients with dementia. The only groups that did not differ significantly were healthy controls and patients with SCI. Logistic regression analyses showed that the EC-10 total score increased the classification rate above and beyond the influence of neuropsychological tests, except when attempts were made to differentiate between patients with MCI and dementia. In sum, the results provide evidence that the ratings obtained using the EC-10 are reliable and valid. In addition, the EC-10 should be considered important when evaluating patients’ dysexecutive behavior, in that the EC-10 ratings were able to increase classification rates above the rates achieved when using only neuropsychological test data.

Reliability of the EC-10

With regard to the strengths that support the use of EC-10 during neuropsychological assessment, there are several important findings to discuss. First, behavioral observations are a vital part of neuropsychological assessment. However, the level of detail and specificity of these behavioral observations differ greatly between clinicians, and we therefore suggest using the EC-10 as a means of quantifying some of those qualitative observations the clinician is already doing. We believe that this will increase inter- and intra-professional communication, increase specificity and validity of these observations, and better show the severity of dysexecutive behavior and their probable effect on test results.

Second, it is important to acknowledge that the EC-10 showed overall good psychometric properties and most notably, excellent agreement between two independent raters (ICC ranging between .88 and .91). Other commonly used expert-rating scales assessing other constructs, such as the Hamilton Depression Rating Scale (Morriss, Leese, Chatwin, Baldwin, & Group, 2008) or the clinician-rated version of the Yale-Brown Obsessive Compulsive Scale (Goodman et al., 1989), have shown ICC of about .90. As for common EF self- and partner rating scales, the inter-rater reliability has generally been shown to be much weaker, for instance, .47 for the Dysexecutive Questionnaire (Barker, Morton, Morrison, & McGuire, 2011), between .62 and .72 for the Adult Executive Functioning Inventory (ADEXI; Holst & Thorell, 2018), and between .66 and .79 for the Barkley Deficits in Executive Functioning Scale (BDEFS; Barkley, 2011). However, EF interview measures, such as the Executive Interview, have been shown to have excellent (r = .90) inter-rater reliability (Royall, Mahurin, & Gray, 1992). This provides an interesting perspective on both the difficulties associated with assessing EF using ratings and the question of when to use expert ratings, self-, or partner ratings. Future studies should investigate the relationship between the EC-10 and EF rating instruments based on ratings made by other sources, such as the patients themselves or a significant other.

Another interesting aspect concerning the reliability of the EC-10 is that no manual for EC-10 exists at this stage, and that the clinical psychologists who performed the ratings did not receive any formal training in using the instrument. As the inter-rater reliability was high, this could be taken to suggest that psychologists are familiar with carrying out behavioral observations during assessment and therefore have little trouble quantifying them. This further supports the use of EC-10, as it takes only 5–10 min to complete and involves things the psychologist is essentially already doing – without the benefit of obtaining measurable data.

Convergent and discriminative validity of the EC-10

When examining convergent validity, correlation analysis showed that the EC-10 was significantly related to all of the cognitive measures when studying relations for the whole group, but a different pattern emerged when conducting the analyses separately for cognitively non-impaired and impaired individuals. In previous studies studying the relation between neuropsychological tests and self-ratings of EF deficits, such as the BDEFS (Barkley, 2011) and the ADEXI (Holst & Thorell, 2018), several non-significant relations have been found and the correlation coefficients have been between .20 and .30 or lower. Thus, the strengths of the correlations found between the EC-10 and the neuropsychological tests when conducting separate analyses for cognitively impaired and non-impaired individuals are comparable to or even higher than those found in previous studies. This finding also supports the notion that tests and ratings capture somewhat different constructs, and that these measures should not be used interchangeably but rather as a complement to each other (cf. Toplak et al., 2013). This is particularly important in clinical studies, as it is well-known that individuals with executive dysfunction can perform within the expected range on commonly used cognitive tasks, while still being classified as having severe deficits by self- and partner ratings that reflect everyday functioning (e.g. Burgess et al., 1998). Therefore, when considering the present results in a clinical setting, the EC-10 shows promise in bridging the gap between performance-based tests and rating scales, while potentially increasing the ecological validity of the neuropsychological assessment. However, further investigation is required of the ecological validity of the EC-10 using measures that reflect everyday functioning.

With regard to the ability of the EC-10 to differentiate between groups, we found a significant group difference between patients with different levels of cognitive impairment, with large effect sizes for most comparisons. In addition, we found that the EC-10 can increase the classification rates above and beyond the influence of neuropsychological tests. This is an important finding, as it demonstrates that the EC-10 provides additional information that increases the predictive value of the neuropsychological assessment when used in a clinical setting. These findings also imply that there is a high frequency of dysexecutive behavior among patients with MCI and dementia, which is in line with previous findings (Rabin et al., 2006; Razani et al., 2007; Shinagawa et al., 2007; Stokholm, Vogel, Gade, & Waldemar, 2005).

It should also be noted that when the EC-10 was added in Step 2 of the regression analyses, this did not improve the ability to differentiate between MCI and DEM. One possible explanation for this finding is that the EC-10 is more sensitive to subtle EF deficits than to the more severe executive deficits that are usually present among patients with dementia. Moreover, this finding might also reflect the heterogeneity within both groups, in that a wide range of cognitive deficits are present in patients with these disorders. It should also be noted that the results could have been influenced by the small sample size of the present study, especially with regard to the group with dementia and the fact that this group included patients with either Alzheimer’s disease or vascular dementia. Thus, based on the present results, the EC-10 appears to be a more sensitive instrument for discriminating between SCI and MCI, and less sensitive for discriminating between MCI and DEM. However, further studies are needed to determine whether these findings can be replicated in larger sample sizes.

Limitations, future directions and conclusions

As for the limitations of the present study, there are some aspects that need to be discussed. First, as mentioned above, the sample size was relatively small. That said, we did find significant group differences between patients with SCI, MCI, and dementia despite our small sample size, and it was only the comparison between healthy controls and patients with SCI that was non-significant. The effect size for the comparison between healthy controls and patients with SCI was large with regard to frequency, thus it is likely that the EC-10 would have been able to differentiate between these two groups as well if the sample size had been larger. In order to extend the findings of the present study, it would be of great value to include a larger number of patients with MCI and follow these patients over time in order to investigate to what extent the EC-10 can be used to predict conversion of MCI to dementia, as previous research has revealed that memory impairment in combination with EF deficits might influence both the pace and severity of cognitive decline in patients with MCI (Dickerson et al., 2011; Hessen et al., 2014; Martín et al., 2016; Nordlund et al., 2010; Rapp & Reischies, 2005; Tabert et al., 2006). Additionally, it would be of value to investigate associations between the EC-10 and ratings made by the patient him-/herself or by a close relative. It would also be of value to study the relation between the EC-10 and other neuropsychological functions besides memory, such as inhibitory control and cognitive flexibility. As executive deficits are common in many neurological and neuropsychiatric disorders, validating the EC-10 in broader patient groups should be an important avenue for future research and we encourage other researchers to also conduct such research as the instrument is freely available in the Supplementary Appendix.

In conclusion, the present findings clearly suggest that clinical observations provide important information and that they can be quantified using a rating instrument such as the EC-10. The scores obtained from the EC-10 are reliable and they can increase the ability to differentiate between SCI and MCI as well as cognitively non-impaired and cognitively impaired above and beyond the influence of neuropsychological tests.