Psychometric Properties of Cognitive Assessment in Amyotrophic Lateral Sclerosis: A Systematic Review

Figures & data

Table 1 List of the Reported Psychometric Properties in Phase II – Definitions, Explanations and Recommendations of Preferred Statistical Method and Minimum Sample Size Included in Analyses

Psychometric Property – Definition and Explanations^Citation21	Preferred Statistical Method^Citation21	Sample Size^Citation21,Citation22
Internal consistency:
Internal consistency measures the degree of	-Cronbach’s alpha	50
interrelatedness among items. Internal consistency is only relevant when items form a reflective model, meaning all items are a manifestation of the same construct.	-Goodness of fit statistics	100-500
Assessment of the unidimensionality of the test is a prerequisite for internal consistency statistics to be evaluated objectively.	-Factor analysis or item response theory analysis	100-500
	-Internal consistency coefficient for each (unidimensional) (sub)scale separately.	50
Reliability:
Reliability is the degree to which a test is free from measurement error.	-ICC -Cohen’s kappa or weighted kappa depending on the scale used.	50
-Test-retest reliability refers to the robustness of the measurement error related to different testing times.
-Interrater reliability refers to the robustness of the measure-error related to different scorers. For evaluation of inter-rater reliability, the raters need to be independent.
Content validity (including face validity)
Content validity represent the degree to which the content of a test adequately reflect the construct to be measured. Content validity of a test is determined by asking patients for patient-reported tests or experts within the field of interest to judge the relevance and comprehensiveness of a test’s items. Evaluation of face validity is subjective; therefore, no standard exists.
Construct validity
Construct validity represent the degree to which the scores of the test are consistent with hypotheses based on the assumption that the test validly measures the construct to be measured.
-The structural validity aspect of construct validity refers to the degree to which the score of a test is an adequate reflection of the dimensionality of the construct to be measured. It is only relevant for tests that are based on a reflective model.	-Factor analysis	100
-The hypothesis testing aspect of construct validity concerns the relationships to scores of other instruments. It is recommended to formulate a priori hypotheses, to specify the expected direction (positive or negative) of correlations one expects, or to quantify the range of expected differences between groups or scores. It is important that the comparator test is clearly described and has adequate measurement properties in order to interpret correlations properly.	-Correlations	50
Criterion validity
Criterion validity refers to the degree to which the scores of a test are an adequate reflection of a ‘gold standard’ (an external criterion of the phenomenon being measured). For health-related patient reported outcome measures, the only possibility is to compare a shortened version of a test to the original long version.	-Correlations or the area under the ROC. -Sensitivity and specificity for dichotomous scores	50
Responsiveness
Responsiveness is the ability of a test to detect change over time in the construct being measured. When evaluating responsiveness, at a minimum, at least some of the patients should display changed performance (eg, improvement or decrement). Therefore, it is important to describe precisely what occurs during the follow-up. Responsiveness may be evaluated by hypotheses testing or comparison to a ‘gold standard’.	-Hypotheses testing: Correlations -Comparison to a gold standard: Correlations or the area under the ROC. -Sensitivity and specificity for dichotomous scores	50

Abbreviations: eg, for example; ICC, intraclass correlation coefficient; ROC, receiver operating characteristic.

Figure 1 Flow of information through Phase I of the systematic review: Identifying cognitive tests.

Notes: Adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10. Creative Commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.¹²

Abbreviation: ALS, amyotrophic lateral sclerosis.

Figure 2 Flow of information through Phase II of the systematic review: Evaluating psychometric properties.

Notes: Adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10.  Creative Commons license and disclaimer available from: http://creativecommons.org/licenses/by/4.0/legalcode.¹²

Abbreviation: ALS, amyotrophic lateral sclerosis.

Table 2 List of the Most Frequently Reported Cognitive Tests, Test Batteries and Scales Used in Patients with Amyotrophic Lateral Sclerosis (ALS) in 319 Articles Identified in Phase I

Cognitive Domains and Tests (Total Number of Tests Identified per Category)^a		Number of Articles^b
Cognitive domain, followed by 2 most-frequently used tests^c
Attention, orientation and working memory (n=34)
	Digit Span backward	89
	Stroop Test/Color- Word Interference Test	78
Concept formation and reasoning (n=15)
	Card sorting tests	77
	Raven’s Coloured Progressive Matrices	52
Construction and motor performance (n=12)
	Block Design	17
	Complex Figure Test (copy)	9
Executive functioning (n=12)
	Phonemic fluency tests	148
	Category fluency tests	99
Learning and memory (n=37)
	Complex Figure Test (recall)	43
	Rey Auditory- Verbal Learning Test	41
Perception (n=14)
	Benton Judgement of Line Orientation	23
	Visual Object and Space Perception Battery	16
Social cognition (n=12)
	Reading the Mind in the Eyes Test	10
	Judgement of Preference Test	4
Verbal function and language skills (n=42)
	Boston Naming Test	52
	North American Adult Reading Test	34
Other kind of batteries or inventories, followed by 2 most-frequently used tests^c
Test batteries (n=13)
	Wechsler Adult Intelligence Scale	12
	Mental Deterioration Battery	2
Rating scales, inventories and screening tests (n=32)
	Mini-Mental State Examination	134
	Frontal Assessment Battery	44

Notes: ^aCognitive tests are categorised according to the cognitive domain they assess, whereas the other kinds are categorised according to type of test (eg, scale, battery, etc.), as indicated. ^bA given article could report on more than one test. ^cSee Supplemental material 2 for the complete list of all reported tests.

Table 3 Characteristics of Articles Included in Phase II

Study Characteristics	Values
Articles, number	46
Publication year, range	1985-2019
ALS-sample size per study, range	5-274
Mean age (year) of ALS patients, range	50.7 – 68.5
Recruitment setting, number of articles
Hospital or clinical	40
Register data	2
Others	4
Continent, number of articles
America	14
Asia	6
Europe	22
Oceania	4
Study design, number of articles
Cross sectional	38
Longitudinal	8
Cohort study	3
Case-control	19
Diagnostic	22
Combination of designs	2
Psychometrics, number of articles
Internal consistency	9
Reliability	5
Content validity	1
Construct validity – structural validity	4
Construct validity - hypothesis testing	28
Criterion validity	29
Responsiveness	3
Feasibility	5

Abbreviation: ALS, amyotrophic lateral sclerosis.

Table 4 List of Cognitive Tests Evaluated in an ALS Population, Identified in the First and Second Search (2017 and 2019) of the Review and Examined in Phase II, Classified into Cognitive Domains

Test	Country of Validation (Alphabetical Order)	Reference Number
Executive functioning
Category fluency tests	Italy	^Citation25
Phonemic fluency tests	Italy, Scotland	^Citation25,Citation26
Learning and memory
Prospective memory test	China	^Citation27
Social cognition
Awareness of Social Inference Test	Australia	^Citation28
Ekman 60 Faces Test	England	^Citation29
Reading the Mind in the Eyes Test	Ireland	^Citation30
Screening tests
Addenbrooke’s Cognitive Examination	Australia, Canada, China, Greece, Japan	^Citation31^–^Citation36
ALS Brief Cognitive Assessment	United States of America	^Citation37
ALS Cognitive Behavioural Screen	Brazil, Italy, Spain, United States of America	^Citation38^–^Citation42
Edinburgh Cognitive and Behavioural ALS Screen	China, Czech Republic, Germany, Greece, Ireland, Scotland, Spain, Switzerland	^Citation5,Citation35,Citation43^-^Citation53
Frontal Assessment Battery	Australia, Canada, Italy, Japan, Korea, United States of America	^Citation25,Citation36,Citation54^-^Citation58
Mini-Mental State Examination	China, Italy, United States of America	^Citation25,Citation34,Citation37
Montreal Cognitive Assessment	Canada, Japan	^Citation55,Citation58
Penn State Brief Exam of Frontal and Temporal Dysfunction Syndromes	United States of America	^Citation59
University of California San Francisco Screening Battery	United States of America	^Citation60,Citation61
Test batteries^a
ALS-Computerized Frontal Battery	Canada	^Citation62
Battery I: Frontal Behavioral Inventory, Phonemic fluency test, Beck Depression Inventory	United States of America	^Citation63
Battery II: Addenbrooke’s Cognitive Examination-Revised, Digit Span-backword, Hayling Test, Iowa Gambling Test, Ekman 60 Faces-modified, Cambridge Behavioural Inventory-Revised	Australia	^Citation64
Battery III: ALS Cognitive Behavioural Screen, Phonemic fluency test, Semantic fluency test, Boston Naming Test-short version, Comprehension of Instructions, Peabody Picture Vocabulary Test-Revised	Greece	^Citation65
Battery IV: Token Test-modified, Graded Naming Test, British Picture Vocabulary Scale-II, Test for the Reception of Grammar, Pyramids and Palm Trees Test, Kissing and Dancing Test, Cookie Theft Picture	England	^Citation66
Battery V: Frontal Behavioural Inventory-Modified, Mini-Mental State Examination, Rey Auditory- Verbal Learning Test, Controlled Oral Word Fluency Test	United States of America	^Citation67
Communicative Participation Item Bank	United States of America	^Citation68
Mental Deterioration Battery	Italy	^Citation69

Notes: ^aBatteries I–V were developed specifically for use in patients with ALS.

Abbreviation: ALS, amyotrophic lateral sclerosis.

Table 5 Levels of Evidence: Quality of Psychometric Properties and Feasibility of Tests in Articles Identified During Phase II

Test	Internal Consistency	Reliability	Content Validity	Structural Validity	Hypothesis Testing	Criterion Validity	Responsiveness	Feasibility
Executive
Category fluency tests						?
Phonemic fluency tests					?	?
Learning memory
Prospective memory test					?
Social cognition
Awareness of Social Inference Test					++
Ekman 60 Faces Test					?
Reading the Mind in the Eyes Test	+++			+++
Screening test
Addenbrooke’s Cognitive Examination	?			?	+++	?	+	x
ALS Brief Cognitive Assessment						?
ALS Cognitive Behavioural Screen						?		x
Edinburgh Cognitive and Behavioural ALS Screen	?	?			++	++	?	x
Frontal Assessment Battery					+++	?	+	x
Mini-Mental State Examination						+/-
Montreal Cognitive Assessment					++	?		x
Penn State Brief Exam of Frontal and Temporal Dysfunction Syndromes
University of California San Francisco Screening Battery		?			+	?
Test batteries ^a
ALS-Computerized Frontal Battery					?
Battery I					?			x
Battery II	?			?
Battery III						?
Battery IV		?			+
Battery V					?
Communicative Participation Item Bank	+		?			?
Mental Deterioration Battery				+

Notes: +++ or — = Strong evidence with consistent positive or negative results from multiple studies with good methodological quality or one study with excellent methodological quality. ++ or – = Moderate evidence with consistent positive or negative results from multiple studies with fair methodological quality or one study with good quality. + or - = Limited evidence with positive or negative results from a study with fair methodological quality. +/− = Conflicting evidence with conflicting results from individual studies. ? = Unknown level of evidence due to results from studies with poor methodological quality. x =Feasibility was evaluated. Gray-highlighted box indicate high to moderate level of evidence. ^aBatteries I–V were developed specifically for use in patients with ALS.

Abbreviation: ALS, amyotrophic lateral sclerosis.

Mokkink L, Terwee C, Patrick D, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–549. doi:10.1007/s11136-010-9606-8

 PubMed Web of Science ®Google Scholar

Terwee C, Bot S, de Boer M, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42. doi:10.1016/j.jclinepi.2006.03.012

 PubMed Web of Science ®Google Scholar

Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:10. doi:10.1016/j.jclinepi.2008.05.006

 Web of Science ®Google Scholar

Floris G, Borghero G, Chiò A, et al. Cognitive screening in patients with amyotrophic lateral sclerosis in early stages. Amyotroph Lateral Scler. 2012;13:95–101. doi:10.3109/17482968.2011.605453

 PubMed Web of Science ®Google Scholar

Abrahams S, Leigh P, Harvey A, Vythelingum G, Grise D, Goldstein L. Verbal fluency and executive dysfunction in amyotrophic lateral sclerosis (ALS). Neuropsychologia. 2000;38(6):734–747. doi:10.1016/S0028-3932(99)00146-3

 PubMed Web of Science ®Google Scholar

Ji Y, Wei L, Chui D, Wang K, Fan D. Prospective memory tasks: a more sensitive method for screening cognitive impairment in ALS? BMC Neurol. 2012;12:142. doi:10.1186/1471-2377-12-142

 PubMed Web of Science ®Google Scholar

Staios M, Fisher F, Lindell A, Ong B, Howe J, Reardon K. Exploring sarcasm detection in amyotrophic lateral sclerosis using ecologically valid measures. Front Hum Neurosci. 2013;7:178. doi:10.3389/fnhum.2013.00178

 PubMed Web of Science ®Google Scholar

Papps B, Abrahams S, Wicks P, Leigh P, Goldstein L. Changes in memory for emotional material in amyotrophic lateral sclerosis (ALS). Neuropsychologia. 2005;43(8):1107–1114. doi:10.1016/j.neuropsychologia.2004.11.027

 Web of Science ®Google Scholar

Burke T, Pinto-Grau M, Lonergan K, et al. Measurement of social cognition in amyotrophic lateral sclerosis: a population based study. PLoS One. 2016;11:8. doi:10.1371/journal.pone.0160850

 Web of Science ®Google Scholar

Chenji S, Mah D, Johnston W, Camicioli R, Fisher N, Kalra S. Utility of the Addenbrooke’s cognitive examination in amyotrophic lateral sclerosis. Can J Neurol Sci. 2018;45(5):527–532. doi:10.1017/cjn.2018.68

 Web of Science ®Google Scholar

Xu Z, Alruwaili A, Henderson R, McCombe P. Screening for cognitive and behavioural impairment in amyotrophic lateral sclerosis: frequency of abnormality and effect on survival. J Neurol Sci. 2017;376:16–23. doi:10.1016/j.jns.2017.02.061

 PubMed Web of Science ®Google Scholar

Hu W, Shelnutt M, Wilson A, et al. Behavior matters-cognitive predictors of survival in amyotrophic lateral sclerosis. PLoS One. 2013;8:2.

 Web of Science ®Google Scholar

Murphy J, Factor-Litvak P, Goetz R, et al. Cognitive-behavioral screening reveals prevalent impairment in a large multicenter ALS cohort. Neurology. 2016;86(9):813–820. doi:10.1212/WNL.0000000000002305

 PubMed Web of Science ®Google Scholar

Turon-Sans J, Gascon-Bayarri J, Reñé R, et al. Cognitive impairment in ALS patients and validation of the Spanish version of the ALS-CBS test. Amyotroph Lateral Scler Frontotemporal Degener. 2016;17(34):221–227. doi:10.3109/21678421.2015.1125500

 Web of Science ®Google Scholar

Abrahams S, Newton J, Niven E, Foley G, Bak T. Screening for cognition and behaviour changes in ALS. Amyotroph Lateral Scler Frontotemporal Degener. 2014;15(12):9–14. doi:10.3109/21678421.2013.805784

 Web of Science ®Google Scholar

Kourtesis P, Christidi F, Margioti E, et al. The Edinburgh cognitive and behavioral amyotrophic lateral sclerosis screen (ECAS): sensitivity in differentiating between ALS and Alzheimer’s disease in a Greek population. Amyotroph Lateral Scler Frontotemporal Degener. 2019;30:1–8.

 Google Scholar

Burkhardt C, Neuwirth C, Weber M. Longitudinal assessment of the Edinburgh cognitive and behavioural amyotrophic lateral sclerosis screen (ECAS): lack of practice effect in ALS patients? Amyotroph Lateral Scler Frontotemporal Degener. 2017;18(34):202–209. doi:10.1080/21678421.2017.1283418

 Web of Science ®Google Scholar

Bezdíček O, Marková L, Soósová N, Forgáč M. Czech version of the Edinburgh cognitive and behavioral amyotrophic lateral sclerosis screen – a pilot study. Cesk Slov Neurol N. 2018;81(2):208–212. doi:10.14735/amcsnn2018208

 Web of Science ®Google Scholar

Ahn S, Kim S, Kim J, et al. Frontal assessment battery to evaluate frontal lobe dysfunction in ALS patients. Can J Neurol Sci. 2011;38(2):242–246. doi:10.1017/S0317167100011409

 PubMed Web of Science ®Google Scholar

Osborne R, Sekhon R, Johnston W, Kalra S. Screening for frontal lobe and general cognitive impairment in patients with amyotrophic lateral sclerosis. J Neurol Sci. 2014;336(12):191–196. doi:10.1016/j.jns.2013.10.038

 Web of Science ®Google Scholar

Wei Q, Chen X, Zheng Z, et al. Screening for cognitive impairment in a Chinese ALS population. Amyotroph Lateral Scler Frontotemporal Degener. 2015;16(12):40–45. doi:10.3109/21678421.2014.966311

 Web of Science ®Google Scholar

Ohta Y, Sato K, Takemoto M, et al. Behavioral and affective features of amyotrophic lateral sclerosis patients. J Neurol Sci. 2017;381:119–125. doi:10.1016/j.jns.2017.08.024

 PubMed Web of Science ®Google Scholar

Flaherty-Craig C, Brothers A, Yang C, Svoboda R, Simmons Z. Declines in problem solving and anosognosia in amyotrophic lateral sclerosis: application of Guilford’s structure of intellect theory. Cog Behav Neurol. 2011;24(2):26–34. doi:10.1097/WNN.0b013e3182138454

 Web of Science ®Google Scholar

Christodoulou G, Gennings C, Hupf J, et al. Telephone based cognitive-behavioral screening for frontotemporal changes in patients with amyotrophic lateral sclerosis (ALS). Amyotroph Lateral Scler Frontotemporal Degener. 2016;17(78):482–488. doi:10.3109/21678421.2016.1173703

 Web of Science ®Google Scholar

Murphy J, Ahmed F, Lomen-Hoerth C. The UCSF screening exam effectively screens cognitive and behavioral impairment in patients with ALS. Amyotroph Lateral Scler Frontotemporal Degener. 2015;16(12):24–30. doi:10.3109/21678421.2014.960873

 Web of Science ®Google Scholar

Gillingham S, Yunusova Y, Ganda A, et al. Assessing cognitive functioning in ALS: A focus on frontal lobe processes. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18(34):182–192. doi:10.1080/21678421.2016.1248977

 Web of Science ®Google Scholar

Gordon PH, Wang Y, Doorish C, et al. A screening assessment of cognitive impairment in patients with ALS. Amyotrophic Lateral Sclerosis. 2007;8:362–365. doi:10.1080/17482960701500817

 PubMed Web of Science ®Google Scholar

Lillo P, Savage S, Mioshi E, Kiernan M, Hodges J. Amyotrophic lateral sclerosis and frontotemporal dementia: A behavioural and cognitive continuum. Amyotroph Lateral Scler. 2012;13(1):102–109. doi:10.3109/17482968.2011.639376

 PubMed Web of Science ®Google Scholar

Nidos A, Kasselimis D, Simos P, et al. Frontotemporal dysfunction in amyotrophic lateral sclerosis: A discriminant function analysis. Neurodegener Dis. 2016;16(34):140–146. doi:10.1159/000381186

 Web of Science ®Google Scholar

Tsermentseli S, Leigh P, Taylor L, Radunovic A, Catani M, Goldstein L. Syntactic processing as a marker for cognitive impairment in amyotrophic lateral sclerosis. Amyotroph Lateral Scler Frontotemporal Degener. 2015;17(12):69–76. doi:10.3109/21678421.2015.1071397

 Web of Science ®Google Scholar

Heidler-Gary J, Hillis A. Distinctions between the dementia in amyotrophic lateral sclerosis with frontotemporal dementia and the dementia of Alzheimer’s disease. Amyotroph Lateral Scler. 2007;8(5):276–282. doi:10.1080/17482960701381911

 PubMed Web of Science ®Google Scholar

Baylor C, Yorkston K, Eadie T, Kim J, Chung H, Amtmann D. The Communicative participation item bank (CPIB): item bank calibration and development of a disorder-generic short form. J Speech Lang Hear Res. 2013;56(4):1190–1208. doi:10.1044/1092-4388(2012/12-0140)

 PubMed Web of Science ®Google Scholar

Gallassi R, Montagna P, Ciardulli C, Lorusso S, Mussuto V, Stracciari A. Cognitive impairment in motor neuron disease. Acta Neurol Scand. 1985;71(6):480–484. doi:10.1111/j.1600-0404.1985.tb03231.x

 PubMed Web of Science ®Google Scholar