The reliability and validity of brief cognitive screening tools used in traumatic brain injury: A systematic review

ABSTRACT

Reliable and valid cognitive screening tools are essential in the assessment of those with traumatic brain injury (TBI). Yet, there is no consensus about which tool should be used in clinical practice. This systematic review assessed psychometric properties of cognitive screening tools for detecting cognitive impairment in TBI. Inclusion criteria were: peer-reviewed validation studies of a cognitive screening tool(s); with a sample of adults aged 18–80 diagnosed with TBI (mild-severe); and with psychometrics consistent with COSMIN guidelines. Published literature was retrieved from MEDLINE, Web of Science Core Collection, EMBASE, CINAHL, and PsycINFO on 27 January 2022. A narrative synthesis was performed. Thirty-four studies evaluated the psychometric properties of a total of 22 cognitive screening tools, in a variety of languages. Properties assessed included structural validity, internal consistency, reliability, criterion validity (or diagnostic test accuracy), convergent/divergent validity, and discriminant validity. The Montreal Cognitive Assessment (MoCA) and the Mini Mental State Examination (MMSE) were the most widely validated cognitive screening tools for use in TBI. The MoCA had the most promising evidence of its psychometric properties, which has implications for clinical practice. Future research should aim to follow standard criteria for psychometric studies to allow meaningful comparisons across the literature.

KEYWORDS:

• Traumatic brain injury
• Screening tool
• Cognitive assessment
• Neuropsychological test
• Reliability
• Validity
• Psychometric assessment

Introduction

Traumatic brain injuries (TBI) are a prevalent cause of hospital admission, with 155,919 cases of head injury presenting to UK hospitals annually (Headway, 2017). The Global Burden of Disease study estimated that there were 27 million new cases of traumatic brain injury (TBI) in 2016 (James et al., 2019). Cognitive impairments, particularly in attention, memory and executive function domains, are common in those with moderate-severe TBI, and in the acute phase of mild TBI (Barman et al., 2016). In addition, cognitive impairment is associated with real-life functional impairment in TBI populations (Wilson et al., 2021).

A complete neuropsychological assessment is the gold-standard method of identifying cognitive impairment in TBI, allowing the detection and characterization of difficulties post-TBI in multiple domains, and synthesizing the findings with consideration of the wider biopsychosocial framework. However, this approach is costly and resource intensive and if all patients were to access neuropsychological assessment, this may result in lengthy waits. Given the prevalence of TBI and the demands on services, there is a need for brief, valid, and reliable screening tools. Cognitive screening tools are tools which are designed or used to screen for cognitive impairment; and typically include a brief set of cognitive test items. They can be multidimensional, with different sub-scales each aiming to measure different cognitive functions separately, or unidimensional, having an overall score representing overall cognitive function. A standard brief cognitive test, often assessing one domain, may be trialled as a screening tool. Screening tools have the potential to identify cognitive impairment in an efficient and cost-effective way at an early stage, allowing the targeting of gold-standard neuropsychological assessment, as well as rehabilitation (NICE, 2014; Scottish Acquired Brain Injury Network, 2017; Teager et al., 2020). While they have the benefit of detecting possible cognitive impairment at an early stage, they do not allow for the same level of detail and understanding of a person’s cognitive function as a neuropsychological assessment.

A variety of cognitive screening tools exist which may be helpful for detecting cognitive impairment in TBI (Abd Razak et al., 2019; Cullen et al., 2007). However, the heterogeneity of cognitive impairments in TBI has presented a challenge to researchers to identify valid and reliable tools (Teager et al., 2020). A number of individual studies were identified which reported on the validity of certain cognitive screening tools, such as the Mini-Mental State Examination (MMSE) and the Montreal Cognitive Examination (MoCA), in TBI populations (Gaber, 2008; Zhang et al., 2016). These tools were initially developed for and are validated for use in dementia populations. Major trauma centres often use widely available tools to assess TBI patients, such as the Addenbrooke’s Cognitive Examination-III (Hsieh et al., 2013) and the MoCA (Nasreddine et al., 2005; Teager et al., 2020), although the psychometric properties of these tools are either unknown or are yet to be summarized in a meaningful way.

A rigorous review of the psychometric properties of cognitive screening measures for use in TBI has never been completed. One previous review was identified; however, it focused only on diagnostic test accuracy designs and collated the findings of included studies, without evidence-based critical review (Canadian Agency for Drugs and Technologies in Health (CADTH), 2014). Other systematic reviews have been published on cognitive screening tool use in stroke populations, and have found that multi-domain tools such as the MoCA and Oxford Cognitive Screen (OCS) demonstrated acceptable psychometric qualities (Kosgallana et al., 2019; Stolwyk et al., 2014). However, important differences exist between stroke and TBI populations in pathology and the most prevalent profiles of cognitive impairment. Zhang et al. (2016) compared the validity of two different cognitive screening tools in TBI and stroke populations. The sensitivity of these tools to identify cognitive impairment differed between TBI and stroke groups, suggesting the psychometric properties of cognitive screening tools should not be assumed to be consistent across brain injury groups. No systematic review or meta-analysis could be identified on the psychometric properties of cognitive screening tools, specifically in TBI populations.

Therefore, there is a need to review the evidence for the psychometric properties of cognitive screening tools in TBI. This will support the refinement of clinical guidelines which, in turn, would contribute to evidence-based practice.

Objectives

This systematic review aimed to determine the validity and reliability of cognitive screening tools for detecting cognitive impairment in TBI populations. Research questions were:

1. How reliable are the screening tools used in TBI populations?

2. How valid are the screening tools used in TBI populations?

A secondary objective was to determine the comparative validity and reliability of identified screening tools between different severities (mild and moderate/severe) of TBI.

Materials and methods

Protocol and registration

This systematic review was written in accordance with PRISMA (Page et al., 2021; Appendix 1 in the Supplement). The protocol was registered on PROSPERO on 11/01/22, and amended on 27/07/22 with additional exclusion criteria (Registration number CRD42022297346 [available from: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID = 297346]).

Eligibility criteria

All eligible articles with an English language version, up to the search date (27/01/22) were included. During screening and extraction, eligibility criteria were clarified to account for unanticipated nuances in the literature.

Patient population

Studies were required to include human adults (aged 18–80) diagnosed with TBI of any severity or stage (mild, moderate, or severe; acute or post-acute). The TBI may have been sustained when the individual was <18 years old. TBI was defined as an injury to the brain caused by an external force. Mixed samples which included conditions other than TBI (including other acquired brain injuries [ABI] such as stroke) in the same analysis with people with TBI, were excluded. If the study included a mixed sample of children and adults, a separate analysis with adults 18–80 only was required. Studies with adult participants which did not specify an upper or lower age limit were included.

Index test

Studies were required to include a cognitive screening test or tool, using the definition outlined in Cullen et al. (2007); the test must have been designed to screen for cognitive impairment or be used for that purpose, have an administration time of less than 20 min and be available in English. Screens were required to be administered to patients and directly measure their cognitive performance. The screen could assess multiple domains or a single domain of cognitive function. Screening tests were excluded which were measures of, or were being used to measure: functional ability (including driving); malingering, effort, or performance validity; or consciousness, lower-order, subcortical or sensory functions (including basic visuo-ocular, vestibular, or auditory function). Where relevant, it was recorded whether a “Modified” (i.e., an adapted or non-standard version of the original tool) or non-English language version was utilized.

Comparator test

Studies which included a comparator test (i.e., any other screening test that may have been used in the study for comparison) were included, but this was not a requirement.

Outcomes of interest

A variety of psychometric outcomes were accepted depending on the study design. Accepted psychometric qualities and their measurement properties were consistent with those outlined in the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) guidelines (Mokkink et al., 2018). Examples of permitted psychometric qualities, designs, and statistics are found in Table 1; for full details of permitted outcomes please see the COSMIN Manual (pp. 28–29; Mokkink et al., 2018).

Table 1. Examples of permitted psychometric qualities, designs, and statistics.

Psychometric Design	Description	Examples of recorded outcomes
Structural Validity	The degree to which the scores of the cognitive screening tool are an adequate reflection of the dimensionality of the construct to be measured.	See COSMIN manual for full list of structural validity outcomes (pp. 28–29; Mokkink et al., 2018)
Internal Consistency	The degree of the interrelatedness among the items on the cognitive screening tool (per dimension).	Cronbach’s Alpha
Inter-rater Reliability	The extent to which scores for patients who have not changed are the same for repeated measurement under several conditions (across different examiners)	Intra-class correlations (ICC) or weighted Kappa
Test-retest Validity	The extent to which scores for patients who have not changed are the same for repeated measurement under several conditions (across two separate tests)	Intra-class correlations (ICC) or weighted Kappa
Criterion Validity (or Diagnostic Accuracy)	Ability to detect cognitive impairment (measured by a standard neuropsychological test and appropriate reference criteria, as defined by the authors) or traumatic brain injury diagnosis.	Correlation, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), area under the curve (AUC).
Convergent or Divergent Validity	Correlation with a measure expected to be related or divergent to the cognitive screening tool (such as a functional measure), where both measures are completed at the same time.	Correlation coefficient (such as Pearson’s R or Spearman’s Rho)
Discriminant Validity	Ability to distinguish between TBI and control groups, or between severity of TBI status (mild, moderate, severe).	Independent t-tests with effect size statistic.

Test-retest reliability was only recorded where the researchers deliberately administered the measure twice in close succession, within a baseline phase during which natural change is unlikely to have occurred (e.g., consecutive days within the first week in a rehabilitation ward, or during the baseline phase of an intervention study before the intervention was introduced). In addition to the above outcomes, we aimed to extract information on measurement error, cross-cultural validity, and responsiveness (only where a standard cognitive test was used as a comparator) in accordance with COSMIN (Mokkink et al., 2018). Outcomes related to clinical utility, feasibility or acceptability were recorded where available. These outcomes were all of equal interest and none were prioritized.

Study type

Included reports were full-length articles (in a peer-reviewed journal) reporting primary research (not systematic reviews or conference abstracts etc.). Eligible studies assessed the psychometric properties of a cognitive screening tool, in accordance with the outcomes of interest, and had a group design (i.e., an observational or experimental design, but not a single case study or case series design). Examples of excluded designs were:

• the screening tool is compared to a self-report measure or brain injury measure or across different groups; but is being used as an outcome measure, there is no clear psychometric relevance or it is not being validated as a screen

• a cognitive screening tool is used to track improvement in an intervention or recovery study (where no additional tool is used to compare the screen to a gold-standard or any additional relevant psychometrics are calculated)

• the tool is evaluated in conditions not ecologically valid for clinical use; for example outside of a clinic setting

• psychometrics are present but are focused on an aspect of the tool other than cognitive function (e.g., performance validity).

In addition to the above criteria, after full-text screening, additional exclusion criteria were applied to account for unanticipated nuances in the literature:

1. Sports-related concussion (SRC) criteria: studies with a SRC sample were excluded. These studies often used measures designed for use in SRC settings (i.e., at pitch-side, in the acute phase, integrating neurological exam elements). Results were often compared to “baseline” screening, conducted pre-morbidly. Given these factors, it was felt this area of literature was distinct, less applicable to clinical settings; and would increase the heterogeneity of included studies.

2. COSMIN criteria: COSMIN outlines that studies should be excluded if they “only use the PROM as an outcome measurement instrument … ” or “ … studies in which the PROM is used in a validation study of another instrument”, as screening would be “extremely time consuming” without this criterion (Mokkink et al., 2018, p. 20). Therefore, studies were excluded where the purpose of the study was not to validate the screen and/or where the screen is used as an outcome measure or to validate another tool. Studies which did not have a design consistent with COSMIN (e.g., using multiple regression) were excluded.

Information sources, search strategy, and study selection

Scoping searches were conducted to refine search terms. Key papers identified in scoping were noted. The sensitivity of the search strategy was evaluated by its ability to detect the key papers. Search terms included variants of keywords related to population (e.g., “Traumatic head injury”, “head injury”, “brain injury” or “concussion”), construct (e.g., “cognition”, “neuro psychological”, “executive function” or “memory”) type of instrument (e.g., “screening tool”, “screening assessment” “screening test” or “screening instrument”) and measurement properties (e.g., “psychometric”, “validity”, “reliability”, “sensitivity”). The full search strategy is outlined in Appendix 2 in the Supplement. No additional limits were used in search filters. Published literature was retrieved from MEDLINE, Web of Science Core Collection, EMBASE, CINAHL, and PsycINFO, from inception up until the date of extraction (27/01/22). Search results were managed in the first author’s RefWorks library (www.refworks.com/refworks2/). Duplicates were removed during database extraction. Information of interest was extracted into an Excel spreadsheet and decision making was recorded.

Titles and/or abstracts were screened by J.M. for eligibility. The second reviewer (A.F.) independently repeated this process for 100 records to check for consistency. For the second screening phase, the full text was read by J.M. for all records identified as “maybe eligible” at the title/abstract stage and a decision was made about its eligibility. A.F. independently repeated this process for 50 records to check for consistency.

The reference lists of eligible papers and relevant systematic reviews were then searched by J.M. by hand (backward citation). Subsequent papers which have cited eligible papers, identified electronically using the “cited by” function in Google Scholar, were searched (forward citation) by J.M. on 10/07/22. No additional studies or data were sought by contacting authors.

Data collection and data items

Three data extraction templates were created to extract relevant data from eligible studies. Data extracted included: patient characteristics (sample size, age, sex, TBI severity, time since injury; for both TBI and relevant control groups); cognitive screens (tool name, original reference, domains assessed, items, range of scores, time to administer etc.) and psychometric properties (including psychometric quality assessed, details of the study design and relevant measurement property; as defined in COSMIN). Additional information was gathered (including inclusion/exclusion criteria, TBI diagnosis criteria, and study setting) but was not reported for brevity. J.M. completed all data extraction and A.F. checked extraction for five of the papers. Only minor formatting and spelling errors were identified that would have been recognized by the primary author at the synthesizing stage.

Risk of bias

The COSMIN Risk of Bias (RoB) checklist was used by J.M. to assess the quality of all included studies (Mokkink et al., 2021). It is a consensus-based checklist for evaluating the methodological quality of psychometric studies. Risk of bias was rated as: “very good” (V), “adequate” (A), “doubtful” (D), “inadequate” (I) or “not applicable” (N) according to COSMIN criteria. A “worst counts score” approach was used for overall rating. Additional details are given in Appendix 3 in the Supplement. It should be noted that the COSMIN (Mokkink et al., 2018) was developed for evaluation of patient-reported outcome measures (PROMS); however, it can be utilized for performance-based measures (for example, De Martino et al., 2020). Risk of bias (for any relevant measurement properties) was assessed independently for five studies by A.F.

The Quality Assessment of Diagnostic Accuracy Studies (QUADAS 2) tool was used to assess the quality, only for studies which specifically claimed to have a “diagnostic accuracy” design (or similar) (Whiting et al., 2011). It includes items which assess both the risk of bias and applicability of results. J.M. and A.F. independently rated the quality of three such studies.

Summary measures and synthesis of results

The evidence was graded using the COSMIN checklist for good measurement properties by J.M. Measurement properties were rated as (+) sufficient; (-) insufficient; or (?) indeterminate, using criteria defined in COSMIN, which varied for each type of design (for example, criterion validity designs should report an AUC ≥0.7 for a sufficient (+) rating). A narrative synthesis was performed, with data presented in text, tables, and figures. Each study was summarized including a description of the sample characteristics. Details of cognitive screening tool(s) used were reported. The type(s) of validity and reliability, relevant details of the study design, and the resulting psychometric statistic(s) were reported. Measurement properties of the tools in TBI populations were discussed, considering the quality of the studies. The review employed the COSMIN-recommended Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach to synthesize the evidence across studies. GRADE ratings were determined by A.F. and ambiguities were resolved by discussion with the other authors. Please see Appendix 3 in the Supplement for more detail on how this review adapted the COSMIN process.

Results

The initial title-abstract calibration between J.M. and A.F. resulted in “fair” agreement (81.2% agreement: Cohen’s Kappa = 0.29). After ambiguities within the criteria were clarified, 100% agreement was reached. Independent full-text screening resulted in 92.0% agreement between reviewers (Cohen’s Kappa = 0.73, “substantial agreement”). Remaining disagreements were resolved by consensus between all three authors.

Eighty studies were deemed eligible for inclusion via the above screening process (see PRISMA flow chart in Figure 1). Several studies which might have been expected to be included were found to be ineligible, including Gaber (2008) and Hazan et al. (2017), whose inclusion criteria did not meet the age criteria.

Figure 1. PRISMA flowchart displaying excluded and included studies at each stage of the review process.

PRISMA flow chart showing that there were 2135 records screened at title and abstract stage, 410 assessed at full text stage, and 34 studies included in the review.

After applying the additional COSMIN and SRC criteria, and with the addition of six papers from citation searching, 34 publications were eligible for inclusion, in which 22 cognitive screening tools were evaluated. Within these 34 publications there were a greater number of individual studies (i.e., studies of each individual psychometric property per each tool): 4 structural validity, 7 internal consistency, 3 inter-rater reliability, 3 test-retest reliability, 32 criterion validity, 14 convergent or divergent validity, and 36 discriminant validity studies. Sample characteristics and cognitive screens are outlined in Tables 2 and 3, respectively. Demographic, clinical, and screening tool metrics sought by this review were not consistently reported, and missing data indicates this information was not available. Information not reported was sought from referenced sources within the paper where available. The studies were heterogeneous in terms of TBI severity (13 mild or “concussion”; 6 moderate-severe; 11 mixed; and 4 unclear) and time since injury (range of mean durations: 5.6 h to 60.33 months).

Table 2. Sample characteristics.

ID	Reference	TBI group (n)	TBI mean age (years) (SD)	TBI % male	TBI severity (n)	TBI time since injury M (SD)	Control group (n) (if present)	Controls mean age (years) (SD)	Controls % male
1	(Adjorlolo, 2018)	50	36.28 (7.20)	70.00%	Moderate	8 months (4.58)	50	27.98 (6.75)	58.00%
2	(Adjorlolo, 2016)	50	36.28 (7.20)	70.00%	Moderate	8 months (4.58)	50	27.98 (6.75)	58.00%
3	(Borgaro & Prigatano, 2002)	42	Moderate = 34.09 (14.80); Severe = 38.05 (14.72)	Moderate 77.3%; severe 66.7%	Moderate (22); severe (20)	Moderate = 6.05 days (11.91) severe = 20.43 days (11.68)	21	37.67 (17.22)	76.20%
4	(Borgaro et al., 2003)	65	38.8 (16.02)	75.40%	Mild (20), moderate (12); severe (33)	23.02 days (18.31)	25	38.63 (16.64)	59.30%
5	(Cheng et al., 2021)	86	51.651 (16.487)	70.90%	Mild (78) moderate (4) severe (4)	Not reported	40	53.250 (13.897)	70.00%
6	(Cheng et al., 2022)	50	51.92 (19.070)	78%	Mild (43); Moderate (3) Severe (4)	8 days (7–11.25) (median, IQR)	32	50.88 (17.716)	68.80%
7	(Cole et al., 2018)	59^a	27.0 (6.0)	95.40%	Mild	4.97 days (1.76)	68^a	34.4 (7.9)	79%
8	(Doninger et al., 2006)	120	37.5 (12.6)	82%	Mild (30.6%), moderate (24.6%) severe (44.8%)	29 days (29)
9	(Ettenhofer et al., 2021)	191	35.04 (8.05)	93.2%	Mild	27.12 months (8.28–85.81) (median, IQR)
10	(Fischer et al., 2016)	11	33 (16.5)	72.72%	Mild	5.9 h (5.2)	12 HC; 7 O	O = 31 (11.6); HC = 33 (15.0)	O = 100%; HC = 75%
11	(Frenette et al., 2019)	134; 92 “complicated mTBI”, 42 uncomplicated	Uncomplicated: 47.5 (26.0–59.0) Complicated: 58.5 (35.5–76.5)	Uncomplicated 66.67%; Complicated 60.87%	Mild	“Within the first 2 weeks following the injury”	25	21.92 (5.26)	48%
12	(Gupta & Kumar, 2009)	30	33.5 (10.7)	83.30%	Moderate-severe	27.6 months (33.8)	55	26.53 (8.48).	47.27%
13	(Hatta et al., 2012)	42	30.7 (11.0)	80.95%%	“Mostly mild TBI”	7.7 months (2.6)	32	50.88 (17.716)	68.80%
14	(Joseph et al., 2019)	100	43.47 (16.43);	Overall 65%; mild 65%; moderate/severe 65%	Mild (60) moderate-severe (40)	386.14 days (454.49)
15	(Tay et al., 2019)	61 (50 outpatient; 11 inpatient)	52.6 [17.4]	80.30%	Moderate-severe TBI	17.5 months (18.0)
16	(Nabors et al., 1997)	45^a	39.5 (15.7)	78.00%	Mild (21); moderate (8); severe (13); unknown (2)	34.7 days (25.3)
17	(Pinasco et al., 2023)	28	36.59 (13.76)	71.43%	Mild (3); moderate-severe (25)	60.33 months (61.02)
18	(Rojas & Bennett, 1995)	25	27.44 (7.11)	52%	Mild TBI
19	(Silverberg et al., 2014)	26^a	36.6 (12.2)	73.10%	Mild TBI		33	42.8 (12.2)	51.5%
20	(Srivastava et al., 2006)	43^a	66 (7.2)	37.20%	Mild (28); moderate (15)	“one year post-TBI”
21	(Vissoci et al., 2019)	192	33.87 (13.32)	82.80%	Mild(175); Moderate-severe (17)
22	(Waldron-Perrine et al., 2019)	117^b	31.33 (8.48)	95.7% (total sample)	Mild TBI only
23	(Walsh et al., 2016)	100	26.31 (5.83)	87%	mTBI only	≤ 72 h after injury	100	26.31 (5.83)	79%
24	(Wu et al., 2023)	98	51.17 (16.81)	75.51%	Moderate-severe	20.23 days (5.16)	30	46.77 (16.13)	60%
25	(Zhang et al., 2016)	103	35.9 (13.08)^c	84.47%		In the last “1–12 months”	42	31 (11.3)	80.95%%
26	(Zhang et al., 2021)	42	47.00 (15.814)	76.20%		Not reported	30	44.07 (11.435)	60.00%
27	(Maruff et al., 2009)	50	43.2 (5.6)	80%	mTBI only	72 days (14)	50	44.7 (4.9)	80%
28	(Wong et al., 2013)	48	50 (16)	73%	Median (IQR) GCS score: 14 (11–15).	3–5 years	40
29	(Luoto et al., 2014)	44	37.5 (12.1)	67.30%	mTBI only	38.4 h (30.3)	33	42.8 (12.2)	51.50%
30	(Coldren et al., 2010)	71	26.5	96%	“Concussive event” (moderate/severe excluded)	>12 h after the event	102 HC and 64 I	27.3	88%
31	(Doninger et al., 2000)	186	34	75%		32 months (Median)
32	(Stone et al., 2015)	84	36.1 (13)	76%	“Concussion”	23.6 hrs (14)	30	36.9 (13)	76.7
33	(Bin Zahid et al., 2018)	118 (87 [-CT]; 31 [ + CT])	41.09 (11.37) (-CT); 37.86 (13.91) (+CT)	76% (-CT); 81% (+CT)		0–5 days	98 HC; 46 O	HC = 32.14 (11.26); O = 35.72 (10.58)	HC  = 49%; O = 80%
34	(Cicerone & Azulay, 2002)	32	39.4 (9.6)	43.75%	Mild/“concussion”	14.3 months (9.5)	32	37.2 (9.7)	43.75%

Abbreviations: TBI, Traumatic Brain Injury; mTBI, mild traumatic brain injury; CT+, TBI with “positive”: CT scan; CT-, TBI with “negative” CT scan; HC, health controls; O, orthopedic injury; I, injured controls.

^a Maximum sample size (not all participants completed all screening tests).

^b This total sample includes patients who were then confirmed NOT to have a diagnosis of TBI as part of this study; sample size of confirmed TBI and non-TBI group not reported.

^c Age at “onset” of TBI reported.

Table 3. Screening tools.

Screening Tool	Domains	Items	Range of Scores	Time to administer	Included studies which evaluate the screen (study ID)
2 & 7 Selective Attention Test (“2&7”)	Selective Attention	20 trials		5 min	34
Axon Sport's CogState Sport (CogState)	Psychomotor function, information processing, decision making, working memory, and visual memory (recognition)	4		?	7, 27
Barrow Neurological Institute Screen for Higher Cerebral Functions (BNIS)	Language functions, orientation, memory, concentration/attention, visual spatial problem-solving skills, affect expression and perception, and awareness			0–25 minutes^a	3, 4
Bethesda Eye & Attention Measure (BEAM)	Visual attention			10 min	9
Cognistat (Neurobeh-avioral Cognitive Status Examination; NCSE)	Orientation, attention, language, constructional praxis, memory, calculations, and verbal reasoning. Level of conscious also reported	51	Average, mildly impaired, moderately impaired, or severely impaired	10–20 min	8, 12, 16, 31
Continuous Performance Test of Attention (CPTA)	Attention	450 consecutive stimuli across 5 conditions	Error score 0–450	10 min	34
COntrolled Word Association Test (COWAT)	Verbal fluency	3		3 min	1, 2
Digit cancellation test (D-CAT)	Focused attention, sustained attention or concentration, and selective attention			5 min	13
Digit Span	Attention	32 (16 for each domain)	0–32	1–3 min	34
Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT)	Processing speed, reaction time, visual memory, and verbal memory			20 min	7
INECO Frontal Screening (IFS)	Executive function (motor programming, conflicting instructions, motor inhibitory control, verbal and visual working memory, verbal abstraction ability, and inhibitory control)		0–30	10 min	17
King-Devick Test (K-D)	Processing speed and rapid gaze shifting			∼2 min	10, 19, 23
Mini-Mental State Examination (MMSE)	Orientation, memory, calculation and attention, naming, language, and visuospatial function	11	0–30	5–10 mins	5, 6, 20, 21, 24, 25, 26, 28
Montreal Cognitive Assessment (MoCA)	Visuospatial and executive function, naming, memory, attention, language, abstraction, orientation	30	0–30	∼10 mins	6, 11, 15, 21, 22, 24, 25, 26, 28
Paced Auditory Serial Addition Test (PASAT)	Attention	60 trials	0–60	10–15 min	34
Revised Quick Cognitive Screening Test (RQCST)	Orientation, Attention/concentration (verbal), Attention/concentration (visual), Memory: Immediate recall (verbal), Spatial neglect, Arithmetic, Constructional praxis, Memory: Immediate recall (visual), Vocabulary, Naming, Abstract reasoning: Similarities, Abstract reasoning: Analogies, Unusual views, Spatial orientation, Memory: Delayed recall (visual), Memory: Delayed recall and Memory: Working memory	48		10–15 mins	1, 2
Rey’s Tangled Lines Test (RTLT)	Processing speed			< 5 min	14
Rowland Universal Dementia Assessment Scale (RUDAS)	Memory, visual spatial orientation, visual structure painting, practical imitation, judgment, language	6	0–30	<10 min	5, 6
Saint Louis University Mental Status examination (SLUMS)	Orientation, memory, calculation, attention, language, visuospatial and executive function	11	0–30	5–10 min	24, 26
Standard Assessment of Concussion (SAC)^b	Orientation, immediate memory, concentration, and delayed recall		0–30	5 min	10, 19, 29, 30, 32, 33
King-Devick Test (K-D)	Processing speed and rapid gaze shifting			∼2 min
Stroop Colour Word Test (SCWT) / Stroop Neuropsychological Screening Test (SNST)	Selective attention and response inhibition			4 min	1, 2, 18, 34
Trail Making Test (TMT)	Speed of processing, sequence alternation, cognitive flexibility, visual search, motor performance, complex attention	2			1, 2, 34

^a Estimates vary across the literature; as some are less than 20 min, this measure was included.

^b The Standard Assessment of Concussion (SAC) was administered as part of the Military Assessment of Concussion (MACE) (French et al., 2008) or the Sport Concussion Assessment Tool (SCAT) (Guskiewicz et al., 2013); it was only reported if it was analysed separately (as both of these tools contain ineligible elements).

Of the 22 tools, administration time ranged from 2 to 20 min (where reported). Some measures were designed as screening tools e.g., MoCA; others were designed as brief tests and were being evaluated as a cognitive screen e.g., Controlled Oral Word Association Test (COWAT). Some evaluated multiple cognitive domains e.g., MMSE; others focused on one area of cognitive function e.g., Bethesda Eye and Attention Measure (BEAM).

Risk of bias

COSMIN RoB ratings are reported within the detailed psychometric results tables which can be found in the Supplement (Tables S1-S3). Initial agreement in RoB ratings between independent reviewers was low (<50%). Key areas of ambiguity were clarified, and after independent rating for an additional three papers, 100% agreement was reached.

While RoB relating to structural validity, criterion validity, and internal consistency was generally rated as “very good”, issues were raised within reliability and discriminant validity designs. Of note, many studies did not report an effect size or correlation statistic for their discriminant validity analyses, giving an inadequate understanding of the magnitude of difference between groups. Other issues included: limited information of diagnosis procedures; the time delay between reliability measurements; and poor reporting of statistical information.

Independent rating using the QUADAS 2 tool resulted in 76% agreement between the reviewers, and differences in opinion were discussed until consensus was reached. The results (Table 4) revealed that all diagnostic accuracy studies had a source of potential bias (diagnostic accuracy for either TBI or cognitive impairment). In most studies, examiners were not blinded to the results of the patient status, participants were recruited into pre-determined groups based on diagnosis status and exploratory cut-offs were used in ROC curve analyses; all of which are potential sources of bias, as outlined in QUADAS 2.

Table 4. QUADAS 2 Risk of Bias and Applicability Ratings (Diagnostic Test Accuracy Studies only).

Study ID	Risk of Bias				Applicability
	Patient selection	Index test	Reference standard	Flow and timing	Patient selection	Index test	Reference standard
1	HIGH	HIGH	LOW	UNCLEAR	LOW	LOW	LOW
5	HIGH	HIGH	UNCLEAR	UNCLEAR	UNCLEAR	LOW	LOW
6	HIGH	HIGH	UNCLEAR	UNCLEAR	LOW	LOW	LOW
15	UNCLEAR	HIGH	UNCLEAR	LOW	LOW	LOW	UNCLEAR
19	HIGH	HIGH	LOW	HIGH	LOW	UNCLEAR	LOW
24	HIGH	HIGH	UNCLEAR	UNCLEAR	LOW	LOW	LOW
28	HIGH	HIGH	UNCLEAR	UNCLEAR	LOW	LOW	LOW
29	HIGH	HIGH	LOW	UNCLEAR	UNCLEAR	LOW	LOW
32	LOW	HIGH	LOW	LOW	LOW	LOW	LOW
34	HIGH	HIGH	LOW	UNCLEAR	UNCLEAR	LOW	UNCLEAR

Note: “HIGH” = risk of bias; “LOW” = low/no identified risk of bias; “UNCLEAR” when insufficient data to make a judgement (Whiting et al., 2011).

Measurement properties

Detailed psychometric properties of the tools are reported in Tables S1–S3 in the Supplement.

No studies measured cross-cultural validity, measurement invariance, or measurement error or responsiveness (in a way that was suggested in the COSMIN guidelines and which could be evaluated using their criteria). Consequently, the psychometric properties that were able to be assessed for the tools reviewed were structural validity, internal consistency, inter-rater reliability, test-retest reliability, and convergent or discriminant hypothesis testing. There was significant heterogeneity across studies in types of psychometric qualities assessed and subsequent outcome statistic. Discriminant validity was the most common type of validity assessed (36: results per study, per tool), followed by criterion validity (37 results), convergent/divergent validity (14 results), internal consistency (7 results), structural validity (4 results) and reliability (3 results). Ten studies also specified that they had a “diagnostic accuracy” design (or similar) (Table 4).

After consideration of risk of bias, attention can be paid to the measurement properties of sufficient (+), insufficient (-) or indeterminate (?). It should be noted that tools varied in language version, which may impact on the confidence in results. Given the heterogeneity of the literature and the poor quality of reporting (e.g., in discriminant validity studies) it was not possible to evaluate the psychometrics of tools across severities. As per the recommended COSMIN process, overall ratings were produced for each assessed property for each instrument. These ratings, together with GRADE ratings of the quality of the evidence used to form these ratings, can be found in Tables 5 and 6. GRADE ratings were not produced when the overall rating was indeterminate or inconsistent, or in cases where a firm rating was given but sample size information was lacking (as this prevents evaluation of the GRADE imprecision criterion).

Table 5. Overall property rating and GRADE rating of quality of evidence for structural validity, internal consistency, and reliability.

Instrument	Structural validity		Internal consistency		Reliability (Inter-rater)		Reliability (Test-Retest)
	Overall rating	Quality of evidence	Overall rating	Quality of evidence	Overall rating	Quality of evidence	Overall rating	Quality of evidence
Cognistat	?	Ungraded^a	-	Low	+	Low	-	Very Low
MMSE	+	High	-	High
MoCA	+	Moderate	+	High
RQCST			-	High
RUDAS			+	Moderate	+	Moderate	+	Moderate
SLUMS			+	Low	+	Low	-	Very Low

Notes: Overall property rating: sufficient (+), insufficient (-), inconsistent (+/-) or indeterminate (?).

Abbreviations: MMSE, Mini Mental State Exam; MoCA, Montreal Cognitive Assessment; RQCST, Revised Quick Cognitive Screening Test; RUDAS, Rowland Universal Dementia Assessment Scale; SLUMS, Saint Louis University Mental Status Examination.

^a Where results are rated as inconsistent or indeterminate, no quality level for evidence is to be given as per COSMIN guidelines.

Table 6. Overall property rating and GRADE rating of quality of evidence for criterion validity and hypothesis testing.

Instrument	Criterion Validity		Hypothesis Testing (Convergent unless specified)		Hypothesis Testing (Discriminant)
	Overall rating	Quality of evidence	Overall rating	Quality of evidence	Overall rating	Quality of evidence
2&7	?	Ungraded^a
AxonSport CogState	–	Moderate			–	Low
BNIS					+	Very Low
BNIS (orientation)					?	Ungraded^a
BEAM	–	Ungraded^b
Cognistat	?	Ungraded^a	+/-	Ungraded^b	+	Very Low
COWAT	+	High	-	Moderate
CPTA	?	Ungraded^a
DCAT					+	Very Low
Digit Span	?	Ungraded^a
ImPACT	-	Moderate
IFS	+	Moderate			+	Very Low
K-D	-	Ungraded^b			-	Moderate
MMSE	+	Moderate	+/-	Ungraded^a	+	Moderate
MOCA	+	High	+	High	+	Moderate
PASAT	?	Ungraded^a
RQCST	+	High	+	High
RTLT					-	Very Low
RUDAS	+	High	+	High	+	Moderate
SLUMS	+	High	+	High	+	Low
SAC	+/-	Ungraded^a	-^c	Very Low	+	High
SCWT	+	High	-	Ungraded^b	+	High
SNST					+	Very Low
TMT	+	High	-	Moderate	+	High

Notes: Overall property rating: sufficient (+), insufficient (-), inconsistent (+/-) or indeterminate (?).

Abbreviations: 2&7, 2 & 7 Selective Attention Test; AxonSport Cogstate, Axon Sport's CogState Sport; BNIS, Barrow Neurological Institute Screen for Higher Cerebral Functions; BEAM, Bethesda Eye & Attention Measure; Cognistat, Neurobehavioral Cognitive Status Examination (NCSE); COWAT, Controlled Oral Word Association Test; CPTA, Continuous Performance Test of Attention; D-CAT, Digit Cancellation Test; ImPACT, Immediate Post-Concussion Assessment and Cognitive Testing; IFS, INECO Frontal Screening; K-D, King-Devick Test MMSE, Mini-Mental State Examination; MoCA, Montreal Cognitive Assessment; PASAT, Paced Auditory Serial Addition Test; RQCST, Revised Quick Cognitive Screening Test; RTLT, Rey Tangled Lines Test; RUDAS, Rowland Universal Dementia Assessment Scale; SAC, Standard Assessment of Concussion; SCWT, Stroop Colour Word Test; SLUMS, Saint Louis University Mental Status Examination; SNST, Stroop Neuropsychological Screening Test; TMT, Trail Making Test.

^a Where results are rated as inconsistent or indeterminate, no quality level for evidence is to be given as per COSMIN guidelines.

^b Denotes situations where sample size was not provided, meaning that the imprecision GRADE criterion could not be rated.

^c Hypothesis testing was divergent rather than convergent.

Structural validity

The MMSE and MoCA (Swahili versions) both demonstrated sufficient structural validity in a TBI sample with high and moderate quality of evidence respectively. Cognistat structural validity was rated as indeterminate due to insufficient outcomes reporting from the Rasch analyses.

Internal consistency

Sufficient internal consistency properties were established for the MoCA (based upon high-quality evidence), RUDAS (Rowland Universal Dementia Assessment Scale, based on moderate quality evidence), and SLUMS (Saint Louis University Mental Status Examination, based on low-quality evidence). Insufficient internal consistency in particular subtests lead to insufficient property ratings for the Cognistat (based on low-quality evidence) and RQCST (Revised Quick Cognitive Screening Test, with high-quality evidence). One high-quality study was extracted, which found internal consistency of the MMSE to be insufficient.

Reliability

The inter-rater reliability property was found sufficient for three measures, the Cognistat (based on low-quality evidence), RUDAS (moderate quality evidence), and SLUMS (based on low-quality evidence). The Cognistat received an insufficient rating for test-retest reliability (based on very low-quality evidence), the RUDAS a sufficient rating (based on moderate quality evidence) and the SLUMS an insufficient rating (based on very low-quality evidence).

Criterion validity

Four studies found the MoCA had sufficient criterion validity to distinguish between TBI patients and controls. An additional study found the MoCA had sufficient criterion validity for those who were impaired/not impaired on the RBANS (Repeatable Battery for the Assessment of Neuropsychological Status). Similarly, four studies found the MMSE had sufficient criterion validity to distinguish between TBI patients and controls. However, one study found varying sensitivity and specificity values for MMSE domains to predict impairment on corresponding standard neuropsychological tests; many of these fell below the 80% standard typically suggested. A number of measures all had initial promising evidence of their criterion validity, however were only supported by 1–2 studies. These were the COWAT (high-quality evidence), IFS (INECO Frontal Screen, moderate quality evidence), RQCST (high-quality evidence), RUDAS (high-quality evidence), SLUMS (high-quality evidence), SCWT (Stroop Colour-Word Test, high-quality evidence), and TMT (Trail Making Test, high-quality evidence). Insufficient overall ratings were awarded to the ImPACT (Immediate Post-Concussion Assessment and Cognitive Testing) based on moderate quality evidence, as was the case for the CogState (moderate quality evidence), together with the BEAM and King-Devick tests (which were not given GRADE quality ratings due to lacking sample size information). The SAC overall rating was inconsistent and the overall rating was indeterminate for the 2&7, Cognistat, CPTA, Digit Span, and PASAT.

Hypothesis testing: convergent validity

Given the heterogeneity of the measures used in convergent validity analyses it is difficult to draw conclusions across the studies. Cognitive screening tools, many of which have limited evidence of established validity in TBI samples, are mainly used as comparators. The two studies involving the MMSE showed different property ratings, producing an overall rating of inconsistent. Cognistat also received an overall rating of inconsistent because the one study which evaluated convergent validity found inconsistent results across different sub-test comparisons. Insufficient property ratings were awarded to the SCWT (no GRADE rating due to non-reporting of sample size) and the TMT and COWAT (based on moderate quality evidence), as well as the SAC (based on very low-quality evidence). However, the MoCA, RQCST, RUDAS, and SLUMS were each awarded a sufficient property rating, based on high-quality evidence. The MoCA was found to be moderately positively correlated with functional ability measures and the RQCST was found to be correlated with activities of daily living and quality of life measures; the RUDAS and SLUMS were each positively correlated with other cognitive screen scores (MMSE and MoCA).

Hypothesis testing: discriminant validity

There was a lack of clear hypotheses and poor effect size reporting across the discriminant validity studies; however, by estimating effect sizes and applying hypotheses (outlined in the method section) this allowed estimation of measurement properties across a wider range of studies. In general, most tools evaluated this way were awarded ratings of sufficient discriminant validity between TBI and control groups although the GRADE ratings of evidence were variable. These tools were: BNIS, Cognistat, DCAT, IFS, and SNST (very low-quality evidence); SLUMS (low-quality evidence); MMSE, MoCA and RUDAS (moderate quality evidence); and SAC, SCWT and TMT (high-quality evidence). Three tools were awarded insufficient ratings: CogState (low-quality evidence), K-D (moderate quality evidence) and RTLT (very low-quality evidence). The BNIS orientation subscale was assessed in a separate study but the rating was indeterminate (very low-quality evidence).

Discussion

This systematic review found a substantial literature on the psychometric properties of cognitive screening tools in TBI populations. The MoCA, in particular, demonstrated sufficient internal consistency and discriminant, convergent, structural, and criterion validity. The MMSE demonstrated sufficient structural, discriminant, and criterion validity. In contrast, convergent validity for the MMSE varied depending on the standard neuropsychological test comparator and there was evidence to suggest the internal consistency of the measure was insufficient. Tools such as RUDAS, SLUMS, SCWT, and RQCST demonstrated some promising indications of validity. The properties of other tools were generally inconsistent, insufficient, or indeterminate; and limited to a small number of studies. These findings have implications for the refinement of clinical guidelines and should inform clinical practice. With the current evidence base, the MoCA can be tentatively considered to be the most well-validated tool for TBI populations based on sufficient ratings for all assessed properties with quality of evidence being moderate or high for all ratings. Notable limitations are that there were no studies, identified through this review, which investigated its reliability or responsiveness. These gaps are particularly important, given the implications for detecting change over time in clinical and research contexts. The MMSE, though widely used and evaluated in moderate to high-quality studies, warrants some caution, given that it demonstrated sufficient validity in only some respects and there was high-quality evidence that its internal consistency was insufficient.

These findings should be considered within the context of the heterogeneity and poor reporting across studies in terms of key sample characteristics (TBI severity and time since injury). Many studies assessed the psychometric properties in acute or mild TBI settings, and these findings are unlikely to be generalizable to other clinical contexts, such as post-acute rehabilitation settings (Barman et al., 2016). Poor statistical reporting, poor reporting of TBI diagnosis methods, and time delays between reliability measurements were some of the risk of bias issues identified.

These findings are broadly consistent with the findings in ABI (including stroke) populations, which find that tools which include items that reflect a variety of cognitive domains, particularly those which assess executive function, such as the MoCA, are typically the most well-validated (Kosgallana et al., 2019; Stolwyk et al., 2014). TBI often results in multi-domain cognitive difficulties, including attention, memory, and executive function, which may explain this finding (Barman et al., 2016). It is not apparent why the MMSE did not demonstrate adequate internal consistency. As it is more heavily weighted towards language, attention, and memory, its items may comparatively neglect domains more relevant to TBI than to dementia samples. It is notable that other multi-domain tools such as the ACE-III (which is referenced in the SABIN guidelines) or Oxford Cognitive Screen (OCS) (which has been identified to have good criterion validity in stroke samples) have not been validated in TBI (Demeyere et al., 2015; Hsieh et al., 2013; Kosgallana et al., 2019; Scottish Acquired Brain Injury Network, 2017). Notably, many of the screening tools investigated, including the MoCA and MMSE, were originally designed for detecting cognitive impairment associated with dementia and hence tend to feature fewer executive function items (Folstein et al., 1975; Nasreddine et al., 2005). Many cognitive screens evaluated in this review assessed only one domain (for example; the BEAM, which measures visual attention) or did not measure key domains impacted in TBI (for example the COWAT does not measure key aspects of memory function). Notably, the MMSE does not have an executive function subscale, but did show promising psychometric properties for use as a cognitive screen in this population in this review. It would be interesting to evaluate whether its psychometric properties may be improved with a supplementary executive function measure. An alternative approach may be to design a new, bespoke measure designed for use in TBI populations, which is tailored to capture the executive difficulties which are common in these patients (Barman et al., 2016).

A possible limitation is the exclusion of SRC samples, as those with SRC do present to clinical settings. However, the SRC literature had key distinctive features (noted in the methods section) which may have increased the heterogeneity of the studies further. It may therefore be beneficial to conduct a separate review on the SRC literature. Using COSMIN criteria may have excluded studies which contained relevant psychometric information. However, it would have been impractical to screen every study which inadvertently measured a tool’s validity. It also would not have been feasible or meaningful to attempt to rate the measurement properties of studies which used unusual designs or statistics which did not allow for comparison with COSMIN. Applying age criteria resulted in several studies being excluded. However, this decision was made as there are differences in the nature and assessment of TBI in adolescents or older adults, due to the interaction of developmental factors (Christensen et al., 2021; Peters & Gardner, 2018). As COSMIN was not designed for use with performance-based measures, the criteria for rating of test-retest reliability designs does not consider the possible risk of practice effects, for example if a cognitive screen is administered in short succession, without an alternate version. On balance, this review chose to include test–retest reliability data as this is an important indicator of reliability; however, we acknowledge practice effects may be a significant potential source of bias in the test–retest reliability designs. A further limitation is that in the absence of structural validity data, tools were assumed to be unidimensional. However, for tools which may in fact be multi-dimensional, internal consistency across domains may not be relevant to consider. Moreover, some tools can be considered to assess multiple cognitive domains (e.g., MoCA), yet statistically be considered unidimensional. A strength of this review is the use of two rigorous quality assessment tools, the COSMIN and the QUADAS 2, which contain detailed considerations on the assessment of psychometric methodology, and which have been specifically designed for use in these types of studies. However, we acknowledge the methodological limitation that we did not use independent double-rating at all stages of the review, from screening to data extraction and quality rating. A second reviewer independently evaluated a pre-defined number of records at each stage according to the pre-registered protocol, and agreement between raters was carefully monitored and revised to achieve concordance, but our results may not be fully robust given the absence of gold-standard full double-rating (although also recognizing that full double-rating is not always practical and does not preclude the production of a high-quality systematic review; Siddaway et al., 2019).

In conclusion, this review found that the MoCA had the most evidence for good psychometric qualities in TBI populations. However, its validity across key TBI characteristics requires further clarification and additional psychometric qualities including reliability and measurement error need to be evaluated. Tools such as RUDAS, SLUMS, and RQCST show promise but require additional investigation. The MMSE was found to have insufficient evidence for internal consistency. However, if future structural validity designs find that the MMSE may have multi-dimensional properties, then the finding of poor internal consistency may be less notable. Future research should clarify the validity of tools identified in this review, and additional multi-domain tools which have been validated in similar populations or which are used in clinical practice, particularly the ACE-III and OCS. Psychometric studies should take consideration of COSMIN and QUADAS 2 guidelines in their design, to allow for meaningful comparison across tools.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.