An overview of outcome measures used in neuropsychological rehabilitation research on adults with acquired brain injury

ABSTRACT

Outcome measurement is the cornerstone of evidence-based health care including neuropsychological rehabilitation. A complicating factor for outcome measurement in neuropsychological rehabilitation is the enormous number of measures available and the lack of a standard set of outcome measures. As a first step towards such a set, we reviewed intervention evaluation studies of the last 20 years to get an overview of instruments used for measuring outcome. The instruments were divided into two main categories: neuropsychological tests (International Classification of Functioning (ICF) level of functions) and other instruments (all other ICF domains). We considered the most common cognitive domains: memory, attention, executive functions, neglect, perception, apraxia, language/communication and awareness. Instruments used most for measuring outcome were neuropsychological tests (n = 215) in the domains of working memory, reaction times, neglect and aphasia. In the second category (n = 166) the multi-domain instruments were most represented. Several steps can be taken to select a standard set of outcome measures for future use. Next to evaluation of quality and feasibility of the instruments, expert opinion and consensus procedures can be applied.

KEYWORDS:

• Acquired brain injury
• Outcome measures
• ICF
• Neuropsychological
• Rehabilitation

Introduction

Measuring the outcome of health care is “a central component of determining therapeutic effectiveness and, therefore, the provision of evidence-based healthcare” (van der Putten, Hobart, Freeman, & Thompson, 1999, pp. 480–484). This is not different for neuropsychological rehabilitation. In clinical practice it is common to test and measure extensively as part of the diagnostic assessment (i.e., neuropsychological assessment). However, neuropsychological assessment is less often used to measure post-treatment outcomes (Casaletto & Heaton, 2017). This is not surprising given the traditional focus of (clinical) neuropsychology on diagnostics. Also, neuropsychological assessment has a longstanding tradition in deficit measurement for which many instruments with excellent psychometric properties are available (i.e., Lezak, Howieson, Bigler, & Tranel, 2012). The focus on other domains besides neuropsychological impairments (i.e., activities, participation, and quality of life) has evolved later and therefore less validated instruments are available for use in clinical neuropsychology. Since the aim of neuropsychological rehabilitation is much broader than reducing cognitive deficits but rather to enable patients to live with, manage, by-pass, or come to terms with cognitive deficits precipitated by injury to the brain (Wilson, 1997) other outcome measures, in addition to neuropsychological tests are needed.

There are many reasons why outcome measurement is important. Tate (2014) mentions the following reasons: “to document improvement or deterioration, for differential diagnosis, to evaluate treatment effectiveness, to identify areas of need, plan treatments, help people make practical decisions, and educate people with traumatic brain injury (TBI) and their family professionals and other professionals” (Tate, 2014, pp. 163–164). Others have also mentioned quality assurance and accountability purposes (Haigh et al., 2001). Outcome measurement can have different goals for various stakeholders. For (fundamental) researchers and clinicians efficacy is most relevant: does this treatment work, why and how and for whom? For applied researchers, clinicians but also for the patients and their family effectiveness is more important: Is the treatment beneficial/helpful for the patient group or for the individual patient? And finally, one can also consider efficiency: does the treatment work as expected and what is the cost–benefit ratio? In current health care, quality, cost-effectiveness and patient-centricity are key elements leading to many different systems and ways of measuring outcome. Along this line, the National Institute of Health (NIH) stimulates the development of Common Data Elements (CDEs) in neuroscience to improve research quality and the ability to transfer information between centres and allows for comparison and meta-analyses (Van Heugten, 2017). Just to name a few recent developments related to CDEs: in the field of neurology NEURO-QOL was developed which is a brief, reliable, valid, standardized quality of life assessment which can be applied across different neurological diseases (Cella et al., 2012), and the NIH toolbox for assessment of neurological and behavioural function is now available consisting of a standard set of concise, well-validated instruments (Gershon et al., 2012). Recently, an international standard set of Patient Centered Outcome Measures (PCOMs) after stroke was proposed to enable the assessment of healthcare value and stimulate value-based health care in which patients and costs are brought together (Salinas et al., 2016).

Over the years, recommendations on common outcome measures in traumatic brain injury research have been put forward, such as by Wilde and colleagues in 2010 (Wilde et al., 2010) and Tate and colleagues in 2013 (Tate, Godbee, & Sigmundsdottir, 2013). Recently, recommendations were published on outcome instruments for use in psychosocial research on patients with moderate-to-severe traumatic brain injury (TBI) (Honan et al., 2017). In total 56 instruments were recommended through nomination, literature search and expert opinion. These instruments represent all levels of the International Classification of Functioning (ICF): body functions, activities/participation, environmental and personal factors; and additionally, health-related quality of life. However, neuropsychological rehabilitation is aimed at other groups of patients with acquired brain injury besides TBI. Measuring outcome of neuropsychological rehabilitation may therefore entail other instruments than the ones recommended by Wilde et al. (2010), Tate et al. (2013), and Honan et al. (2017). In addition, none of these studies is aimed at neuropsychological rehabilitation. The recommendations of Honan et al. aim at psychosocial research. Psychosocial studies entail the mental, emotional, social, and spiritual effects of a disease. Although psychosocial research is closely related to the aims of neuropsychological rehabilitation, it does not fully overlap. From our point of view, therefore, an overview of outcome measures for neuropsychological rehabilitation research is lacking.

To measure the outcome of neuropsychological rehabilitation, and allow for comparison and meta-analyses, a standard set of outcome measures is not yet available and researchers and clinicians are confronted with an overwhelming number of available instruments to choose from. To facilitate this choice, and as a first step towards a possible standard set of outcome measures in neuropsychological rehabilitation we performed a review of currently used outcome measures with a focus on intervention evaluation. We therefore reviewed outcome measures used in studies evaluating the outcomes of neuropsychological rehabilitation programmes in the last 20 years in our field. for each outcome measure we will present the frequency of use and we will group the instruments according to the framework of the International Classification of Functioning (ICF; World Health Organization [WHO], 2001) which informs us about the outcome domains which have been considered (most). Our overview is mainly aimed at the most common forms of acquired brain injury in adult (neuropsychological) rehabilitation being stroke and traumatic brain injury. The aim of this review was to identify outcome measures, which have been used in intervention research until now in order to offer researchers and clinicians an overview of the many outcome measures which are available. The overview can help to choose measurement instruments for future use. In this paper, we present the results of this review and propose future steps to come to a standard set of outcome measures and to help researchers and clinicians to choose among these many instruments in future. The instruments are divided into two main categories: neuropsychological tests and other measures.

Methods

Eligibility criteria. We aimed to give an overview of outcome measures, which were used in studies evaluating the outcome of neuropsychological interventions over the past 20 years. Reviews presenting the best evidence in terms of treatment effects based on outcome measurement in the field of neuropsychological rehabilitation as our starting point. Since many well-established, well-received and cited reviews are available in our field, we decided to extract the highest quality studies from those reviews. We considered the following categories: cognitive rehabilitation, both single domain (i.e., memory rehabilitation) and multi-domain (i.e., comprehensive neuropsychological rehabilitation), rehabilitation of emotional consequences (i.e., depression and anxiety) and rehabilitation of behavioural problems (i.e., aggression). For most of these categories, the highest quality studies were Randomized Controlled Trials (RCTs). If available, those were eligible for our review. In the category multi-domain cognitive rehabilitation, no RCTs are performed yet and the highest level of evidence with corresponding designs were taken into account in this category. This was not our decision but followed from the available reviews.

Information sources and search. Our search was conducted in August 2015. First, reviews were considered in the field of cognitive rehabilitation and comprehensive neuropsychological rehabilitation for patients with acquired brain injury such as stroke and traumatic brain injury (i.e., Cicerone et al., 2000, 2005, 2011). We also included the recommendations from an international team of researchers and clinicians known as INCOG for management of cognition following traumatic brain injury. From the INCOG papers, we extracted the RCTs on which the recommendations for cognitive rehabilitation in the areas of memory, attention, executive functions and self-awareness and aphasia were based. We considered the most common cognitive domains: memory, attention, language/communication, executive functions, neglect, perception, apraxia and awareness. Next, we searched the Cochrane library using the search terms stroke and (traumatic) brain injury which led to additional reviews in the field of aphasia, depression and anxiety. Additionally, we searched Pubmed with the search term “neuropsychological rehabilitation” using the filter “reviews” which led to additional reviews in the field of awareness, computer-based cognitive rehabilitation and behavioural interventions.

Study selection. The first and last author of this paper (CvH, IW) selected the reviews. The list of selected reviews was presented to the special interest group on neuropsychological rehabilitation of the World Federation for NeuroRehabilitation (WFNR SIG-NR) executive committee leading to the inclusion of additional reviews. We excluded one meta-analysis (Elliot & Parente, 2014) because the papers underlying this review were difficult to retrieve and mainly based on other reviews already considered.

Data collection process. We made an excel file of all RCTs presented in the selected reviews and removed duplicates as well as studies older than 1995. If a Cochrane review contained an ongoing study, we searched whether the study had been published meanwhile and included the most recent publication. In the field of comprehensive neuropsychological rehabilitation (Geurtsen, van Heugten, Martina, & Geurts, 2010) we also included designs other than RCTs because the evidence in this field is still weak and hardly any RCTs have been conducted. For each study included in the final list, we extracted all outcome measures used.

Synthesis of results. We translated the excel file into two lists of outcome measures (neuropsychological tests and other measures) categorized according to domain. For the categorization we used Lezak et al. (2012) for neuropsychological tests measuring cognitive functioning. If a test was not presented in Lezak, we followed the author’s own categorization. We used Tate (2010) for the categorization of the other domains arranged using the framework of the International Classification of Functioning (ICF; WHO, 2001).

Results

Our search led to a total of 30 reviews in which studies on the effectiveness of neuropsychological rehabilitation were discussed. Reviews were found in the following domains: memory (n = 4), attention (n = 2), executive functioning (n = 3), neglect (n = 2), perception (n = 1), apraxia (n = 1), aphasia (n = 2), awareness (n = 4), multi-domain (n = 3), emotional functioning (n = 7) and behavioural functioning (n = 1). The references of these reviews are presented in Appendix. In Figure 1 we present a flow diagram of the search and selection process.

Figure 1. Flow chart of studies and instruments included in the overreview.

In total 215 neuropsychological tests were used in outcome studies (Table 1) and 166 other outcome measures (Table 2). The detailed excel tables in which the references are added for each instrument can be obtained from the first author upon request.

Table 1. Neuropsychological tests categorized according to Lezak et al. (2012).

Domain	Subdomain	Category	Test	Number of times test is used in studies
Orientation & attention	Orientation	Time	Temporal Orientation Test	1
	Attention, concentration & tracking	General	Test of Everyday Attention:	3
			Map Search,	3
			Telephone Directory,	2
			Elevator count;	1
			Attention Process Training programme;	1
			d2 test of attentional performance;	1
			Konzentrations-Verlaufs-Test;	1
			Test of Attentional Performance,	3
			Divided attention;	3
			Behavioural Test of Inattentiveness in Daily Life;	1
			Inattentive Behaviour Task	2
		Reaction Time	Schenkenberg test;	1
			Simple Reaction Time;	2
			Go no-go test;	3
			Choice Reaction Time;	4
			Visual choice reaction time task;	1
			Computer based test of choice and reaction time;	1
			Cognispeed reaction time software;	1
			Visual Attention Task Overall Reaction Time;	1
			Wahl reaktions test;	1
			Complex reaction timer;	1
			Speed/accuracy trade off;	1
			Four choice reaction time task	1
		Vigilance	Ruff 2 & 7 of Selective attention test;	2
			Selective attention test;	1
			Digit Vigilance Test;	1
			Useful Field of View test;	1
			Gordon diagnostics of vigilance and distraction;	1
			Continuous Performance Task;	1
			Integrated Visual and Auditory Continuous Performance Test;	4
			Sustained Attention to Response Task;	3
			Sustained arousal and attention task	1
		Speed of information processing	Information intake task;	1
			Mental Slowness Observation Test;	1
			Tempo lern test;	1
			Adult Memory and Information Processing Battery	1
		Short term storage capacity/working memory	Stroop test;	9
			Stroop Neuropsychological Screening Test;	1
			Letter-Number Sequencing;	4
			Auditory Consonant Trigrams/Brown-Peterson Technique;	3
			Digit span;	10*
			Visual span;	3
			Paced Auditory Serial Addition Task;	13
			Paced Visual Serial Addition Task;	1
			Span board;	2
			2-back verbal working memory test;	1
			Tone counting task;	1
			The Memory Span & Tracking Task;	1
			Letter Number Sequencing Task;	1
			Räumlicher Suppressions-Arbeitsgedächtnis-Test	1
		Divided attention	Dual task	3
		Complex attention	Digit symbol test;	5
			Symbol Digit Modalities Test;	5
			Trail Making Test;	16*
			Matching accuracy test from “ ‘Brain Game Programme” ’;	1
			Color trail trial	1
Perception	Visual perception	General	Rivermead Perceptual Assessment Battery;	2
			Motor-free visual perception test	2
		Visual inattention	Letter cancellation test;	8
			Barrage test;	1
			Sentence reading test;	3
			Wundt-Jastrow Area illusion test;	1
			Behavioural Inattention Test;	10
			Mesulam Symbol Cancellation test;	3
			Mesulam Verbal Cancellation test;	1
			Line cancellation test;	3
			Line bisection test;	7
			Star Cancellation;	7
			Albert test;	1
			Bells test;	4
			Catherine Bergego Scale;	3
			Comb and Razor Test of personal neglect;	1
			Landmark test;	1
			o'clock test;	1
			Clock Drawing;	2
			Neglect test	1
		Visual recognition	Judgement of drawings test;	1
			Facial recognition;	1
			Oxford recurring faces test;	1
			Object recognition test;	1
			Spatial recognition test;	1
			Pattern recognition memory;	1
			Facial Expression Naming Task;	2
			Facial Expression Matching Task;	2
			Facial Expression Same/Different Task;	1
		Visuo-spatial/mental rotation	Money road map test	1
	Auditory perception		Dichotic listening;	1
			Auditory concentration test	1
Memory	General		Rivermead Behavioural Memory Test;	10
			Wechsler Memory Scale;	3*
			Wechsler Memory Scale-Revised;	4
			Wechsler Memory Scale-III;	3
			Memory assessment clinic (self and family) rating scale;	1
			Wechsler Adult Intelligence Scale – Revised;	4*
			Wechsler Adult Intelligence Scale;	1
			Raven's progressive matrices;	3
			Leistungsprüfsystem;	1
			Lern- und Gedächtnistest;	1
			Zahlen-verbindungstest;	1
			The brief multiparametric memory test	1
	Verbal memory		Vocabulary subtest of the Wechsler Adult Intelligence Scale— – Third Edition;	1
			The Hong Kong List Learning Test;	2
			California Verbal Learning Test;	5
			California Verbal Learning Test 2;	2
			Verbal learning test;	1
			Text reproduction;	1
			Claeson-Dahl Task;	1
			Auditory Verbal Learning Test;	1
			Rey Auditory verbal learning test;	4*
			Hopkins Verbal Learning Test(-Revised);	4
			Logical Memory	1
	Visual memory		Name-Face Paired Associated Memory Test;	1
			Visual paired associates;	1
			Brief Visual Memory Test Revised;	3
			Picture span test;	1
			Contextual memory test;	1
			Benton Visual Retention Test;	1
			Sabadel Task;	1
	Remote memory		15-words test	3
Language/ communication	Aphasia		Western Aphasia Battery;	8*
			Aachen Aphasia Test;	4
			Psycholinguistic Assessments of Language Processing in Aphasia	5
			(Shortened) Porch Index of Communicative Ability;	2
			Communication Abilities in Daily Living;	3
			The awareness of social interference test;	5
			Communicative Effectiveness Index;	5
			Boston Diagnostic Aphasia Examination;	4
			American Speech-Language Hearing Association Functional Assessment of Communication Skills;	1
			Functional Communication Therapy Planner;	1
			Functional communication profile;	2
			Functional communication	1
			Listening span task;	2
			Stichting Afasie Nederland Test;	1
			Norsk Grunntest for Afasie;	1
			Aphasia Quotient;	2
			Chinese Rehabilitation Research Centre Aphasia Examination;	1
			Aphasia Battery for Chinese;	2
			Chinese Functional Communication Profile;	1
			Quick Aphasia Screening Test;	1
			Functional Outcomes Questionnaire for Aphasia;	1
			Profile of Functional Impairment in Communication	1
	Verbal expression		Categorical word fluency;	1
			Boston Naming Test;	3*
			Amsterdam-Nijmegen Everyday Language Test;	6
			Picture description with structure modelling;	1
			Caplan and Hanna’s Sentence Production Test;	1
			Object and Action Naming Battery;	1
			Probe picture naming	1
	Verbal comprehension		Semantic Association Test;	4
			Communicative Activity Log;	1
			Sentence and Picture comprehension from the Philadelphia Comprehension Battery;	1
			Speed and Capacity of Language Processing Test;	2
			Token Test;	1
			Sentences verification task (adapted from Speed of Comprehension Task);	1
	Verbal academic skills	Reading	Wechsler Test of Adult Reading;	4
			National Adult Reading Test	2
Construction	Drawing		Copy of a drawing;	1
			Rey Complex Figure Test	1
Concept formation & reasoning	Concept formation		Hayling & Brixton Tests;	2
			Category Test of Halstead-Reitan Battery;	1
			Wisconsin Card Sorting Test;	7
			Booklet Category Test	2
	Reasoning		Matrix-Reasoning subtest of the WAIS-III;	2
			Test of Strategic Learning	1
Executive functions & motor performance	Executive functions	Volition	Color word interference test	2
		Problem solving	Social Problem Solving; Means-Ends Problem Solving	1
			Design Fluency Test;	2
			D-KEFS, Tower test;	1
			Design and Verbal Fluency Switching;	1
			Frontal assessment battery;	1
			Category Fluency/semantic word fluency;	3
			Controlled Oral Word Association Test;	4
			Letter fluency;	2
		Wide range assessment	D-KEFS,	1
			Color-Word Interference Test;	7
			Behavioural Assessment of the Dysexecutive Syndrome,	5
			(Modified) Six Elements Test;	5
			Dysexecutive test;	2
			Hotel Task;	1
			Removals Task;	1
			Scales of Cognitive Ability for Traumatic Brain Injury;	1
			Executive Observational Scale; Executive Secretarial Task;	1
			JAAM virtual reality task;	1
			Mass transit railway
		Motor performance	Functional motor test	1
Neuropsychological test batteries			Wechsler Abbreviated Scale of Intelligence;	2
			Wide Range Achievement Test 3;	1
			Woodcock-Johnson III	1

*Instruments recommended in Honan et al. (2017).

Table 2. Other outcome measures categorized according to Tate (2010) following the ICF framework.

Domain				Instrument	Number of times test is used in studies
Functioning and disability	Mental functions	Global	Temperament & personality	Beck Hopelessness Scale;	1
				Beck Scale for suicidal ideation;	1
				Personality Questionnaire	1
			Energy & Drive	Fatigue Severity Scale;	1*
				Apathy Evaluation Scale	1
		Specific	Attention	Attention questionnaire;	1
				Rating Scale of attentional behaviour;	2
				mental slowness questionnaire	1
			Memory	Memory Questionnaire;	2
				Memory Awareness Rating Scale;	1
				Everyday Memory Questionnaire;	2
				report of everyday memory failures	1
			Higher level cognitive functions	Patient Competency Rating Scale;	2*
				Self-Awareness of Deficits Interview;	2*
				Self-Awareness of Deficits Interview-	1
				Chinese version;	2
				Assessment of Awareness of Disability;	6
				Awareness Questionnaire;	1
				Clinician Rating Scale;	3
				Self-Regulation Skills Inventory;	1
				Social Problem Solving Inventory-Revised;	2
				Problem solving questionnaire
			Emotions	Hamilton Anxiety Rating Scale;	3
				Social Phobia and anxiety inventory;	1
				Beck Anxiety Inventory;	1
				Hamilton Depression Rating Scale;	21
				Beck Depression Inventory (-II);	17
				Wakefield Depression Inventory;	1
				Montgomery Asberg Depression Rating Scale;	5
				Yale Single Question;	1
				Geriatric Depression Scale;	2
				Center for Epidemiological Studies-Depression Scale;	5
				Bradburn scale of psychologic wellbeing-affect balance scale;	1
				Depression scale;	1
				State Trait Anger Expression Inventory;	1
				Anger log;	1
				Hospital Anxiety and Depression Scale;	10*
				Emotional State Questionnaire;	1
				visual analogue mood scales;	1
				Depression Anxiety and Stress scale;	4
				25-item affect-mood scale	1
		Multi-category		Mini-Mental State Examination;	13
				Stroke Unit Mental Status Examination;	1
				Lowenstein occupational therapy cognitive assessment;	1
				Neurobehavioural Cognitive Status Examination;	4
				Clinical Global Impression scale;	7
				Present State Exam;	1
				National Institutes of Health Stroke Scale;	1
				Scandinavian Stroke Scale;	5
				Chinese stroke scale;	1
				Profile of Mood states;	1
				Impact of event scale;	1
				Brief Psychiatric Rating Scale;	1
				DSM-III-R symptom checklist;	1
				dysexecutive questionnaire;	4
				Alzheimer's diseases assessment scale-cognitive subscale	1
	Neuromuscular and movement-related			Action Research Arm test;	1
				Motricity Index	2
	Multi-domain			Symptom Checklist 90-R;	7
				Patient Health Questionnaire;	1
				Rivermead Post-Concussion Symptoms Questionnaire	4
Activities & Participation	General tasks & demands			Perceived Stress Scale;	1
				Questionnaire on Resources and Stress;	2
				Maslach Burnout Inventory	2
	Language/communication			La Trobe Communication Questionnaire;	2*
				Self Communication Questionnaire
					1
	Mobility	Driving		Charron test;	1
				Center for fitness to drive evaluation and Car adaptations	1
	Interpersonal interaction & relationships			Behavioural-referenced Rating System of intermediate social skills-revised	2
	Major life areas			Employability Rating scale;	1
				Vocational Independence Scale;	1
				Vocational integration scale	1
	Multi-domain			Assessment of Motor and Process Skills;	3
				Barthel Index (and modified version);	21
				Functional Independence Measure;	12
				Independent Living Scale;	1
				Edmans ADL Index;	1
				ADL checklist (Therapist Observation Checklist);	1
				ADL observation-NL;	1
				ADL observation;-It	1
				ADL scale;	1
				Rivermead ADL;	1
				Canadian Occupational Performance Measure;	4
				Frenchay Activities Index;	1
				Bayer activities of daily living;	1
				adapted Nottingham stroke dressing ability;	1
				Nottingham Extended ADL scale;	2
				Lawton Instrumental Activities of Daily Living;	1
				Lawton IADL chinese version;	1
				Hong Kong Chinese version of the Lawton IADL scale;	1
				IADL scale;	1
				American Heart Association Stroke Outcome Classification;	1
				Functional Autonomy Rating Scale;	1
				Interest checklist;	1
				London Handicap Scale;	1
				Community Integration Questionnaire;	16
				Quality of community Integration Questionnaire;	1
				Role Resumption List;	1
				Role checklist;	1
				Craig Handicap Assessment and Reporting Technique;	3*
				Social Performance Survey Schedule;	1
				Sydney Psychosocial Reintegration Scale;	2*
				Social Communication skills Questionnaire-Adapted;	1
				Measure of Participation in Communication; Adapted Kagan scales;	1
				Adapted measure of participation in conversation;	1
				Adapted measure of support in conversation;	1
				Social activity interview;	1
				Weekly Social Activity Survey	1
Multi-dimensional				Rankin Scale;	4
				Neurobehavioral Functioning Inventory-R;	1
				Neurobehavioural Rating Scale;	1
				Clinician administered PTSD scale;	1
				General Health Questionnaire-28/12;	7
				Short-form health survey / Medical Outcome Scale Short Form 36;	9
				Disability Rating Scale;	2
				Functional status examination;	1
				Wahler Physical complaints checklist;	1
				Functional Assessment Measure;	3
				Cognitive Failure Questionnaire;	6
				Cognitive difficulties scale;	1
				Trauma Complaints List;	1
				Target Complaints list;	1
				Goal Attainment Scale;	5
				Treatment Goal Attainment;	1
				Katz adjustment scale;	1
				European Brain Injury Questionnaire;	1
				Brain injury community rehabilitation outcome-39;	1
				Rivermead Follow-up Questionnaire;	2
				Problem checklist;	2
				Glasgow outcome scale-extended;	1*
				Stroke Impact Scale;	1
				Sickness Impact Profile;	2
				Mayo Portland Adaptability Inventory;	3*
				Health and activity limitation survey;	1
				Clinical Dementia Rating Scale	1
Contextual	Environmental			UCLA loneliness;	1
				6-item Interpersonal Support evaluation List	1
	Personal			Physical self-description questionnaire;	2
				Self-perception in rehabilitation questionnaire;	2
				Coping Scale for Adults;	1
				Utrecht Coping List;	2
				Trier Coping Scale;	1
				The Ways of Coping scale revised;	1
				Holmes Rahe survey of recent experiences;	1
				Rosenberg Self-Esteem Scale;	4*
				Coopersmith Self-Esteem Inventories;	2
				Dutch Self-Esteem Questionnaire;	1
				Self-efficacy Questionnaire;	2
				general perceived self-efficacy scale;	1
				Stroke Expectations Questionnaire;	1
				Multidimensional Health locus of control scale;	1
				stress appraisal coping measure;	1
				knowledge of stroke questionnaire;	1
				treatment expectations scale	1
Concepts not covered by ICF	Quality of life			European Quality of Life (EuroQOL);	5*
				Quality of life after brain injury;	1
				Perceived Quality of life scale;	1
				Life Satisfaction questionnaire;	3
				Scale for subjective well-being for the elderly;	1
				Satisfaction with life scale;	1
				Job satisfaction checklist;	1
				stroke and aphasia quality of life scale;	1
				quality of life scale;	1
				scale of psychological well-being;	1
				patient global impression scale	1
Caregiver				Caregiver Strain Index;	1
				Modified burden scale	1

*Instruments recommended in Honan et al. (2017).

As can be seen in Table 1, the number of neuropsychological tests is large. Many instruments were used in only one RCT. Instruments which have been used in five or more studies are: digit span, Paced Auditory Serial Addition Task (PASAT), Stroop test, digit symbol test, Symbol Digit Modalities Test (SDMT), Trail Making Test (TMT), letter cancellation test, Behavioural Inattention Test (BIT), line bisection, star cancellation, Rivermead Behavioural Memory Test (RBMT),California Verbal Learning Test (CVLT), Western Aphasia Battery (WAB), Communication Abilities in Daily Living, the awareness of social interference test (TASIT), Psycholinguistic Assessments of Language Processing in Aphasia, the communicative effectiveness index (CETI), Amsterdam-Nijmegen Everyday Language Test, Wisconsin Card Sorting Test (WCST), (modified) six elements test, Color-Word Interference Test, Behavioural Assessment of the Dysexecutive Syndrome (BADS), and the DysExecutive Questionnaire (DEX). Outcome measurement with neuropsychological tests was most often performed in the domains of working memory, reaction times, visual attention/neglect and aphasia.

In Table 2 the other instruments are presented. Although the number of instruments in this category is somewhat lower, it is still very high and also in this category some instruments are only used once. Instruments used in five studies or more are only a few: Awareness Questionnaire, Hamilton Depression Rating Scale (HDRS), Beck Depression Inventory-II (BDI-II), Montgomery Asberg Depression Rating Scale (MADRS), Hospital Anxiety and Depression Scale (HADS), Mini Mental State Examination (MMSE), Clinical Global Impression scale (CGI), Scandinavian Stroke Scale (SSS), Symptom Checklist-90 (SCL-90), Barthel Index, Functional Independence Measure (FIM), Community Integration Questionnaire (CIQ), General Health Questionnaire 28 or 12 (GHQ-28, GHQ-12), Short-form health survey / Medical Outcome Scale (SF-36), Cognitive Failure Questionnaire (CFQ), Goal Attainment Scaling (GAS), and the EuroQOL. Instruments other than neuropsychological tests were most often measures on emotional functioning, on multi-category mental functions, multi-domain measures of activities and participation, multidimensional measures and measures on personal factors.

Discussion

In this overview, outcome measures used in intervention evaluation studies of neuropsychological rehabilitation are gathered. Most measures are found on the level of cognitive functioning, which is in line with the many neuropsychological tests that are available. Many countries have their own preferences and many neuropsychologists have personal favourites. Most common domains, which are often presented in neuropsychological assessment, are also found in outcome assessment. Some more than others, such as working memory, neglect and aphasia, but this is to be expected given that most RCTs are performed in these fields.

The focus of (neuropsychological) rehabilitation, however, is much broader than cognitive functioning in helping patients with brain injuries to achieve their optimum level of physical, psychological, social, and vocational well-being (Wilson, 2002). Moreover, most of the evidence on the effectiveness of neuropsychological rehabilitation is found in treatments focusing on compensation or adjustment to cognitive impairments rather than restoring them (i.e., Cicerone et al., 2011; Spikman & Fasotti, 2017). Repeating the neuropsychological tests after treatment, therefore, may not be the most evident form of outcome measurement.

The British Society for Rehabilitation Medicine (BSRM) and Royal College of Physicians (RCP) in the United Kingdom define rehabilitation as “a process of active change by which a person who has become disabled acquires the knowledge and skills needed for optimal physical, psychological and social function” and in terms of service provision this entails “the use of all means to minimize the impact of disabling conditions and to assist disabled people to achieve their desired level of autonomy and participation in society” (BSRM/RCP National Clinical Guidelines, 2003, p. 7). Considering the ICF framework this would mean that the outcome of (neuropsychological) rehabilitation should be measured on the level of activities and participation.

Concerning the other measures, many multi-domain, multi-category and multi-dimensional instruments are used covering many different aspects of human functioning. From efficiency and feasibility point of view this is even to be preferred instead of using separate instruments for each domain.

It is remarkable that this overview resulted in about half of the number of instruments covered by Tate et al. (2013) (381 versus 728). On the one hand more instruments would have been expected because the current overview is not only about traumatic brain injury. On the other hand, neuropsychological rehabilitation is aimed at specific domains of human functioning and not all ICF domains are relevant in our field. In comparison, for instance, many body functions (i.e., sensory functions, pain, neuromusculoskeletal functions, movement-related functions) are not measured as outcomes of neuropsychological rehabilitation, nor is mobility on the level of activities. Most overlap is found on the measurement of mental functions which is to be expected.

Future steps towards a standard set of outcome measures

The tables presented in this paper can be used as a reference guide for future research. There are different ways to choose a core set of instruments from the enormous number of instruments that have been used so far. One could select from those currently used, or the most common ones or most used ones. These are, however, not necessarily the best instruments (Van Heugten, 2017). Another or additional criterion for selection could be the quality of the instrument in terms of its psychometric properties. Many different sets of criteria are suggested (i.e., Andresen, 2000; Terwee et al., 2007), but for outcome measurement one could argue that the instrument should at least have adequate responsiveness, the ability of the instrument to detect clinically relevant changes over time. One could determine the minimally important difference (MID) which represents the smallest change on a patient reported outcome measure that is relevant to patients. For the most commonly used outcome measures we provide references to sources where information on the psychometric properties of these instruments relevant to brain injury can be found. At this stage, it is up to the clinician or researcher to decide which properties are most relevant to their intended use of the instrument. Additionally, the feasibility in clinical practice can be considered in terms of, for instance, the availability of the instrument and the duration of assessment.

We took such an approach in our reviews on assessments instruments to measure awareness deficits after brain injury (Smeets, Ponds, Verhey, & van Heugten, 2012) and coping after brain injury (Gregório, Brands, Stapert, Verhey, & van Heugten, 2014). In both reviews, we performed a systematic search for instruments and made an inventory of the information, which is available on the psychometric properties and feasibility aspects in the next step. This led to cautious recommendations for future use of only a few instruments on these specific outcome domains. Recommendations are cautious because often information is not available and decisions are difficult to make. If we would take the current list of measures presented in this review and apply a set of quality criteria, there will probably still be too many instruments left to choose from. As a first step, we added a table (Table 3) with a list of the most commonly used instruments with reference to the ICF domain which they represent and a reference to information on their psychometric properties.

Table 3. Properties of most used instruments.

Instrument	ICF domain	Reference to information on psychometric properties in relation to use in brain injured people
Awareness Questionnaire	Function & disability, mental functions, specific, higher level cognitive functions	Tate (2010) and Smeets et al. (2012)
Hamilton Depression Rating Scale	Function & disability, mental functions, specific, emotions	Bech (2015) and Timmerby, Andersen, Søndergaard, Østergaard, and Bech (2017)
Beck Depression Inventory	Function & disability, mental functions, specific, emotions	Burton and Tyson (2015)
Montgomery Asberg Depression Rating Scale	Function & disability, mental functions, specific, emotions	Khan, Khan, Shankles, and Polissar (2002)
Hospital Anxiety and depression Scale	Function & disability, mental functions, specific, emotions	Burton and Tyson (2015)
Mini Mental State Examination	Function & disability, mental functions, multicategory	Tate (2010) and Van Heugten, Walton, and Hentschel (2015)
Clinical Global Impression Scale	Function & disability, mental functions, multicategory	Khan et al. (2002)
Scandinavian Stroke Scale	Function & disability, mental functions, multicategory	Lindenstrøm, Boysen, Christiansen, Hansen, and Nielson (1991)
Symptom CheckList-90 R	Function & disability, multi-domain	Carrozzino, Morberg, Siri, Pezzoli, and Bech (2018) and Bech, Bille, Møller, Hellström, and Østergaard (2014)
Barthel Index	Activity & Participation, multi-domain	Tate (2010)
Functional Independence Measure	Activity & Participation, multi-domain	Tate (2010)
Community Integration Questionnaire	Activity & Participation, multi-domain	Tate (2010)
General Health Questionnaire	Multi-dimensional	Graetz (1991)
Short Form-36	Multi-dimensional	Polinder, Haagsma, van Klaveren, Steyerberg, and van Beeck (2015)
Cognitive Failure Questionnaire	Multi-dimensional	Tate (2010)
Goal Attainment Scaling	Multi-dimensional	Grant and Ponsford (2014) and Bouwens, van Heugten, and Verhey (2009)
EuroQol	Quality of Life	Polinder et al. (2010)

A future step could then be to seek the opinion of experts and/or adopt Delphi or consensus procedures. An example in which common use, quality criteria and expert opinion were all combined is the European consensus on outcome measures for psychosocial intervention research in dementia care (Moniz-Cook et al., 2008). Recently, a similar approach was used to recommend outcome instruments for psychosocial research in moderate-to-severe TBI (Honan et al., 2017). Especially the latter of the two may provide a good starting point for the minimization of the set of reviewed instrument in neuropsychological rehabilitation in this paper albeit that neuropsychological rehabilitation is not only aimed at TBI patients and psychosocial research may not fully overlap with neuropsychological rehabilitation. To illustrate the partial overlap between neuropsychological rehabilitation and psychosocial functioning, the instruments recommended by Honan et al. (2017) are marked in Tables 1 and 2; only 17 of the 59 recommended instruments are used in the studies described here. However, the stepwise procedures for the selection of instruments may be followed.

Limitations

This review does have some limitations. First, we took existing reviews in our field as starting point instead of applying a full search on studies in neuropsychological rehabilitation. However, we expect that we have not missed many studies by incorporating the Cochrane, Cicerone and INCOG reviews. Extracting the search terms from these reviews would most probably lead to the selection of the same evaluation studies, which is why we choose to start from there. Second, these reviews do focus on the most common forms of acquired brain injury (i.e., stroke, TBI) which means that we did not incorporate other forms of brain injury such as Multiple Sclerosis (MS) for which also a Cochrane review on neuropsychological rehabilitation exists (Rosti-Otajärvi & Hämäläinen, 2014). Often, however, MS specific outcome measures are used in these studies, which would not be considered for a standard set of outcome measures in neuropsychological rehabilitation. Next, we limited our selection of studies to those performed in the last 20 years. Again, however, we did not miss many studies this way because not much high quality RCTs were published before 1995 as can be seen from the included studies in the Cochrane reviews we selected. Finally, we undertook our search in August 2015 which is already three years ago. We therefore did not include the most recent studies which may have introduced new outcome measures. Since 2015, there are three new Cochrane reviews which are relevant to our field: memory deficits after stroke (das Nair, Cogger, Worthington, & Lincoln, 2016), anxiety after stroke (Knapp et al., 2017) and aphasia after stroke (Brady, Kelly, Godwin, Enderby, & Campbell, 2016). The evidence has increased rapidly in the field of memory rehabilitation: in 2007 two trials including 18 participants were included in the review, while in the latest review 13 trials including 514 participants were taken into account. However, no new instruments were included that we have missed in our overview. Moreover, the authors concluded, among other things, that there was a lack of consistency in the choice of outcome measures which hinders comparison of effects. This means that, although the evidence has increased, there is still a large variation in outcome measures and further step towards consensus on outcome measurement in memory rehabilitation has not been made so far. We therefore expect that our review is sufficiently up to date to draw conclusions about outcome measurement. Finally, we cover many domains in which outcome measurement is established, while emerging domains such as social cognitive functioning are not taken into account yet. Since there have not been many intervention evaluation studies in these emerging fields yet, outcome measures are also not represented either.

Conclusions

In this paper the importance of outcome measurement is emphasized and an overview of instruments used in the past 20 years is presented. Clinicians wanting to evaluate an intervention or researchers setting up an evaluation study can use the overview of commonly used outcome measures to guide their measurement choices. Since the number of outcome measures is too large, the development of a standard set of measures is a necessary next step. In current health care, clinical neuropsychologists are probably better off by suggesting our own preferred instruments than waiting for policy makers, management or governments forcing routine outcome measurements upon them. Suggestions are offered on how to select outcome measures and to come to a core set of outcome measures on the basis of different selection criteria. The overall quality of the instrument can be considered, and the feasibility of the instrument for use in clinical practice or research can be taken into account. Additionally, expert opinion and/or consensus can be sought as has been done successfully in other, related fields of health care.

Disclosure statement

No potential conflict of interest was reported by the authors.

Appendix. Selected reviews

Cognitive functioning, domain-specific

Table

Memory:	das Nair R, Lincoln N. Cognitive rehabilitation for memory deficits following stroke. Cochrane Database Syst Rev 2007.
	Spreij L, Visser-Meily J, Van Heugten C, Nijboer T. Novel insights into the rehabilitation of memory post acquired brain injury: a systematic review. Frontiers in human neuroscience 2014: Dec 16 (epub ahead of print).
	Jamieson M, Cullen B, McGee-Lennon M, Brewster S, Evans JJ. The efficacy of cognitive prosthetic technology for people with memory impairments: a systematic review and meta-analysis. Neuropsychol Rehabil. 2014;24(3–4):419–44.
	Velikonja D, Tate R, Ponsford J, McIntyre A, Janzen S, Bayley M; INCOG Expert Panel. INCOG recommendations for management of cognition following traumatic brain injury, part V: memory. J Head Trauma Rehabil. 2014 Jul-Aug;29(4):369–86.
Attention:	Loetscher T, Lincoln NB. Cognitive rehabilitation for attention deficits following stroke. Cochrane Database Syst Rev 2013;5:CD002842.
	Ponsford J, Bayley M, Wiseman-Hakes C, Togher L, Velikonja D, McIntyre A, Janzen S, Tate R; INCOG Expert Panel. INCOG recommendations for management of cognition following traumatic brain injury, part II: attention and information processing speed. J Head Trauma Rehabil. 2014 Jul-Aug;29(4):321–37.
Exec funct:	Krasny-Pacini A, Chevignard M, Evans J. Goal Management Training for rehabilitation of executive functions: a systematic review of effectiveness in patients with acquired brain injury. Disabil Rehabil. 2014;36(2):105–16.
	Chung CS, Pollock A, Campbell T, Durward BR and Hagen S. Cognitive rehabilitation for executive dysfunction in adults with stroke or other adult non-progressive acquired brain damage. Cochrane Database Syst Rev 2013; 4
	Tate R, Kennedy M, Ponsford J, Douglas J, Velikonja D, Bayley M, Stergiou-Kita M. INCOG recommendations for management of cognition following traumatic brain injury, part III: executive function and self-awareness. J Head Trauma Rehabil. 2014 Jul-Aug;29(4):338–52.
Neglect:	Lisa LP, Jughters A, Kerckhofs E. The effectiveness of different treatment modalities for the rehabilitation of unilateral neglect in stroke patients: a systematic review. NeuroRehabilitation 2013;33(4):611–20.
	Bowen A, Hazelton C, Pollock A, Lincoln NB. Cognitive rehabilitation for spatial neglect following stroke. Cochrane Database Syst Rev 2013;7:CD003586.
Perception:	Bowen A, Knapp P, Gillespie D, Nicolson DJ, Vail A. Non-pharmacological interventions for perceptual disorders following stroke and other adult-acquired, non-progressive brain injury. Cochrane Database Syst Rev 2011;(4):CD007039.
Apraxia:	West C, Bowen A, Hesketh A, Vail A. Interventions for motor apraxia following stroke. Cochrane Database Syst Rev 2008;(1):CD004132.
Language/communication:	Brady MC, Kelly H, Godwin J, Enderby P. Speech and language therapy for aphasia following stroke. Cochrane Database SystRev. 2012;5:CD000425.
	Togher L, Wiseman-Hakes C, Douglas J, Stergiou-Kita M, Ponsford J, Teasell R, Bayley M, Turkstra LS; INCOG Expert Panel. INCOG recommendations for management of cognition following traumatic brain injury, part IV: cognitive communication. J Head Trauma Rehabil. 2014 Jul-Aug;29(4):353–68.
Awareness:	Schrijnemakers AC, Smeets SM, Ponds RW, van Heugten CM, Rasquin S. Treatment of unawareness of deficits in patients with acquired brain injury: a systematic review. J Head Trauma Rehabil. 2014 Sep-Oct;29(5):E9-E30.
	Schmidt J Lannin N, Fleming J, Ownsworth T. FEEDBACK INTERVENTIONS FOR IMPAIRED SELF-AWARENESS FOLLOWING BRAIN INJURY: A SYSTEMATIC REVIEW. Journal of Rehab Med 2011; 43: 673–680.
	Ownsworth T, Haslam C. Impact of rehabilitation on self-concept following traumatic brain injury: An exploratory systematic review of intervention methodology and efficacy. Neuropsych Rehab 2014.
	Tate R, Kennedy M, Ponsford J, Douglas J, Velikonja D, Bayley M, Stergiou-Kita M. INCOG recommendations for management of cognition following traumatic brain injury, part III: executive function and self-awareness. J Head Trauma Rehabil. 2014 Jul-Aug;29(4):338–52.

Cognitive functioning, multi-domain

Cha and Kim; Effect of computer-based cognitive rehabilitation for people with stroke: a systematic review and meta-analysis; Neurorehabilitation 2013: 32;359–368.

Heugten CM van, Wolters Gregorio G, Wade DT. Evidence Based Cognitive Rehabilitation after acquired brain injury: systematic review of content of treatment. Neuropsychological Rehabilitation 2012: 22(5): 653–73.

Geurtsen GJ, Heugten CM van, Martina JD, Geurts ACH. Comprehensive rehabilitation programmes in the chronic phased after brain injury: a systematic review. Journal of Rehabilitation Medicine 2010; 42(2): 97–110.

Emotions: non-pharmacological treatment

Anger WH, Jr. Interventions for treating anxiety after stroke. International Journal of Evidence-Based Healthcare. 2012;10(1):82–3.

Campbell Burton CA, Holmes J, Murray J, Gillespie D, Lightbody CE, Watkins CL, et al. Interventions for treating anxiety after stroke. Cochrane Database of Systematic Reviews. 2011(12):CD008860.

Hackett ML, Anderson CS, House A, Xia J. Interventions for treating depression after stroke. Cochrane Database Syst Rev 2008.

Soo, Tate. Psychological treatment for anxiety in people with TBI. Cochrane review 2007.

Stalder-Luthy et al. Effect of psychological interventions on depressive symptoms I long-term rehabilitation after an acquired brain injury: a systematic review and meta-analysis. Archives of Physical Medicine and Rehabilitation 2013;94:1386–97.

Barker-Collo S, Starkey N, Theadom A. Treatment for depression following mild traumatic brain injury in adults: a meta-analysis. Brain Injury 2013; 17(10): 1124–1133.

Fann J, Hart T, SChomer K. Treatment for depression after traumatic brain injury. J Neurotrauma 2009; 26(12): 2383–2402.

Behaviour: non-pharmacological treatment

Cattelani R, Zettin M, Zoccolotti P. Rehabilitation treatments for adults with behavioural and psychosocial disorders following acquired brain injury: a systematic review. Neuropsychol Rev. 2010 Mar;20(1):52–85.