Examining the repeatable battery for the assessment of neuropsychological status validity indices in people with schizophrenia spectrum disorders

Abstract

Objective: We examined the frequency of possible invalid test scores on the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) in patients with schizophrenia spectrum disorders, and whether there was an association between scores on the embedded RBANS performance validity tests (PVTs) and self-reported symptoms of apathy as measured by the Initiate Scale of the Behavior Rating Inventory of Executive Function-Adult Version (BRIEF-A). Methods: Participants included 250 patients (M = 24.4 years-old, SD = 5.7) with schizophrenia spectrum disorders. Base rates of RBANS Effort Index (EI), Effort Scale (ES), and Performance Validity Index (PVI) test scores were computed. Spearman correlations were used to examine the associations between the RBANS PVTs, the RBANS Index scores, and the BRIEF-A Initiate Scale. Regression analyses were used to investigate how well the RBANS PVTs predicted scores on the BRIEF-A Initiate Scale. Results: The frequency of invalid scores on the EI (>3) and the PVI (<42) in participants with schizophrenia spectrum disorders was 6%. The frequency of invalid ES scores (<12) was 28% in the patients compared to 15% in the U.S. standardization sample. There was a small significant correlation between the EI and the BRIEF-A Initiate Scale (rho=.158, p<.05). Conclusions: The rates of invalid scores were similar to previously published studies. Invalid scores on the BRIEF-A were uncommon. Apathy measured with the BRIEF-A Initiate Scale was not associated with performance on the RBANS validity measures or with measures of cognition.

Keywords:

• Schizophrenia
• Test Validity
• RBANS
• BRIEF–A
• Negative Symptoms

Introduction

Cognitive impairment is commonly experienced by patients with schizophrenia spectrum disorders (Barder et al., 2013; Keefe, 2014; Øie et al., 2011) along with apathy and avolition (Barch et al., 2014; Blanchard et al., 2011; Dollfus & Lyne, 2017; Dorofeikova et al., 2018; Galaverna et al., 2014; Morra et al., 2015; Strauss et al., 2013). Apathy and avolition have a negative impact on functional outcome (Rabinowitz et al., 2012; Strauss et al., 2013) and afflict about 40% of patients (Patel et al., 2015; Rabinowitz et al., 2013; Schennach et al., 2015). Researchers have reported that about 20% of clinical samples with schizophrenia spectrum disorders perform at levels suggesting possible invalid performance on neuropsychological tests (Morra et al., 2015). Depending on sample characteristics and type of performance validity tests (PVTs; Heilbronner et al., 2009), the frequency of invalid scores reported by previous researchers has ranged from about 5% (Duncan, 2005; Egeland et al., 2003) to 72% (Gorissen et al., 2005; Hunt et al., 2014). About 15–35% of the variance in performance on cognitive tests can be accounted for by invalid scores and/or negative symptoms in patients with schizophrenia spectrum disorders (Foussias et al., 2015; Gorissen et al., 2005; Whearty et al., 2015).

Symptoms of apathy and avolition in schizophrenia spectrum disorders are commonly assessed using rating scales, such as the Scale for the Assessment of Negative Symptoms (SANS), the Positive and Negative Syndrome Scale (PANSS), or the Negative Symptom Assessment (NSA; Blanchard et al., 2011; Van Erp et al., 2014). Instruments like these have been criticized as problematic and outdated by some authors (Blanchard et al., 2011; Kumari et al., 2017; Strauss et al., 2012). Concerns have been raised that these measures do not adequately address cognitive factors (Kumari et al., 2017) and omit patients’ self-report of relevant symptoms and internal states (Blanchard et al., 2011). There is some evidence that self-report questionnaires, such as the Behavior Rating Inventory of Executive Function–Adult Version (BRIEF–A; Isquith et al., 2005) can provide clinically important information on executive functions, including symptoms of motivation and initiation problems, in everyday life in patients with neurological or psychiatric problems (Løvstad et al., 2016; Power et al., 2012). Patients with schizophrenia report greater dysfunction on the BRIEF–A Working Memory and Shift scales compared to healthy controls, indicating more difficulty holding information in mind and adjusting to changes in routine (Kumbhani et al., 2010). In patients diagnosed with schizophrenia, greater self-reported dysfunction on the BRIEF–A Working Memory scale was associated with smaller bilateral frontal lobe volumes and with worse performances on neuropsychological tests of working memory (Garlinghouse et al., 2010). Further, patients with schizophrenia have reported greater dysfunction on the BRIEF–A Initiate Scale compared to the other BRIEF–A scales (Bulzacka et al., 2013), indicating that patients rated their problems with beginning a task or activity and independently generating ideas, responses, or problem solving strategies as comparatively worse than problems with inhibition, flexibility, emotional control, social awareness, working memory, and with planning.

There is no consensus regarding what symptoms or behaviors best constitute the term “negative symptoms” in schizophrenia spectrum disorders, but a central concept is reduction or absence of behaviors related to motivation and interest (Correll & Schooler, 2020). The BRIEF–A Initiate Scale has items related to motivation and interest, such as behavioral descriptions concerning enthusiasm and engagement, passiveness, and ability to independently start tasks or assignments. This suggests that the BRIEF–A Initiate Scale, in particular, might capture important aspects of negative symptoms associated with schizophrenia spectrum disorders (e.g., affective blunting, avolition, apathy, and anhedonia).

Poor performance on cognitive tasks might, in part, be due to lack of engagement or motivation to do well rather than difficulties with the cognitive processes measured by the tasks (Barch, 2005; Iverson & Binder, 2000). Empirically derived PVTs can assist clinicians in evaluating whether patients’ test results are attributable to invalid test performance as opposed to a cognitive weakness or impairment (Chafetz et al., 2015; Sherman et al., 2020; Slick et al., 1999). Evaluation of invalid test performance is particularly important when assessing cognitive functions in patients with schizophrenia spectrum disorders, due to the high prevalence of motivation and initiation problems. Researchers have even proposed that empirically derived PVTs can be used to identify patients in need of intervention for motivational difficulties (Morra et al., 2015). Interventions for increasing intrinsic motivation in patients with schizophrenia disorders when learning cognitively demanding tasks have been promising (Choi & Medalia, 2010), underscoring the importance of assessing and targeting both motivational problems and cognitive deficits when choosing and adapting treatment and rehabilitation protocols for this patient population.

The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Randolph, 1998) has been used to assess cognitive functioning in a large range of clinical populations of neurological and psychiatric diseases (Randolph, 1998, 2012), including patients with schizophrenia spectrum disorders (Dickerson et al., 2004; Hobart et al., 1999; Iverson et al., 2009; Loughland et al., 2007; Wilk et al., 2004). Several empirically derived embedded PVTs have been constructed for the RBANS (Novitski et al., 2012; Paulson et al., 2015; Silverberg et al., 2007). Results from meta-analyses have indicated that some of these PVTs are sufficiently specific to use as indicators for invalid test performance in neuropsychological assessments (Goette & Goette, 2019; Shura et al., 2018). However, in schizophrenia spectrum disorders, patients with greater negative symptoms are more likely to score below cutoffs on the RBANS PVTs (Galaverna et al., 2014; Morra et al., 2015; Whearty et al., 2015), suggesting that performances on these measures may indicate the effects of apathy and avolition on test performance in this patient population (Morra et al., 2015). That is, based on previous literature, clinicians can conclude that many patients with schizophrenia spectrum disorders may have problems engaging in cognitive testing, but might want to know if a patient had these difficulties during the cognitive assessment, and if so, to what extent. This approach differs somewhat from forensic and litigation cases, where the detection of feigned cognitive symptoms and deficits is more important (Williams et al., 2020).

The RBANS allows for the concurrent assessment of cognitive deficits and motivational problems in patients with schizophrenia spectrum disorders. The frequencies at which patients with schizophrenia exceed cutoffs on RBANS PVTs have been examined for some, but not all, possible PVTs deriving from the battery, and have ranged from 9–24% (Bailie et al., 2012; Bayan et al., 2018; Moore et al., 2013; Morra et al., 2015; Toofanian Ross et al., 2015; Williams et al., 2020). No previous study has reported the frequencies of BRIEF–A scores indicating problems with motivation and interest in patients with schizophrenia disorders. The aim of this study was to investigate the rates of possible invalid performance on three PVTs derived from the RBANS in a large sample of patients with schizophrenia spectrum disorders. We further aimed to investigate whether the RBANS PVTs scores were associated with a self-report BRIEF–A measure of motivational problems (i.e., the Initiate Scale), which is a novel approach to investigating negative symptoms in schizophrenia spectrum disorders. A better understanding of the relationship between PVTs and measures of negative symptoms in this patient population may inform clinicians that invalid test performance corresponds to amotivation as a symptom of the disorder, rather than purposeful underperformance.

Based on previous reports of the RBANS PVTs (Bailie et al., 2012; Bayan et al., 2018; Moore et al., 2013; Morra et al., 2015; Toofanian Ross et al., 2015; Williams et al., 2020), we expect that about 20% of the participants will have scores exceeding the PVTs cutoffs. We assume that BRIEF–A scores suggesting problems with motivation and interest will be comparable (i.e., about 40%) to findings from other studies describing the prevalence of problems with motivation in schizophrenia spectrum disorders (Patel et al., 2015; Rabinowitz et al., 2013; Schennach et al., 2015). Even though the RBANS PVTs and the BRIEF–A Initiate Scale differ conceptually and methodologically, they purportedly measure some aspects of patients’ motivation and ability to engage in cognitive testing, so we presume that they should correlate.

Method

Participants

The current study used anonymized archival data from a neuropsychological testing database of 462 patients referred for neuropsychological assessment from psychiatric hospitals in Bergen, Norway. The study is part of a research project that has been evaluated by the Regional Committee for Medical and Health Research Ethics, and by the regional Data Protection Official on behalf of the Norwegian Data Protection Authority (DPA), which is the legislative authority for The Personal Data Act in Norway. Approval from the DPA was granted January 13, 2017. These patients were evaluated for clinical purposes. All patients were informed that the neuropsychological assessment was to be used in diagnostic evaluations and for treatment and rehabilitation planning. They were also informed that participation was voluntary and that they could withdraw from the assessment procedure at any time. All were offered a feedback session and a written neuropsychological report of the assessment findings. Patients were informed that valid test results were contingent on cooperation and motivation to perform to the best of their abilities, but they were not screened for potential external gain or motivation to perform poorly. Inclusion criteria were minimum 18 years of age, Norwegian as their first language, and a confirmed diagnosis of a schizophrenia spectrum disorder or undergoing a diagnostic evaluation due to manifest symptoms of schizophrenia, psychosis, or hallucinations. Diagnoses were according to the International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD–10; WHO, 2016) and were decided by consensus of a team of psychiatrists and certified clinical psychologists. Patients with psychotic symptoms due to known affective disorders were excluded (n = 36). Given the purpose of the current study, inclusion criteria included completion of the RBANS and the BRIEF–A Initiate Scale, resulting in a final sample of 250 participants.

Comorbid problems with substance use were recorded in 94 participants (37.6%). Most patients had long-term problems with polysubstance use, but neither type nor duration of substances used were recorded. Patients with substance use problems were older (M = 25.9 years, SD = 5.9) and had less education (M = 12.1, SD = 1.9) compared to patients without substance use problems (age: M = 23.5 years, SD = 5.3, t(248) = –3.39, p<.001; education: M = 12.8, SD = 1.7, t(248) = 3.04, p = .003). Demographic data for the entire sample are presented in Table 1.

Table 1. Age, education, neuropsychological test results and self-reported executive deficits.

	N	Mean	SD	Minimum	Maximum
Age in years	250	24.37	5.67	18	51
Education in years	250	12.48	1.84	9	18
NART predicted FSIQ	176	100.73	4.28	92	117
WAIS-IV FSIQ	49	85.37	11.66	69	112
RBANS Indices
Immediate Memory Index	250	80.52	20.36	40	133
Visuospatial Index	250	88.39	16.10	40	110
Language Index	250	79.54	17.12	40	121
Attention Index	250	71.14	18.81	40	118
Delayed Memory Index	250	81.00	21.14	40	131
Total Scale	250	71.20	18.81	40	117
BRIEF-A Initiate Scale	250	65.23	13.45	37	89

Note. NART = National Adult Reading Test; WAIS-IV FSIQ = Wechsler Adult Intelligence Scale, Fourth Edition Full Scale Intelligence Quotient; RBANS = Repeatable Battery for the Assessment of Neuropsychological Status; BRIEF-A = Behavior Rating Inventory of Executive Function-Adult Version.

Measures

All patients completed the authorized Norwegian version (Nicholas & Solbakk, 2006) of the BRIEF–A (Isquith et al., 2005) and the Norwegian version of the RBANS (Randolph, 2013). The BRIEF–A is a self-report standardized inventory that measures behaviors associated with executive functions in daily life. It consists of 75 items which yields nine theoretically and empirically derived scales (Inhibit, Self-Monitor, Plan/Organize, Shift, Initiate, Task Monitor, Emotional Control, Working Memory, and Organization of Materials), two broader indices (Behavioral Regulation and Metacognition), and an overall summary score (Global Executive Composite). T scores ≥ 65 are considered clinically significant. The BRIEF–A also includes three validity scales, with cutoffs based on infrequent raw scores in the normative sample and clinical samples (i.e., Negativity ≥ 6; Inconsistency ≥ 8; Infrequency ≥ 3). The Norwegian version applies U.S. normative data, which includes U.S. men and women from ages 18 to 90 years and from a wide range of ethnic and educational backgrounds, as well as geographic regions matched to U.S. census data (Isquith et al., 2005). The BRIEF–A takes about 10–15 minutes to administer.

The RBANS yields five age-corrected Index scores with a mean score of 100 and standard deviation of 15 (Immediate Memory, Visuospatial/Constructional, Language, Attention, and Delayed Memory), as well as a Total Scale score. The Norwegian version of the RBANS applies Scandinavian norms (Randolph, 2013), and takes about 20–30 minutes to administer. Each patient was administered the RBANS (Form A: n = 159 and Form B: n = 91), and the BRIEF–A using the standardized instructions in the manuals. A subset of the participants (n = 176) completed the Norwegian research version of the National Adult Reading Test (NART), which provides age adjusted measures of estimated full scale intelligence quotient (FSIQ; Sundet & Vaskinn, 2008; Vaskinn et al., 2020). FSIQ level was measured by the Norwegian version of the Wechsler Adult Intelligence Scale, Fourth Edition (Wechsler, 2011) in a minority of participants (n = 49), for most to assure that low intelligence would not be a formal hindrance to stand trial, to make decisions of treatment options or to self-discharge etc.

The RBANS Effort Index (EI; Silverberg et al., 2007), RBANS Effort Scale (ES; Novitski et al., 2012), and RBANS Performance Validity Index (PVI; Paulson et al., 2015) were computed as described by the authors. These measures are primarily derived from raw scores on the RBANS subtests List Recognition, a forced-choice recognition task, and Digit Span, a task in which participants immediately repeat a string of numbers read by the examiner. List Recognition and Digit Span tests are somewhat insensitive to a wide range of cognitive disorders and poor performances on these tests have been considered to be a type of embedded validity indicator ( Iverson et al., 1994; Iverson & Tulsky 2003; Miele et al., 2012; Shura et al., 2020; Silverberg et al., 2007).

The EI is calculated by assigning raw scores on the Digit Span and List Recognition subtests a weighted score ranging from 0–6. The weighted scores are based on the frequency of raw scores in clinical populations (i.e., the raw scores associated with the following percentile ranges: 0, 0.1–1.9, 2–4.9, 5–8.9, 9–15.9, 16–24.9, and ≥25). Less frequent scores have higher weighted scores. The sum of the two weighted scores is the EI score. An EI score of >0 was found to be optimal compared to scores indicating invalid performance on the Test of Memory Malingering (TOMM; Tombaugh, 1997) in a sample of people with mild traumatic brain injury (mTBI) versus three malingering groups (i.e., a clinical sample consisting of probable or definite neurocognitive malingering and two samples of healthy participants instructed to malinger; Silverberg et al., 2007). Based on these findings, and the frequencies of EI scores in a heterogenous sample of clinical patients with cognitive impairment and no evidence of invalid test performance, Silverberg et al. (2007) suggest that EI scores >3 should be considered indicative of invalid performance in clinical populations referred for neuropsychological testing.

The ES is calculated by subtracting the sum of raw scores of the RBANS free recall subtests (i.e., List Recall, Story Recall, and Figure Recall) from the sum of raw scores obtained on List Recognition and Digit Span. Novitski et al. (2012) noted that ES scores <12 occurred in 15.1% in the RBANS standardization sample, and found this cutoff to have excellent discriminability between a sample of patients with mTBI scoring below the standard cutoff scores on a free-standing PVT and an amnestic sample consisting of probable Alzheimer’s disease and amnestic mild cognitive impairment (aMCI). Novitski et al. (2012) caution that the ES will produce high false positive rates in people with adequate free recall performance and should be limited to cases where there is evidence of cognitive impairment or possible invalid test performance. They suggest that calculating ES scores should be limited to participants having Digit Span raw scores of <9, List Recognition raw scores of <19, or sums of Digit Span and List Recognition raw scores of <28.

In constructing the PVI, Paulson et al. (2015) performed a series of independent-samples t-tests to identify which RBANS subtests participants with valid responding performed better on, compared to participants with invalid responding based on the TOMM and behavioral criteria. Valid responders had better performance on the RBANS subtests Digit Span, List Recall, Story Recall, Figure Recall, and List Recognition. The PVI score equals the sum of raw scores of these subtests. Paulson et al. (2015) found that invalid test performance responding was optimally identified by PVI scores <42.

Statistical analyses

Frequency analysis tables were used to compute base rates of raw scores on the RBANS subtests Digit Span and List Recognition; the sum of Digit Span and List Recognition raw scores; scores on the EI, ES, and PVI; and the BRIEF–A Initiate Scale. Spearman’s rank order correlations were computed between the EI, ES, PVI, BRIEF–A Initiate Scale, RBANS Indices and Total Scale score, NART FSIQ and WAIS-IV FSIQ, and years of age and education. In the correlation analysis, we included an aggregate measure of global performance that excluded the Digit Span and List Recognition subtests, by summing the RBANS Index scores to which neither of these subtests contributes (i.e. the Immediate Memory, Visuospatial/Construction, and Language indices), as described in Silverberg et al. (2007). We then repeated the Spearman’s rank order correlation computations for participants having a T score ≥65 (n = 128) on the BRIEF–A Initiate Scale, in effect including only those participants acknowledging clinically significant initiation problems.

Results

Descriptive statistics for age, years of education, NART estimated FSIQ, WAIS-IV FSIQ, the RBANS Indices and Total Scale score, and BRIEF–A Initiate Scale are presented in Table 1. We note that mean NART estimated intelligence was in the average range and that WAIS-IV FSIQs were in the low average range (i.e., a standard deviation below the normative mean). The means for the RBANS Indices and Total Scale score indicate that patients had, on average, mild to moderate cognitive deficits. There were no differences on any of these measures between patients with and without comorbid substance use problems (p>.05).

Most patients were able to answer all BRIEF–A items. Having two or more unanswered items was uncommon and occurred in just 1.6% of participants. On the BRIEF–A validity measures, 3.6% had scores exceeding the cutoff for inconsistent responding, 2.8% had scores exceeding the cutoff for a negative response pattern, and 0.4% had scores exceeding the cutoff for an unusual response pattern. On the BRIEF–A Initiate Scale, 51% had a T score above the suggested clinical cutoff (i.e., T ≥ 65); 36% had a T ≥ 70 and 19% had a T ≥ 80. The maximum T score is 89, which was obtained by 1.2%, whereas T scores ≤50 occurred in 17.2%.

The frequencies of low raw scores on the Digit Span and List Recognition, the frequency of scores <28 for the sum of Digit Span and List Recognition raw scores, EI, ES, and PVI scores exceeding the cutoffs in the current sample are presented in Table 2. We also report (a) the corresponding frequencies of low raw scores on the Digit Span and List Recognition, and frequencies of EI scores in the EI derivation sample (Silverberg et al., 2007); (b) the frequencies of low raw scores on the Digit Span and List Recognition, the frequency of scores <28 for the sum of Digit Span and List Recognition raw scores, and ES scores <12 from the RBANS standardization sample (Novitski et al., 2012); and (c) frequencies of EI scores >0, ES scores <12, and PVI scores <42 in the PVI derivation sample (Paulson et al., 2015) in Table 2 for comparison. In the current sample, 128 participants (51%) had either a raw score of <19 on List Recognition, a Digit Span raw score of <9, or a combined List Recognition + Digit Span raw score of <28. Using the criterion of List Recognition + Digit Span raw scores of <28 as an indicator for calculating an ES score as suggested by Novitski et al. (2012), 73 (29%) participants qualified for calculating an ES score, of which 70 had an ES score <12, constituting a frequency of invalid ES scores of 28% in the entire sample. The EI has several suggested cutoffs. Silverberg et al. (2007) suggested a cutoff score >0 in post-acute mild TBI cases and a cutoff score of >3 for populations referred for neuropsychological assessment. A cutoff of >4 has been suggested for older and more cognitively impaired patients with schizophrenia disorders (Moore et al., 2013). Using the cutoffs of >0, >3, and >4, 28%, 6%, or 3% of participants, respectively, had EI scores exceeding these cutoffs.

Table 2. Frequency distributions of measures of the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) Performance Validity Tests in Clinical populations and Standardization Sample.

Cutoff Scores	Schizophrenia Spectrum Disorders (current study; N = 250)	Effort Index Derivation Sample (N = 103)	RBANS U.S. Standardization Sample (N = 540)	Performance Validity Index Derivation Sample (N = 234)
Digit Span raw scores
<9	40%	32%	23%	–
<8	21%	11%	7%	–
<7	11%	8%	–	–
List Recognition raw scores
<19	22%	39%	14%	–
<18	13%	24%	7%	–
<17	8%	16%	–	–
<16	5%	10%	–	–
<15	2%	8%	–	–
Digit Span + List Recognition
<28	29%	–	17%	–
Effort Index
>0	28%	34%	–	48%
>1	24%	25%	–	–
>2	14%	16%	–	–
>3	6%	6%	–	–
>4	3%	–	–	–
Effort Scale
<12	28%	–	15%	32%
<1	16%	–	–	–
<−5	6%	–	–	–
Performance Validity Index
<53	29%	–	–	–
<50	16%	–	–	–
<42	6%	–	–	35%

Note. RBANS = Repeatable Battery for the Assessment of Neuropsychological Status. The Schizophrenia Spectrum Disorders column designates frequencies observed in the current study, the Effort Index Derivation Sample was reported in Silverberg et al. (2007), the RBANS U.S. Standardization Sample was reported in Novitski et al. (2012), and the Performance Validity Index Derivation Sample was reported in Paulson et al. (2015).

Non-parametric correlations (i.e., Spearman’s rho) for the entire sample are presented in Table 3. Only the EI had a significant correlation with the BRIEF–A Initiate Scale, albeit very small (rho = .158, p<.05). When including only participants that had acknowledged clinically significant initiation problems (i.e., those with T scores ≥65 on the BRIEF–A Initiate Scale; n = 128), none of the RBANS PVTs were significantly correlated with the BRIEF–A Initiate Scale (EI: rho = .112, p = .304; ES: rho = .091, p = .509; PVI: rho = –.018, p = .870). The new RBANS aggregate global score, which excludes the Digit Span and List Recognition subtests, had lower correlation coefficients with all three PVTs (EI: rho = –.400, p<.001; ES: rho = –.633, p<.001; PVI: rho = .671, p<.001) compared to the RBANS Total Scale score (EI: rho = –.484, p<.001; ES: rho = –.762, p<.001; PVI: rho = .826, p<.001). The ES (rho = –.906, p<.001) and PVI (rho = .951, p<.001) had high correlations with the RBANS Delayed Memory Index compared to the EI (rho = –.456, p<.001). All PVTs correlated with years of education (EI: rho = –.186, p<.001; ES: rho = –.239, p<.001; PVI: rho = .212, p<.001) and the ES with age (rho = .243, p<.001). The EI and PVI correlated with NART predicted FSIQ (EI: rho = –.285, p<.001; PVI: rho = .331, p<.001) and WAIS-IV FSIQ (EI: rho = –.556, p<.001; PVI: rho = .511, p<.001).

Table 3. Spearman’s Rank Order Correlations Matrix.

	1.	2.	3.	4.	5.	6.	7.	8.	9.	10.	11.	12.	13.	14.
1. Effort Index	–
2. Effort Scale	.192*
3. Performance Validity Index	–.574**	–.864**
4. BRIEF-A Initiate Scale	.158*	.052	–.073
5. Immediate Memory	–.407**	–.647**	.710**	–.074
6. Visuospatial/ Constructional	–.260**	–.328**	.356**	–.077	.222**
7. Language	–.214**	–.312**	.353**	–.071	.429**	.189**
8. Attention	–.397**	–.420**	.563**	–.113	.456**	.325**	.390**
9. Delayed Memory	–.456**	–.906**	.951**	–.059	.713**	.357**	.347**	.514**
10. Total Scale	–.484**	–.762**	.826**	–.103	.790**	.540**	.621**	.734**	.822**
11. Aggregate Global Performance	–.400**	–.633**	.671**	–.105	.793**	.601**	.730**	.536**	.677**	.920**
12. Age in Years	–.075	.243**	–.081	–.177**	.018	.065	.068	.010	–.120	.001	0.067
13. Years of Education	–.186**	–.239**	.212**	–.128*	.341**	.178**	.199**	.350**	.224**	.345**	.342**	.215**
14. NART predicted FSIQ	–.285**	–.163	.331**	–.088	.398**	.237**	.315**	.333**	.276**	.482**	.481**	.148	.554**
15. WAIS-IV FSIQ	–.556**	–.247	.511**	.053	.374**	.261	.468**	.639**	.469**	.621**	.549**	–.065	.279	.516**

Note. N = 250 for all variables except for the ES (n = 128). BRIEF-A = Behavior Rating Inventory of Executive Function-Adult Version; Aggregate Global Performance = the sum of Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) Immediate Memory, Visuospatial/Construction, and Language indices. NART = National Adult Reading Test; WAIS-IV FSIQ = Wechsler Adult Intelligence Scale, Fourth Edition Full Scale Intelligence Quotient;.

* Correlation is significant at the 0.05 level (2-tailed).

** Correlation is significant at the 0.01 level (2-tailed).

A standard linear multiple regression was used to explore how much of the variance of BRIEF–A Initiate Scale scores were explained by the RBANS PVT scores (i.e., the EI, ES, and PVI). Preliminary analyses were conducted to ensure no violation of normality, linearity, multicollinearity, and homoscedasticity. The inclusion of the PVI in the model resulted in unacceptable multicollinearity, and this variable was excluded from the regression analysis. The regression analyses revealed that the PVTs did not explain a significant amount of variance in BRIEF–A Initiate Scale scores, F(2, 125) = 1.706, p = .186, R² = .027, R²_Adjusted = .011, or when including only those participants having BRIEF–A Initiate Scale scores of T ≥ 65, F(2, 68) = .630, p = .535, R² = .018, R²_Adjusted = –.011.

Discussion

The aim of this study was to investigate the frequency of several measures suggested as indicators of problems with motivation, interest, and performance validity in schizophrenia spectrum disorders derived from neuropsychological test scores (i.e., RBANS) in a large sample of younger patients with schizophrenia spectrum disorders, and to compare these scores with measures of apathy derived from a self-report symptom questionnaire on executive functions in everyday life (i.e., BRIEF–A Initiate Scale). About 28% of the current sample had scores indicating possible invalid test performance on some of the RBANS PVTs, which was close to the hypothesized frequency (i.e., ∼20%) and is about twice as frequent than the prevalence of the same scores in the U.S. standardization sample (Novitski et al., 2012). Depending on which PVT and cutoff we chose, the prevalence of scores exceeding cutoffs varied from 3% (i.e., EI scores >4), 6% (i.e., PVI scores <42), to 28% (i.e., ES scores <12), which is not substantially different from previous studies of the RBANS PVTs in schizophrenia spectrum disorders (Bayan et al., 2018; Moore et al., 2013; Morra et al., 2015). In the current sample, an ES cutoff of <12 does not yield substantially different rates of scores exceeding the cutoff in comparison to a combined raw score on List Recognition and Digit Span score of <28 or an EI score >0. In the present study, an EI score >3 (occurring in 6% of our sample) seems appropriate, and this is also the most used cutoff in samples of people with schizophrenia (Morra et al., 2015).

The prevalence of scores above the suggested clinical cutoff on the BRIEF–A Initiate Scale was 51%, which was somewhat higher than we hypothesized. The correlations of self-reported initiation problems and empirically derived PVTs were either non-significant, or too small to be of any probable clinical significance, even when including only those participants that had BRIEF–A Initiate Scale scores in the clinical range (T score ≥65). Further, the results of the regression analyses suggest that the BRIEF–A Initiate Scale and the RBANS PVTs are measuring different constructs. There is some evidence that elevated BRIEF–A scores are more related to emotional distress and psychiatric problems than performance on neuropsychological tests (Donders et al., 2015; Donders & Strong, 2016; Hanssen et al., 2014; Løvstad et al., 2012, 2016; Shwartz et al., 2020).

All RBANS PVTs had moderate to high correlations with measures of cognitive impairment as measured with the RBANS Total score and the new RBANS aggregate global score, a score that does not include the subtests (i.e., Digit Span and List Recognition) used to calculate the PVTs scores. The ES and the PVI rely more on raw scores that are used to compute the RBANS Delayed Memory Index compared to the EI, which accounts for their high correlations with that index. The association with cognitive impairment is to be expected. Low motivation, insufficient exertion, or other behaviors that might underlie low scores on embedded PVTs will also result in lower scores on other RBANS subtests. On the other hand, cognitive impairment, by definition, results in low test scores, including those subtests comprising the RBANS PVTs (Burton et al., 2015; Goette & Goette, 2019; Hook et al., 2009; Morra et al., 2015; Shura et al., 2018). It has been suggested that cognitive deficits in schizophrenia spectrum disorders could contribute to motivational deficits, in addition to, or instead of, motivational deficits contributing to low cognitive performances (Barch, 2005). We cannot assume that the RBANS PVTs are a direct and precise measure of “effort,” or “motivation;” they are cognitive tests. The associations of all three RBANS PVTs with years of education, measures of intelligence, and cognitive impairment indicates that patients with genuine cognitive impairment, or patients with lower intelligence and/or lower educational attainment, will have a greater likelihood of obtaining scores that exceed cutoffs on the RBANS PVTs (Burton et al., 2015; Duff et al., 2011; Goette & Goette, 2019; Hook et al., 2009; Morra et al., 2015; O'Mahar et al., 2012; Shura et al., 2018). Given these associations, we cannot rule out that the RBANS PVTs are simply measuring levels of cognitive functioning in the current sample, rather than decreased motivation and interest, particularly considering that the patients’ self-report of initiation problems do not correlate with the RBANS PVTs.

Although the RBANS PVTs can alert the clinician that insufficient engagement and reduced motivation might have influenced test performance, these measures are perhaps more helpful for inferring valid test performance (Bayan et al., 2018; Lippa et al., 2017). That is, a patient that obtained a RBANS PVT score in the acceptable range has probably had reasonable and sufficient engagnement in the cognitive tests during assessment, regardless of the level of cognitive impairment. The RBANS PVTs can thus aid the clinician in ruling out that patients’ problems with decreased motivation, engagement, and interest had substantial effect on test scores.

Limitations

The present study has several limitations. Adding a standalone PVT would have enabled much more detailed analyses, including analyses of classification accuracy for the embedded PVTs and the associated confidence intervals for different cutoffs of the RBANS PVTs. Embedded PVTs are generally considered less sensitive than standalone PVTs in some studies (Armistead-Jehle & Hansen, 2011; Miele et al., 2012; Riordan & Lahr, 2020). That said, the RBANS embedded PVTs have been criticized for yielding too many false positives in people with a schizophrenia diagnosis, particularly forensically committed inpatients (Williams et al., 2020). The addition of other measures of apathy and avolition, such as the SANS, PANSS, or NSA, would allow for better comparisons with previous studies investigating the RBANS PVTs and negative symptoms in schizophrenia spectrum disorders. Further, we have no information regarding patients’ use of psychotropic medications, and thus we cannot examine possible associations between medication use, embedded PVT scores, and cognitive functioning (Ballesteros et al., 2018; MacKenzie et al., 2018). The use of U.S. normative data for the BRIEF–A in a Norwegian patient sample, might also be problematic, because studies conducted in Norway have found that healthy respondents can have mean scores 0.5 to 0.75 SDs below the U.S. normative means (Grane et al., 2014; Løvstad et al., 2016; Sølsnes et al., 2014). A matched control group could have remediated most of these limitations.

Conclusions

In principle, the RBANS allows for the concurrent assessment of cognitive deficits and motivational problems in patients with schizophrenia spectrum disorders. By combining the RBANS and BRIEF–A in a neuropsychological assessment, the clinician can evaluate the degree of cognitive impariment, potential invalid test performance, self-reported initiaton problems, and other behaviors associated with executive functions in daily life, in less than 60 minutes. The majority of participants could complete the RBANS (i.e., 72–94%) and the BRIEF–A (i.e., 96%) without obtaining low scores on the validity indicators, suggesting that these assessment tools are well within the capabilities of most patients seen with schizophrenia spectrum disorders.

Acknowledgments

The authors are indebted to Torill K. Augustin for her contributions to data collection.

Disclosure statement

Grant Iverson, Ph.D. has received research funding from several test publishing companies, including ImPACT Applications, Inc., CNS Vital Signs, and Psychological Assessment Resources (PAR, Inc.). He acknowledges unrestricted philanthropic support from ImPACT Applications, Inc., the Mooney-Reed Charitable Foundation, and the Spaulding Research Institute.