Official position of the american academy of clinical neuropsychology on serial neuropsychological assessments: the utility and challenges of repeat test administrations in clinical and forensic contexts

Abstract

Serial assessments are now common in neuropsychological practice, and have a recognized value in numerous clinical and forensic settings. These assessments can aid in differential diagnosis, tracking neuropsychological strengths and weaknesses over time, and managing various neurologic and psychiatric conditions. This document provides a discussion of the benefits and challenges of serial neuropsychological testing in the context of clinical and forensic assessments. Recommendations regarding the use of repeated testing in neuropsychological practice are provided.

Keywords:

• Serial test administration
• Serial testing
• Repeat testing
• Test-retest

The value of serial neuropsychological evaluations is recognized in numerous clinical and forensic settings. Serial assessments are now common, and even routine, in some populations and settings, and tracking neuropsychological strengths and weaknesses over time has proven invaluable in the care of patients with developmental disorders, traumatic brain injury, acquired neuropathology, chronic illness, and progressive dementia. Serial testing is also common in forensic settings. This document is intended to provide a brief discussion or benefits and challenges of serial neuropsychological testing in clinical and forensic contexts. Recommendations regarding the use of repeated testing in neuropsychological research and practice are provided.

Benefits of serial neuropsychological testing

Serial neuropsychological assessment has many known benefits. Most notably, comparisons of data points can serve to document the natural progression or resolution of illness and injury (e.g., McCrea et al., 2003), which is often critical to differential diagnosis and treatment recommendations. Repeat testing can also be used to measure the clinical effectiveness of pharmacological (e.g., Loring & Meador, 2004), surgical (e.g., Blumenthal et al., 1995; Ghogawala, Westerveld, & Amin-Hanjani, 2008; Rasmussen, Siersman, & The ISPOCD Group, 2004), rehabilitation (Cicerone et al., 2000, 2005), and educational interventions (Feifer, 2008; Rabiner & Malone, 2004) in patient populations, or the effects of medications in both normal and clinical populations (e.g., Beglinger et al., 2004; Rapport, Quinn, DuPaul, Quinn, & Kelly, 1989). Although interpreting data from repeat testing can be particularly complex in populations undergoing active or dynamic cognitive change (e.g., children), serial evaluation has utility in monitoring maturational progression by detecting slowed, arrested, or uneven development or the regression of previously acquired skills (Shapiro, Lockman, Balthazor, & Krivit, 1995). Repeat testing is common in the special education setting not only for documenting treatment efficacy but also for enabling decisions whether or not continuation of services for the child is warranted.

Finally, much as medical laboratory tests are repeated to confirm initial findings, serial neuropsychological tests have value in determining whether initial findings were definitive or accurate (resulting in a correct diagnosis/recommendation), or inaccurate due to confounding examinee, examiner, or environmental factors. In a forensic context, retesting may serve these same purposes and, in addition, can assist in determining whether the pattern of performances across time follows the expected course of recovery from an injury, as well as possibly providing insight into the credibility of the examinee's presentation.

Challenges in serial neuropsychological testing

Repeated observations and longitudinal comparisons are among the most powerful tools in science. Yet, despite its many merits, serial neuropsychological assessment gives rise to unique challenges. At a minimum, repeated testing yields two sets of scores, each of which is comprised of systematic (ability- and procedure-related) variance and error variance attributable to specific factors affecting the examinee, examiner, environment, and context of each individual testing session.

The primary element requiring careful consideration in serial testing is the practice effect, which refers to the improvement in performance on retesting that is due to previous exposure to the same or similar neuropsychological measure rather than to a true change in the individual's ability (cf. Kaufman, 1990). Both methodological (e.g., statistical regression to the mean) and person-specific (e.g., motivation) factors may have explanatory power in understanding the meaning of a change, or a lack of change, on repeated testing. The effects of age, education, and gender may also influence the degree of change in complex ways. It is the examiner's responsibility to judge which factor(s) may influence performance in a scientifically sound way.

A change in scores on repeat testing can lead to different interpretations and can result in diagnostic disagreements, including attributing improved scores to recovery of function or efficacy of intervention when the results are more likely to reflect the effects of test practice. Similarly, the absence of expected practice effects may in one instance reflect insufficient effort, and in another instance may reflect the deleterious effects of an illness or injury. Neuropsychologists who conduct a re-assessment need to be mindful and knowledgeable about, the variables potentially affecting change, including practice effects (i.e., gains related to prior exposure to a test), psychometric factors (test reliability), patient characteristics (demographics, state variables, fatigue, motivation), and disease state of the individual being assessed, and about resulting interpretive complications (Attix et al., 2009).

It is important to recognize that the sources of change in repeated assessments can be multi-factorial and usually do not relate to a single construct. Statistical methods that can aid the interpretation of change in neuropsychological test scores include reliable-change index (RCI) scores using the standard error of differences between test and retest scores and standardized regression-based (SRB) change scores. SRB approaches involve the use of regression models to predict retest scores using formulae that account for baseline performance and demographic or other variables that may impact the rate of test–retest change (Chelune, 2003). These normative SRB equations can then be used to determine whether the patient's observed rate of change deviates significantly from expectation based on the change predicted from the model (see Hinton-Bayre, 2010, for a systematic explication of the calculations and implications of using different SRB models). However, for most tests currently in use, more empirical research is needed to identify relevant clinical change scores and to facilitate their application in clinical and forensic practice (cf. Maassen, Bossema, & Brand, 2009).

Clinical context

The effects of repeated examinations have been studied in both normal (e.g., Collie, Maruff, Darby, & McStephen, 2003) and clinical samples (Dikmen, Heaton, Grant, & Temkin, 1999; Iverson, 2001; McCaffrey, Duff, & Westervelt, 2000a, 2000b; McCaffrey & Westervelt, 1995; Moore et al., 1990; Wilson, Watson, Baddeley, Emslie, & Evans, 2000), generally demonstrating increases in test scores as a result of repeated exposure. Tests requiring speed or an unfamiliar or infrequently practiced mode of response, and those with a single solution are more likely to show significant practice effects when they are repeated (Basso, Bornstein, & Lang, 1999; McCaffrey, Ortega, Orsillo, Nelles, & Haase, 1992). Within testing domains (cf. executive function), measures may vary considerably with regard to the degree of practice effect (Basso et al., 1999). Practice effects are especially challenging—yet important—to consider in serial memory testing because memory measures have been found to produce particularly poor test–retest reliabilities (Dikmen et al., 1999). With the knowledge that memory measures generally have lower reliability, and that subsequent exposures to the same memory stimuli represent additional learning trials, Dikmen et al. (1999) recommend use of multiple memory measures and reliance on composite scores, rather than subtest scores. Several studies have shown that scores may be enhanced on retest even when the test items are different (Benedict & Zgaljardic, 1998; Wilson et al., 2000). This can occur even with the use of alternate forms, as the examinee benefits from knowing how to approach the task more effectively (i.e., the examinee has acquired a test-taking response set or strategy or has become more familiar with the testing procedure). Of course, factors such as the length of the test–retest interval may affect the extent to which the follow-up score may be inflated secondary to the effects of repeated test exposure.

The number of repeat testing sessions is also a factor that deserves consideration, as the degree of gain associated with practice may not be equivalent across multiple repeated testings. With brief test intervals, gains have been reported to stabilize after an initial practice effect on measures of attention and processing speed (Falleti, Maruff, Collie, & Darby, 2006). Similarly, tests with a clear ceiling effect may show their strongest practice effects between the first and second test administrations (Benedict & Zgaljardic, 1998; Ivnik et al., 1999; Rapport et al., 1997), with little to no improvement thereafter. In research settings one can correct for an initially strong practice effect by providing two or more baseline evaluations before introducing the experimental condition (McCaffrey & Westervelt, 1995). Although some data on test–retest changes are available for a limited number of measures, the field would benefit from the development of more extensive normative change data for use in clinical and research settings that require multiple retesting sessions. (Attix et al., 2009).

Large score changes across test administrations are generally not common in patients who have put forth good effort, except for single-solution tests and tests with a significant learning and memory component (Dikmen, Machamer, Temkin, & McLean, 1990; McCaffrey et al., 2000b). Just as baseline interpretations are strengthened when confirmed by multiple complementary findings, so too are conclusions about change in cognitive functions strengthened by identification of change trends, especially when convergent changes are evident on multiple measures. However, it is critical to appreciate that the degree of gain expected with repeated test exposure varies across neuropsychological tests. Not surprisingly, the underlying variables impacting change vary across measures, and these also may differ from the variables impacting baseline performance (Attix et al., 2009). For instance, while age, education, and gender might affect initial performance on a measure, change on that measure might be impacted by only age. This is especially true in pediatric evaluations where the variability in test performance can be inversely related to age, i.e., less variability with increasing age. Variability in test performance, as indexed by standard deviation units, is often greatest in very young children (before the age of 5) with a slow but steady reduction in performance variability that continues throughout childhood and into adolescence, at least on measures of functions that are linearly consistent across development and those that have limited floor and ceiling effects. However, at ages that involve transitions between versions of test measures (e.g., transitioning between preschool and school-age versions of intelligence tests at 6–7 years of age), variability in test scores may be small due to ceiling/floors of the measure for that age. In addition, as brain maturation does not necessarily involve purely linear trends for all cognitive skills, but rather may proceed in rapid developmental progression (i.e., growth spurts), increased variability may be seen for some measures collected during normal periods of accelerated development. For example, increased temporal fluctuation in performance is common on measures of executive functions when sampled during early or pre-adolescent phases during which these skills undergo accelerated development (P. Anderson, V. Anderson, & Garth, 2001; Huizinga, Dolan, & van der Molen, 2006). Such effects, coupled with practice gains, can result in different change trajectories across measures.

Additional consideration is inherently relevant to repeated evaluation of infants and preschool children. Children at such ages often display reduced control over their attention and behavior, and are more heavily influenced by their external environment than older children (Lopez, Menez, & Hernandez-Guzman, 2005; Mahone, 2005). Familiarity with test stimuli can negatively influence performance as the stimuli are viewed as less novel and less stimulating, leading to decreased interest and attention by the young child (Courage, Reynolds, & Richards, 2006; Rothbart & Posner, 2001; Sheese, Rothbart, Posner, White, & Fraundorf, 2008).

Finally, in addition to considering whether or not normative change data are available, one must also consider the cognitive process being measured and how this may change with repeated assessments. It is doubtful, for example, that a repeated administration of a straightforward problem-solving task is measuring the same cognitive process as the initial test administration. Indeed, effortful information processing evolves into perceptual speed processing with practice. Therefore, the effects of repeated exposure on the cognitive process (not just the test) under study need to be considered as important variables when interpreting the meaning of the second test score.

In the case of documented severe brain damage, an individual is less likely to improve with practice alone and improvements may be minimal. The degree to which brain injury impacts change may vary with the nature, site, severity, and chronicity of the brain damage in combination with the patient's age, although the influence of these factors has yet to be elucidated. Measuring change after pediatric acquired insult can be particularly challenging, as the child is not only attempting to recover previously developed skills, but also acquire new age-expected skills as well. Test practice effects that are less than expected can reflect deterioration in skills, failure to develop at age-appropriate rates, or the inability to manage age-related increases in complexity on specific tests (Taylor & Alden, 1997). Moreover, the characteristics of the test can also determine whether or not performance of patients with brain damage will improve with repetition. For example, Wilson et al. (2000) reported that reaction time shows little practice effect, even over as many as 20 assessments. For these reasons, clinicians must carefully consider how change trajectories might be influenced by the illness or injury under study. Both the nature of the patient's cognitive deficits and the nature of the test can impact change, as might their interaction. Ideally, studies would be available using repeated testing of both normal individuals and clinical samples. Such studies might highlight the tests that are most vulnerable to change with repeated administration and which patient groups tend to be most susceptible to change with repeated test administrations (McCaffrey et al., 2000a, 2000b).

Repeated testing can shed light on the validity of responses (to be discussed in greater detail in the forensic context section below). When two assessments are made using the same tests, poor test-taking effort may be reflected in the performance, such as the absence of gains for participants who have demonstrated some learning ability. Similarly, variable effort across tests may result in implausible increases or decreases in scores, very different item responses, or wide variations in intratest response patterns (cf. Lezak, Howieson, & Loring, 2004). Examination of performance reliability can be an important means by which the neuropsychologist can identify patients who are not consistently putting forth adequate effort (Cullum, Heaton, & Grant, 1991). Several tests and test batteries have established test–retest indices that effectively differentiate malingerers from control participants (Dikmen et al., 1999; Strauss et al., 1999). Performance consistency has also been included in some of the specialized validity measures used during a single assessment (Green, Allen, & Astner, 1996).

In clinical research RCI and SRB approaches are particularly well suited for longitudinal studies examining the impact of intervention or natural disease course, and offer a more sophisticated consideration of change beyond subtraction approaches. Such methods have been applied to outcomes research such as cardiac interventions (e.g., Collie, Darby, Falleti, Silbert, & Maruff, 2002), epilepsy surgery (e.g., Chelune, Naugle, Luders, Sedlak, & Awad, 1993), post-operative cognitive dysfunction (Farag, Chelune, Schubert, & Mascha, 2006), and aging (Duff et al., 2005). In addition to direct comparisons of scores indicating deviation from expected change across conditions, these methods can be used for the classification of outcomes (e.g., cases demonstrating change 1 SD below expected change are considered a poor outcome, while those above are not) or in the construction of indices to use as predictors of other outcomes.

Forensic context

In addition to the usual factors encountered in clinical evaluations, serial assessments in forensic contexts give rise to a somewhat different set of challenges and benefits. The issue of test practice effects often arises in litigation, especially in personal injury and disability evaluations. For example, a neuropsychologist retained by a defense attorney in civil litigation often wants the opportunity to examine the plaintiff, who has already been evaluated, either in the context of clinical care or by the plaintiff's neuropsychological expert. The issues described above complicate test interpretation regardless of whether test data from a treating psychologist or an expert hired by a plaintiff attorney are made available. A plaintiff attorney may object to a defense expert's examination on the basis that the plaintiff has “already been tested” and no further testing is warranted. Greiffenstein (2009) has reported that some neuropsychologists inappropriately propose a minimum test–retest interval before someone can be examined a second time, seemingly to justify a treating neuropsychologist's argument that practice effects may “mask” or “hide” the underlying neurocognitive deficits they believe to be present. However, such specific suggestions should not be regarded as professional guidelines, as the interval after which practice effects may be minimized varies across measures and there is no consensually agreed-upon minimum test–retest interval between examinations. Frequent serial testing with numerous types of clinical patients and the available relevant literature provides the same means of differentiating real change versus practice effect-related change, whether the individual is evaluated in a forensic context or in a routine clinical context. There simply are no prohibitions of serial testing that are unique to forensic contexts. When such prohibitions are alleged, neuropsychologists can rely on relevant statements of professional organizations to support the position that retesting can be appropriate, necessary and informative. For example, documenting the effects of a patient's performance over repeated evaluations is recommended in the American Academy of Clinical Neuropsychology (AACN) Practice Guidelines for Neuropsychological Assessment and Consultation (AACN, 2007).

Determining causation of deficits is often a focus of forensic neuropsychological assessments and serial testing may improve the prospects of doing so because of its ability to document the temporal course of cognitive strengths and weaknesses. Indeed, comparison of test results from different points in time is one means of assessing the potential role of numerous factors, other than brain damage or dysfunction that can affect neuropsychological test scores. For example, a change in test scores may correspond to progression of disease or recovery of function in an acute or subacute disorder, or to factors associated with interim changes in pain, sleep efficiency, life stressors, medication side effects, or effective treatment of related or unrelated medical conditions.

There are several considerations pertaining to a defense expert examining a plaintiff. One is when the plaintiff was first examined. For example, as time passes following an initial evaluation of an individual, a second evaluation may be necessary to document recovery of function, address the permanence of symptoms, or track progression of medical conditions. A second consideration is the purpose of the prior examination. If the initial evaluation was carried out by a treating psychologist, there are often questions related to the cause(s) of the impairment that have not been addressed in the same manner as they would be in a forensic context. For example, the first examiner may not have had access to the same background information (e.g., academic history, details about medical conditions possibly causing neuropsychological impairments). A retained expert usually requests and has support in acquiring difficult to obtain information in order to assist in a determination of causality and prognosis. A third consideration is whether the test battery from the previous examination was sufficient to address specific issues that are known to arise more frequently in forensic practice as compared to clinical assessment contexts. For example, absence of symptom validity testing in the initial evaluation would preclude the degree of thoroughness and appropriate consideration of the possible negative influences of effort and response bias within a forensic context (Heilbronner et al., 2009).

As Greiffenstein (2009) aptly notes, no one disputes that practice effects pose an interpretive challenge, but recognition of this challenge does not justify the belief that retesting is prohibited within specific time frames. In reality, the scientific literature does not support such prohibitions. Moreover, examination of the effects of repeated test administration can provide useful information in clinical, forensic, and research settings. Whether present or absent, these effects provide another source of data that needs to be incorporated into diagnostic formulations, relying on relevant literature, when available, that addresses change trajectories.

Recommendations

1. Neuropsychologists who perform a repeat examination should address the potential influence of test practice effects, and may choose to report specifically how these effects were estimated or otherwise taken into consideration in interpreting the data and reaching conclusions. Rather than viewing repeat testing as primarily a confound, sound neuropsychological practice is best served when neuropsychologists consider change as a measurable construct to be used to inform the clinical descriptive and diagnostic process. Consideration may be given to the standard error of measurement from a test manual, empirical findings on the expected magnitudes of score increases over a particular interval, or other relevant research on test operating characteristics for the instruments employed in the neuropsychologist's battery.

2. Neuropsychologists remain up-to-date on research addressing test–retest effects, when available and relevant. Empirical data can clarify which tests are most vulnerable to practice effects, as well as which clinical samples tend to be more or less likely to display to such effects.

3. Neuropsychologists should be familiar with the concept and statistical methods related to reliable change indices (RCI) and regression-based approaches to change measurement and their associated risks and benefits when attempting to account for practice effects.

4. There is an obvious need for more data on normal change trajectories for all types of measures with all types of demographic variables and patient groups. This is especially salient in pediatric neuropsychological assessments where cognitive development, practice, and recovery can become potentially confounded in repeated assessments. Because of the inherent neurobiological and contextual change that is particularly relevant to both early development and aging, repeat assessment is often essential in identifying atypical maturational progression and in planning and evaluating treatment across different developmental stages of the lifespan.

5. In a forensic context, neuropsychologists acting as experts should be aware of the relevant factors affecting change as in clinical settings. More data and extra-test information (e.g., effort) should be considered in forensic settings to assist in the interpretation of test–retest findings.

6. There are no empirical data allowing the development of clinical guidelines regarding minimum test–retest intervals in clinical or forensic settings. In a forensic context, if confronted by an opposing expert who advocates for fixed retesting intervals, the neuropsychologist should be prepared to educate the court (or referral source) on the state of the art and science. Measurement of practice effects represents valuable data bearing on a person's capacity for learning and adaptation. Making clinical sense out of practice effects requires interpretation just as much as any single score. Neuropsychologists are qualified to interpret the significance of test–retest differences, and are especially equipped by virtue of their knowledge of test operating characteristics to understand the many variables contributing to test–retest change over time.

Acknowledgment

The authors gratefully acknowledge the assistance of Kenneth M. Adams, Manfred F. Greiffenstein, Glenn J. Larrabee, Dean Beebe, and Michael Kirkwood for their helpful comments on previous versions of this manuscript.