Introduction to the Special Issue - Neuropsychology from a distance: Psychometric properties and clinical utility of remote neurocognitive tests

Keywords:

• assessment
• technology
• digital neuropsychology
• remote neurocognitive testing
• teleneuropsychology
• web-based cognitive testing
• mobile cognitive testing

Introduction

In the late 20^th and early 21^st centuries, digital technology has fundamentally altered virtually every aspect of human life. In neuropsychology, computerized cognitive tests have been available in some form for over 30 years (Adams & Brown, 1986; Kane & Kay, 1992), and these measures offer a plethora of advantages when compared with traditional paper-pencil tests (Bilder, 2011; Feenstra et al., 2017; Germine et al., 2019; Miller & Barr, 2017; Parsons, 2016). Consequently, some have criticized the conventional cognitive testing paradigm, with corresponding calls for the field to “get with the times” and more formally adopt digital assessment methods (Bilder, 2011; Germine et al., 2019; Marcopulos & Łojek, 2019; Miller & Barr, 2017; Parsey & Schmitter-Edgecombe, 2013; Parsons et al., 2018). Yet, the status quo of in-person face-to-face testing with so-called “legacy tests” has persisted (Rabin et al., 2014), and many practitioners remain skeptical of major alterations to paper-pencil neuropsychology (Schmand, 2019).

The pushback against digital assessment in neuropsychology is not to be taken lightly, as there are a number of issues and challenges to be addressed as the field gradually moves forward. Some of these issues are technical, whereas others are conceptual and psychometric. As an example of the latter, many computerized assessment tools have only recently been developed and do not yet have strong evidence for reliability and validity. Additionally, most of these measures have been tested primarily in healthy participants, with little data available on their utility in clinical populations. Lastly, although technological advances provide immense potential in the measurement of cognitive functioning, many of the advantages have gone underappreciated and underutilized to date. With this in mind, the purpose of the current Special Issue in the Journal of Clinical and Experimental Neuropsychology is to contribute to the digital neuropsychology literature by presenting evidence related to remote neurocognitive testing methods. First, I will attempt to clarify issues related to terminology in the digital neuropsychology space. Second, I will outline select advantages and issues/challenges of remote testing platforms compared to traditional tests. Finally, I will summarize the five papers in this Special Issue and describe how each paper contributes to the advancement of remote neurocognitive testing.

Terminology

For the purposes of the current discussion, “digital neuropsychology” is considered to be an umbrella term that can be defined as: “the neuropsychological assessment of cognition and behavior using digital tools.” (Germine et al., 2019, p. 272). “Remote neurocognitive testing” (RNCT), which is the focus of the current Special Issue, applies to a subset of approaches within digital neuropsychology. RNCT refers to formal brain-behavior assessment paradigms, administered from a distance, and designed to measure cognitive functioning. RNCT exists on a spectrum, from the use of the internet to administer computerized and legacy tests via videoconferencing (“teleneuropsychology”) at one pole, to the administration of structured but unsupervised computerized tests (“web-based cognitive testing”) in the center, to the use of smartphones and other portable devices for brief, repeated, ambulatory assessments of cognitive functioning in everyday life (“mobile cognitive testing”) at the other pole (see, the Figure 1). RNCT involves direct measurement of cognitive functioning that occurs remotely; consequently, it does not include all aspects of digital neuropsychology. For example, in-person computerized testing (Bauer et al., 2012), in-person virtual reality assessment of real-world functioning (Kourtesis et al., 2020), and passive data collection via wearable technology (Cook et al., 2019) are all outside the scope of the current review.

Figure 1. The continuum of remote cognitive testing.

Teleneuropsychology has been and is currently used clinically, with a surge of interest in the technique following the initial COVID-19 outbreak (Bilder et al., 2020; Brearly et al., 2017; Hammers et al., 2020; Marra et al., 2020; Pritchard et al., 2020; Sweet et al., 2021). Although it has the most immediate and direct application in current clinical practice, teleneuropsychology is heavily constrained by the continued reliance on paper-pencil tests, administered by a neuropsychologist, trainee, or psychometrist. Web-based cognitive testing removes the examiner from the administration/scoring process and relies exclusively on automated digital tests. These online tests have also received interest and attention during the COVID-19 pandemic, although issues related to identity verification, performance validity, device characteristics, and privacy (among others), have limited some neuropsychologists’ enthusiasm (Bilder et al., 2020). Finally, due to nearly ubiquitous smartphone ownership and increasing internet access worldwide (Pew Mobile Fact Sheet, 2021; World Internet Users and Population Stats, 2021), mobile cognitive testing may have the greatest potential of all RNCT methods. However, the vast majority of available ambulatory cognitive tests currently have little psychometric support and no normative data, and they also face similar challenges as web-based tests (e.g., touch latency issues, inattention/distraction during testing; Charalambous et al., 2020; Sliwinski et al., 2018; Swendsen et al., 2019). Therefore, although the ceiling is high, much work remains to be done before mobile cognitive tests can be widely implemented in neuropsychological research and clinical practice.

Advantages

The current discussion of advantages and issues/challenges faced by RNCTs will not be exhaustive, and other sources are available for comprehensive and nuanced analysis (Feenstra et al., 2017; Germine et al., 2019; Miller & Barr, 2017; Parsons, 2016), including technical concerns (Parsons et al., 2018). It is also noteworthy that RNCT is digital by its very nature, and all of the technological advantages of in-person, laboratory computerized tests (Bauer et al., 2012) also apply to RNCT (with the exception of paper-pencil teleneuropsychology). With that in mind, the following are prominent advantages of RNCT compared to traditional approaches: 1) increased access for people who are socioeconomically and/or geographically disadvantaged, 2) the ability to rapidly scale up and test large groups of people, 3) greater flexibility in designing complex stimuli, 4) more precision, reliability, and standardization in stimulus presentation, and 5) easier and more reliable scoring, data management, and data export. Items 1–2 will be discussed first, followed by items 3–5.

Access and scalability

Perhaps the most obvious benefit of teleneuropsychology, web-based testing, and mobile cognitive testing is that they can all reach patients who, for a variety of reasons, are unable to travel to a large, metropolitan medical center, where the majority of neuropsychological clinics are located. Specifically, people who (i) live in rural areas, (ii) live in urban areas but cannot afford transportation, and/or (iii) have difficulty navigating advanced healthcare systems, can all benefit from remote testing. Furthermore, if RNCT is automated and unsupervised (i.e., web-based or mobile cognitive testing), typical testing-related resource constraints are reduced, and large numbers of people can be rapidly examined at relatively low cost (Feenstra et al., 2017). This has important implications for big data science (e.g., genetics research) requiring large sample sizes, where in-person testing is impractical. It can also promote equity in the delivery of neuropsychological services to disadvantaged and underrepresented groups. That is, people who would otherwise be unable to afford in-person testing can access RNCT at a reduced cost, presuming that they have access to a computer or mobile device and an internet connection. Of course, in some low/middle income cities and countries, access to digital devices and wireless networks may be limited, and this needs to be taken into consideration (Fernandez, 2019). Still, compared to the travel and resource-intensive requirements of traditional in-person cognitive testing, RNCT offers a substantial step forward in access and availability of neuropsychology for large groups of people (Bauer et al., 2012; Germine et al., 2019; Miller & Barr, 2017).

Flexibility, consistency, precision, and efficiency

When cognitive tests are developed from the ground up using digital platforms, the design process is freed from the constraints of paper and pencils, and developers are afforded greater flexibility in stimulus presentation that can enhance sensitivity to subtle cognitive impairment (Miller & Barr, 2017). For example, the classic Stroop paradigm, which is available in a variety of legacy formats, has also been computerized into a cued-Stroop task. In contrast to conventional card-reading versions, the computerized test presents stimuli one at a time, with multiple task-related dimensions that can be manipulated in order to tax attentional control resources and prevent examinees from entering into a mental set. Specifically, a cue (the word “color” or the word “word”) is presented on the screen, indicating to which stimulus dimension the examinee should respond. The cue then disappears and is followed by the Stroop stimulus (i.e., color-words that are congruent, incongruent, or neutral). Some iterations of the test also include a short delay period (4–5 seconds) between the cue and stimulus to further challenge the working memory system, leading to the possibility of a 2-cue × 3-congruency × 2-delay (e.g., color-incongruent-long delay) design. In comparison to conventional card-reading versions and other legacy tests, the cued-Stroop task has shown high sensitivity to subtle cognitive inefficiencies such as those present in preclinical Alzheimer’s disease (Balota et al., 2010; Duchek et al., 2009; Hutchison et al., 2010; Van Patten et al., 2018).

In addition to increased flexibility in paradigm construction, digital stimuli are also presented more consistently than are examiner-administered stimuli, without the built-in variability that is inherent in human behavior. For example, whereas computers can maintain precision at millisecond timescales, simple tasks such as stopwatch presses and card flips likely vary by seconds based on examiner-specific factors, although this error has been understudied (Parsons et al., 2018). That is, the variability that is due to examiner age, attention span, fatigue, response consistency, “drift,” and other factors is largely absent from machines, and, when present, can be more easily quantified and addressed (Antoniuk & Cormier, 2020; Germine et al., 2019; Overton et al., 2016). Therefore, digital tests have the potential for tighter standardization, which is of utmost importance in reducing error and increasing confidence in cognitive results. Following test administration, digital devices can also rapidly and automatically score and report test results in an accurate and user-friendly manner (Miller & Barr, 2017). Because the process is not mediated by paper and pencils, the results are immediately digitized, so no data entry is necessary (i.e., typing scores from test forms into a data table), and data management/export are timely and efficient.

Issues/challenges

Compared to traditional testing, there are few (if any) fundamental “disadvantages” of RNCT and other digital assessment approaches. That is, most issues and challenges faced by RNCT are tractable problems that are amenable to improvements in science and technology. Ultimately, it is a near certainty that RNCT will greatly alter the historical model of cognitive testing, and neuropsychologists have the potential to benefit from these changes. In the future, our role could be that of expert clinician who gathers background information about our patients, integrates it with results from digital measurements, provides feedback, and engages in brain health interventions (Bilder, 2011; Rao, 2018). In other words, RNCT can enhance the practice of neuropsychology by reducing time spent on administration, scoring, norm searching, and data management, and it can increase the focus on brain-behavior knowledge and direct patient care. But first, we must address the following issues and challenges.

There are four general classes of factors to be considered in the development of RNCT: 1) technical, 2) scientific, 3) examinee-related, and 4) examiner-related. Technical issues arise from variation in local hardware and software, as well as server side and client side processing (Bauer et al., 2012; Parsons et al., 2018). For example, response latencies differ across digital devices such that an input to a smartphone touchscreen or a mouse click on a computer takes longer for some systems to register than others. This means that a reaction time measurement includes both the examinee’s response and system error. Not only is this variability a proverbial “black box” to many users, but possession of particular devices (with varying latencies) is associated with sociodemographic variables in the general population, which could lead to systematic biasing of cognitive test results across people (Germine et al., 2019). This and other technical problems do have potential future solutions, but, in the meantime, it is imperative that characteristics such as response timing are well-documented in technical manuals so that neuropsychologists can become aware of minimum specifications for the tests in their armamentariums. This will allow for the use of appropriate hardware (e.g., laptop, keyboard) to run each RNCT program, thereby minimizing error (Parsons et al., 2018).

Current scientific hurdles to the widespread implementation of RNCT include the vast literature supporting paper-pencil tests that has accumulated over many decades, and that is currently unrivaled by computerized tests (Schmand, 2019). Because “digitizing” a legacy measure fundamentally changes its characteristics (Bailey et al., 2018; Bauer et al., 2012; Steinmetz et al., 2010), new psychometric and clinical data must be collected for RNCTs, even if they are based on historical paper-pencil tests (e.g., the computerized Wisconsin Card Sorting Test). Relatedly, the rapid development of digital software in conjunction with the relatively slow pace of scientific progress creates an unbalanced situation with large numbers of available tests, each with a dearth of empirical support (Charalambous et al., 2020; Parsons, 2016). In other words, it is much easier to design and program a new test (or a new version of a current test) than it is to compile a substantial literature on it, so the field of RNCT is vulnerable to becoming saturated with a plethora of measures that are not backed by strong research evidence. The solution to this issue is not easy, but it is straightforward. Much more research attention should be paid to RNCTs, so that the most robust digital tests can be identified and their characteristics can be comprehensively evaluated, which is a prerequisite to use in everyday clinical practice.

Examinee-related challenges to RNCT include identity verification, performance validity, test security, and computer literacy of the user. Identity verification can be addressed through the use of biometric identifiers (e.g., facial and/or voice recognition) for remote examinees, whereas performance validity can be assessed in real time using embedded indicators (“person fit statistics”) available in item-response theory (Bilder & Reise, 2019). Test security is difficult or impossible to guarantee in RNCT, although the same can be said for paper-pencil tests. One strategy is to use multiple versions of stimuli and large item banks, which would make it difficult to compromise the integrity of any particular test, compared to, for example, the easily-searchable Rey Complex Figure design (Miller & Barr, 2017). Finally, computer literacy is an important issue for RNCT in that experience with computers may be positively associated with performance on some tests (Iverson et al., 2009). However, simple, user-friendly test designs with ample instructions and practice trials can help ameliorate some of the unfamiliarity and stereotype threat associated with technology. Additionally, most future examinees will have used various digital devices throughout their lives, meaning that technological proficiency will gradually improve over time in the general population. Meanwhile, familiarity with paper and pencils will decrease, as schools shift away from these tools, and digital devices will likely become the preferred interface for most examinees.

Examiner-related issues and challenges represent the other side of the coin with respect to the advantages of automation. In other words, scaling up with RNCT by allowing for unsupervised administration removes the examiner from the testing process and reduces the opportunity for behavioral observations, testing of the limits, and the provision of appropriate encouragement to optimize performance. To some degree, this is simply a disadvantage of web-based and mobile cognitive testing that is inherent to the methodology. At the same time, there are also potential technical solutions to some examiner-related factors. For example, with front facing cameras installed on many devices, behavioral data can be recorded and uploaded along with test performance in each patient/participant file. Viewing video files can be labor intensive, but the clinician or researcher may not need to watch long segments in order to sufficiently sample the examinee’s behavior. Alternatively, automated pattern recognition algorithms could be used to detect relevant behaviors (e.g., anxiety, inattention, frustration; Egger et al., 2019; Hibbeln et al., 2017).

Overall, examiner-related limitations and other considerations have led some to propose hybrid in-person/RNCT models, where the two approaches are used in a complementary fashion such that the efficiency and accuracy of RNCT is paired with the clinical rigor of conventional methods (Singh & Germine, 2021). This may be a viable strategy, at least in the near term, as the paradigm of neuropsychology evolves, and we transition toward greater incorporation of RNCT and other digital approaches into our labs and clinics. Meanwhile, RNCT investigators can incorporate best practices into their research programs in order to encourage scientific progress (Germine et al., 2019; Parsons et al., 2018). For example, test creators should prioritize simplicity and user-friendliness and should aim for the “lowest common denominator” with respect to examinees’ technological proficiency. Intuitive, easy-to-operate designs will reduce the influence of factors such as computer literacy on test scores and will improve reliability. In addition, researchers should measure and report factors that may correlate with test scores, such as device type and software requirements (with efforts made to enable compatibility with older, lower-end systems so that more neuropsychologists and patients can ultimately use the test). Finally, RNCTs should be considered living or dynamic entities, in that the software is updated regularly in order to fix bugs and enhance performance. Correspondingly, with ever-evolving software, researchers should plan to gather norms on an ongoing basis, thereby allowing the end user to select a comparison group that matches the most recent iteration of the test.

Summary of contributions

The following contributions reflect the spectrum of RNCT, from paper-pencil teleneuropsychology, to web-based testing, to mobile cognitive testing. Investigations of teleneuropsychology are highly relevant in current clinical practice and can serve to inform clinicians who are seeking to maintain physical distancing for infection control reasons and/or for those who wish to overcome patient transportation barriers. Pulsifer and colleagues investigated of the impact of COVID-19 on teleneuropsychological practice at a lifespan outpatient clinic by examining patient volume and clinical characteristics across four 3-month time periods. Two intervals were pre-COVID-19, when their clinic was conducting traditional, in-person neuropsychological evaluations (spring/summer 2019 and winter 2019/2020), one occurred immediately following the COVID-19 lockdown (spring/summer 2020), when all evaluations used direct-to-home teleneuropsychology, and one occurred while their clinic was re-opened (fall/winter 2020), when they employed a hybrid in-person/direct-to-home teleneuropsychology model. Results showed that most evaluations during the re-opened period were fully in-person, and that remote testing was more popular in adult (34.3%) compared to child (10.9%) assessments. The preference for in-person evaluations in adults and children may have reflected patient preferences, clinician comfort with traditional practices, issues/challenges related to RNCT, or some combination of these factors. Infrequent direct-to-home pediatric evaluations may have been due to a dearth of teleneuropsychological literature in children and/or concerns about the clinical appropriateness of virtual pediatric appointments.

Two papers examined remote, web-based RNCT. Singh and colleagues report on preliminary psychometric properties of the TestMyBrain Digital Neuropsychology Toolkit (DNT) in a large, primarily clinical sample. Test reliability indices were at least acceptable and processing speed, working memory, and general cognition factors emerged from a factor analysis; scores from the mostly clinical sample were lower than scores from a healthy sample. The authors discuss the possibility of using the DNT as a supplement to current clinical practice, although significant limitations remain such as the lack of data on specific clinical populations. LaPlume et al. administered the Cogniciti Brain Health Assessment in amnestic mild cognitive impairment (aMCI; n = 51) and healthy older adults (n = 40); they found greater between-person (interindividual) variability in the aMCI group compared to the control group. The authors interpreted their results as suggesting that remote, web-based testing can be used to assess for aMCI, which is a risk factor for later conversion to Alzheimer’s disease dementia. Together, these two studies provide examples of different approaches to web-based testing research, with some investigators gathering psychometric data to support eventual clinical use and others administering currently available batteries to test important research questions in neuropsychology.

Finally, two papers report on smartphone-based tests administered repeatedly across brief (7- or 14-day) time periods in order to examine clinically relevant fluctuations in cognitive functioning. Bomyea et al. analyzed data from 46 outpatients with bipolar disorder and 20 matched healthy adults. Although mood and cognitive performance were uncorrelated on contemporaneous laboratory-based measures, mobile testing revealed relationships between mood disruption (mania and sadness) earlier in the day and scores on tests of processing speed and working memory later in the day. Meanwhile, Wilks and colleagues evaluated morning and evening-based cognitive performances in 169 adults (aged 61–94) without dementia. Consistent with prior studies in older adults, participants scored worse in the evening compared to the morning hours. Moreover, those participants with cerebrospinal fluid biomarker profiles suggestive of Alzheimer’s disease (beta amyloid and tau) were compared to those without such profiles. The biomarker positive group scored lower than the control group on an associate memory test in the evening but not in the morning, suggesting the possibility of a subtle “cognitive sundowning” syndrome in the earliest detectable stage of Alzheimer’s disease. Taken together, these two studies provide prime examples of how brief, repeated mobile cognitive tests can illuminate time-dependent relationships between cognition and other important aspects of mental and physical health (e.g., mood and circadian rhythms).

Conclusions

Neuropsychology is evolving, with one of the primary areas of growth being the development, evaluation, and incorporation of digital cognitive testing into the field. RNCT, a subset of digital cognitive testing, possess a myriad of possible advantages over the traditional paper-pencil paradigm, but the translation of this potential into everyday clinical practice has proven to be a significant hurdle, with technical, scientific, and conceptual issues to address. The current Special Issue contributes to the advancement of RNCT, primarily by furthering the scientific literature related to 1) paper-pencil teleneuropsychology during the COVID-19 pandemic, 2) psychometric data and research applications in remote web-based cognitive testing, and 3) mobile cognitive tests for measuring circadian and mood-based fluctuations in cognitive functioning. With additional contributions such as these, the psychometric basis, scientific utility, and clinical feasibility/acceptability of RNCTs will be better understood and neuropsychologists will be well-positioned to incorporate RNCTs into their research and clinical activities in the years to come.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.