Reliability and validity of a self-administered tool for online neuropsychological testing: The Amsterdam Cognition Scan

ABSTRACT

Introduction: To facilitate large-scale assessment of a variety of cognitive abilities in clinical studies, we developed a self-administered online neuropsychological test battery: the Amsterdam Cognition Scan (ACS). The current studies evaluate in a group of adult cancer patients: test–retest reliability of the ACS and the influence of test setting (home or hospital), and the relationship between our online and a traditional test battery (concurrent validity). Method: Test–retest reliability was studied in 96 cancer patients (57 female; M_age = 51.8 years) who completed the ACS twice. Intraclass correlation coefficients (ICCs) were used to assess consistency over time. The test setting was counterbalanced between home and hospital; influence on test performance was assessed by repeated measures analyses of variance. Concurrent validity was studied in 201 cancer patients (112 female; M_age = 53.5 years) who completed both the online and an equivalent traditional neuropsychological test battery. Spearman or Pearson correlations were used to assess consistency between online and traditional tests. Results: ICCs of the online tests ranged from .29 to .76, with an ICC of .78 for the ACS total score. These correlations are generally comparable with the test–retest correlations of the traditional tests as reported in the literature. Correlating online and traditional test scores, we observed medium to large concurrent validity (r/ρ = .42 to .70; total score r = .78), except for a visuospatial memory test (ρ = .36). Correlations were affected—as expected—by design differences between online tests and their offline counterparts. Conclusions: Although development and optimization of the ACS is an ongoing process, and reliability can be optimized for several tests, our results indicate that it is a highly usable tool to obtain (online) measures of various cognitive abilities. The ACS is expected to facilitate efficient gathering of data on cognitive functioning in the near future.

KEYWORDS:

• Amsterdam Cognition Scan
• computer skills
• concurrent validity
• intraclass correlation coefficient
• neuropsychological test battery
• oncology
• test development
• test–retest reliability
• unmonitored setting

Acknowledgments

First, we thank all participants who were willing to take the time and effort to complete the assessments. Moreover, thanks to Nick Daems for programming the test portal and Vincent van der Noort for developing the software tool for randomization. Finally, many thanks to Anna Meijer, Daphne Kunst-Stevens, Fleur van Ierschot, Frederique Bambach, and Laura Menschaart for their great help with data gathering and data entry.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental material

The supplemental material for this article can be accessed here.

Additional information

Funding

This work was supported by the Dutch Cancer Society, KWF Kankerbestrijding [grant number NKI 2010-4876].