Reliability and validity of a self-administered tool for online neuropsychological testing: The Amsterdam Cognition Scan

References

• Aalbers, T., Baars, M. A., Olde Rikkert, M. G., & Kessels, R. P. (2013). Puzzling with online games (BAM-COG): Reliability, validity, and feasibility of an online self-monitor for cognitive performance in aging adults. Journal of Medical Internet Research, 15(12), e270. doi:10.2196/jmir.2860
   PubMed Web of Science ®Google Scholar
• Alpherts, W., & Aldenkamp, A. P. (1995). FePsy: The iron psyche, manual. Heemstede: Instituut voor Epilepsiebestrijding.
   Google Scholar
• Anastasi, A. (1998). Psychological testing (6th ed.). New York: Macmillan.
   Google Scholar
• Andreotti, C., & Hawkins, K. A. (2015). RBANS norms based on the relationship of age, gender, education, and WRAT-3 reading to performance within an older african american sample. The Clinical Neuropsychologist, 29(4), 442–465. doi:10.1080/13854046.2015.1039589
   PubMed Web of Science ®Google Scholar
• Assmann, K. E., Bailet, M., Lecoffre, A. C., Galan, P., Hercberg, S., Amieva, H., & Kesse-Guyot, E. (2016). Comparison between a self-administered and supervised version of a web-based cognitive test battery: Results from the NutriNet-Sante cohort study. Journal of Medical Internet Research, 18(4). doi:10.2196/jmir.4862
   PubMed Web of Science ®Google Scholar
• Barak, A., & Buchanan, T. (2003). Internet-based psychological assessment. In R. Kraus, J. S. Zack, & G. Stricker (Eds.), Online counseling: A handbook for mental health professionals (pp. 217–240). San Diego, CA: Elsevier Academic Press.
   Google Scholar
• Barak, A., & English, N. (2002). Prospects and limitations of psychological testing on the internet. Journal of Technology in Human Services, 19, 65–89. doi:10.1300/J017v19n02_06
   Google Scholar
• Bauer, R. M., Iverson, G. L., Cernich, A. N., Binder, L. M., Ruff, R. M., & Naugle, R. I. (2012). Computerized neuropsychological assessment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. The Clinical Neuropsychologist, 26(2), 177–196. doi:10.1080/13854046.2012.663001
   PubMed Web of Science ®Google Scholar
• Bilder, R. M. (2011). Neuropsychology 3.0: Evidence-based science and practice. Journal of the International Neuropsychological Society, 17(1), 7–13. doi:10.1017/S1355617710001396
   PubMed Web of Science ®Google Scholar
• Birnbaum, M. H. (2004). Human research and data collection via the internet. Annual Review of Psychology, 55, 803–832. doi:10.1146/annurev.psych.55.090902.141601
   PubMed Web of Science ®Google Scholar
• Bornstein, R. A., Baker, G., & Douglass, A. (1987). Short-term retest reliability of the halstead-reitan battery in a normal sample. The Journal of Nervous and Mental Disease, 175(4), 229–232. doi:10.1097/00005053-198704000-00007
   PubMed Web of Science ®Google Scholar
• Bruggemans, E. F., Van de Vijver, F. J., & Huysmans, H. A. (1997). Assessment of cognitive deterioration in individual patients following cardiac surgery: Correcting for measurement error and practice effects. Journal of Clinical and Experimental Neuropsychology, 19(4), 543–559. doi:10.1080/01688639708403743
   PubMed Web of Science ®Google Scholar
• Brünken, R., & Plass, J. L. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38(1), 53–61. doi:10.1207/S15326985EP3801_7
   Web of Science ®Google Scholar
• Buchanan, T., & Smith, J. L. (1999). Using the internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90(1), 125–144. doi:10.1348/000712699161189
   PubMed Web of Science ®Google Scholar
• Caine, C., Mehta, M. P., Laack, N. N., & Gondi, V. (2012). Cognitive function testing in adult brain tumor trials: Lessons from a comprehensive review. Expert Review of Anticancer Therapy, 12(5), 655–667. doi:10.1586/era.12.34
   PubMed Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates.
   Google Scholar
• Cole, W. R., Arrieux, J. P., Schwab, K., Ivins, B. J., Qashu, F. M., & Lewis, S. C. (2013). Test-retest reliability of four computerized neurocognitive assessment tools in an active duty military population. Archives of Clinical Neuropsychology, 28(7), 732–742. doi:10.1093/arclin/act040
   PubMed Web of Science ®Google Scholar
• Cromer, J. A., Harel, B. T., Yu, K., Valadka, J. S., Brunwin, J. W., Crawford, C. D., … Maruff, P. (2015). Comparison of cognitive performance on the Cogstate brief battery when taken in-clinic, in-group, and unsupervised. The Clinical Neuropsychologist, 29(4), 542–558. doi:10.1080/13854046.2015.1054437
   PubMed Web of Science ®Google Scholar
• Culbertson, W. C., & Zillmer, E. A. (2001). Tower of london-drexel (TOL-DX) technical manual. North Tonawanda, NY: Multi-Health Systems.
   Google Scholar
• Darby, D. G., Fredrickson, J., Pietrzak, R. H., Maruff, P., Woodward, M., & Brodtmann, A. (2014). Reliability and usability of an internet-based computerized cognitive testing battery in community-dwelling older people. Computers in Human Behavior, 30, 199–205. doi:10.1016/j.chb.2013.08.009
   Web of Science ®Google Scholar
• de Vet, H. C. W., Terwee, C. B., Mokkink, L. B., & Knol, D. L. (2011). Measurement in medicine: A practical guide (2nd ed.). Cambridge: Cambridge University Press.
   Google Scholar
• Devlin, S. J., Gnanadesikan, R., & Kettenring, J. R. (1975). Robust estimation and outlier detection with correlation coefficients. Biometrika, 62(3), 531–545. doi:10.1093/biomet/62.3.531
   Web of Science ®Google Scholar
• Dikmen, S. S., Heaton, R. K., Grant, I., & Temkin, N. R. (1999). Test-retest reliability and practice effects of expanded Halstead-Reitan neuropsychological test battery. Journal of the International Neuropsychological Society, 5(4), 346–356. doi:10.1017/S1355617799544056
   PubMed Web of Science ®Google Scholar
• Dougherty, J. H., Jr., Cannon, R. L., Nicholas, C. R., Hall, L., Hare, F., Carr, E., … Arunthamakun, J. (2010). The computerized self test (CST): An interactive, internet accessible cognitive screening test for dementia. Journal of Alzheimer’s Disease, 20(1), 185–195. doi:10.3233/JAD-2010-1354
   PubMed Web of Science ®Google Scholar
• Feenstra, H. E. M., Vermeulen, I. E., Murre, J. M. J., & Schagen, S. B. (in press). Reference data and reliability for the Amsterdam cognition scan: Reports on a new online tool for self-administered cognitive testing.
   Google Scholar
• Gates, N. J., & Kochan, N. A. (2015). Computerized and on-line neuropsychological testing for late-life cognition and neurocognitive disorders: Are we there yet? Current Opinion in Psychiatry, 28(2), 165–172. doi:10.1097/YCO.0000000000000141
   PubMed Web of Science ®Google Scholar
• Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19(5), 847–857. doi:10.3758/s13423-012-0296-9
   PubMed Web of Science ®Google Scholar
• Gunter, B., Nicholas, D., Huntington, P., & Williams, P. (2002). Online versus offline research: Implications for evaluating digital media. Aslib Proceedings, 54(4), 229–239. doi:10.1108/00012530210443339
   Google Scholar
• Hansen, T. I., Haferstrom, E. C., Brunner, J. F., Lehn, H., & Haberg, A. K. (2015). Initial validation of a web-based self-administered neuropsychological test battery for older adults and seniors. Journal of Clinical and Experimental Neuropsychology, 37(6), 581–594. doi:10.1080/13803395.2015.1038220
   PubMed Web of Science ®Google Scholar
• Hansen, T. I., Lehn, H., Evensmoen, H. R., & Håberg, A. K. (2016). Initial assessment of reliability of a self-administered web-based neuropsychological test battery. Computers in Human Behavior, 63, 91–97. doi:10.1016/j.chb.2016.05.025
   Web of Science ®Google Scholar
• Hayes, A. F. (2012). PROCESS: A versatile computational tool for observed variable mediation, moderation, and conditional process modeling [White paper]. Retrieved from http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/SobelTest
   Google Scholar
• Internet world stats. (2017). World internet usage and population statistics, Retrieved from http://www.internetworldstats.com/stats.htm
   Google Scholar
• ImPACT Applications. Inc.. (n.d.). Frequently asked questions - Can ImPACT be taken at home? Retrieved May, 2016, from https://www.impacttest.com/about/?Frequently-Asked-Questions-7
   Google Scholar
• Iverson, G. L., Brooks, B. L., Ashton, V. L., Johnson, L. G., & Gualtieri, C. T. (2009). Does familiarity with computers affect computerized neuropsychological test performance? Journal of Clinical and Experimental Neuropsychology, 31(5), 594–604. doi:10.1080/13803390802372125
   PubMed Web of Science ®Google Scholar
• Kessels, R. P. C., van Zandvoort, M. J. E., Postma, A., Kappelle, L. J., & Edward, H. (2010). The corsi block-tapping task: Standardization and normative data. Applied Neuropsychology, 7(4), 37–41.
   Google Scholar
• Kløve, H. (1963). Grooved pegboard test user instructions. Lafayette: Lafayette Instruments.
   Google Scholar
• Kuhn, A. W., & Solomon, G. S. (2014). Supervision and computerized neurocognitive baseline test performance in high school athletes: An initial investigation. Journal of Athletic Training, 49(6), 800–805. doi:10.4085/1062-6050-49.3.66
   PubMed Web of Science ®Google Scholar
• Lemay, S., Bedard, M., Rouleau, I., & Tremblay, P. G. (2004). Practice effect and test-retest reliability of attentional and executive tests in middle-aged to elderly subjects. The Clinical Neuropsychologist, 18(2), 284–302.
   Google Scholar
• Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4), 764–766. doi:10.1016/j.jesp.2013.03.013
   Web of Science ®Google Scholar
• Lezak, M. D., Howieson, D. B., Loring, D. W., Hannay, H. J., & Fischer, J. S. (2004). Neuropsychological assessment (4th ed.). Oxford: Oxford University Press.
   Google Scholar
• Lin, N. U., Lee, E. Q., Aoyama, H., Barani, I. J., Baumert, B. G., Brown, P. D., … Wen, P. Y. (2013). Challenges relating to solid tumour brain metastases in clinical trials, part 1: Patient population, response, and progression. A report from the RANO group. The Lancet Oncology, 14(10), e396–e406. doi:10.1016/S1470-2045(13)70311-5
   PubMed Web of Science ®Google Scholar
• Littleton, A. C., Register-Mihalik, J. K., & Guskiewicz, K. M. (2015). Test-retest reliability of a computerized concussion test: CNS vital signs. Sports Health: A Multidisciplinary Approach, 7, 443–447. doi:10.1177/1941738115586997
   Google Scholar
• Lowe, C., & Rabbitt, P. (1998). Test\re-test reliability of the CANTAB and ISPOCD neuropsychological batteries: Theoretical and practical issues. Neuropsychologia, 36(9), 915–923. doi:10.1016/S0028-3932(98)00036-0
   PubMed Web of Science ®Google Scholar
• Medalia, A., Lim, R., & Erlanger, D. (2005). Psychometric properties of the web-based work-readiness cognitive screen used as a neuropsychological assessment tool for schizophrenia. Computer Methods and Programs in Biomedicine, 80(2), 93–102. doi:10.1016/j.cmpb.2005.06.007
   PubMed Web of Science ®Google Scholar
• Miller, E. (1995). CaLCAP psychometric properties. Retrieved March 27, 2017, from http://www.calcaprt.com/metrics.htm
   Google Scholar
• Morrison, G. E., Simone, C. M., Ng, N. F., & Hardy, J. L. (2015). Reliability and validity of the neurocognitive performancetest, a web-based neuropsychological assessment. Frontiers in Psychology, 6(1652). doi:10.3389/fpsyg.2015.01652
   PubMed Web of Science ®Google Scholar
• Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., & Velasquez, R. (2004). Psychological testing on the internet: New problems, old issues. The American Psychologist, 59(3), 150–162. doi:10.1037/0003-066X.59.3.150
   PubMed Web of Science ®Google Scholar
• Nys, G., Van Zandvoort, M., De Kort, P., Jansen, B., Van der Worp, H., Kappelle, L., & De Haan, E. (2005). Domain-specific cognitive recovery after first-ever stroke: A follow-up study of 111 cases. Journal of the International Neuropsychological Society, 11(07), 795–806. doi:10.1017/S1355617705050952
   PubMed Web of Science ®Google Scholar
• Parsey, C. M., & Schmitter-Edgecombe, M. (2013). Applications of technology in neuropsychological assessment. The Clinical Neuropsychologist, 27(8), 1328–1361. doi:10.1080/13854046.2013.834971
   PubMed Web of Science ®Google Scholar
• Raz, S., Bar-Haim, Y., Sadeh, A., & Dan, O. (2012). Reliability and validity of the online continuous performance test among young adults. Assessment, 21(1), 108–118. doi:10.1177/1073191112443409
   PubMed Web of Science ®Google Scholar
• Reips, U.-D. (2002a). Context effects in web surveys. In B. Batinic, U. D. Reips, & M. Bosnjak (Eds.), Online Social Sciences (pp. 95-101). Göttingen: Hogrefe & Huber.
   Google Scholar
• Reips, U.-D. (2002b). Internet-based psychological experimenting: Five dos and five don’ts. Social Science Computer Review, 20(3), 241–249. doi:10.1177/08939302020003002
   Web of Science ®Google Scholar
• Reips, U.-D. (2002c). Standards for internet-based experimenting. Experimental Psychology, 49(4), 243–256. doi:10.1026/1618-3169.49.4.243
   PubMed Web of Science ®Google Scholar
• Reitan, R. M. (1958). Validity of the trail making test as an indicator of organic brain damage. Perceptual and Motor Skills, 8, 271–276. doi:10.2466/pms.1958.8.3.271
   Google Scholar
• Ruano, L., Sousa, A., Severo, M., Alves, I., Colunas, M., Barreto, R., … Bento, V. (2016). Development of a self-administered web-based test for longitudinal cognitive assessment. Scientific Reports, 6. doi:10.1038/srep19114
   PubMed Web of Science ®Google Scholar
• Ruff, R. M., & Parker, S. B. (1993). Gender-and age-specific changes in motor speed and eye-hand coordination in adults: Normative values for the finger tapping and grooved pegboard tests. Perceptual and Motor Skills, 76(3c), 1219–1230. doi:10.2466/pms.1993.76.3c.1219
   PubMedGoogle Scholar
• Ryan, J. J., Lopez, S. J., & Paolo, A. M. (1996). Temporal stability of digit span forward, backward, and forward minus backward in persons aged 75-87 years. Cognitive and Behavioral Neurology, 9(3), 206–208.
   Google Scholar
• Saan, R. J., & Deelman, B. G. (1986). The 15-words test A and B. Groningen: University of Groningen, Department of Neuropsychology.
   Google Scholar
• Schagen, S. B., Klein, M., Reijneveld, J. C., Brain, E., Deprez, S., Joly, F., … Wefel, J. S. (2014). Monitoring and optimising cognitive function in cancer patients: Present knowledge and future directions. European Journal of Cancer Supplements, 12(1), 29–40. doi:10.1016/j.ejcsup.2014.03.003
   Google Scholar
• Schatz, P. (2010). Long-term test-retest reliability of baseline cognitive assessments using ImPACT. The American Journal of Sports Medicine, 38(1), 47–53. doi:10.1177/0363546509343805
   PubMed Web of Science ®Google Scholar
• Schatz, P., & Browndyke, J. (2002). Applications of computer-based neuropsychological assessment. The Journal of Head Trauma Rehabilitation, 17(5), 395–410. doi:10.1097/00001199-200210000-00003
   PubMed Web of Science ®Google Scholar
• Schmand, B., Bakker, D., Saan, R., & Louman, J. (1991). The Dutch reading test for adults: A measure of premorbid intelligence level. Tijdschrift Voor Gerontologie En Geriatrie, 22(1), 15–19.
   PubMedGoogle Scholar
• Schmand, B., Groenink, S. C., & van den Dungen, M. (2008). Letter fluency: psychometric properties and Dutch normative data. Tijdschrift voor gerontologie en geriatrie, 39(2), 64–76. doi:10.1007/BF03078128
   PubMedGoogle Scholar
• Schnirman, G. M., Welsh, M. C., & Retzlaff, P. D. (1998). Development of the Tower of London-revised. Assessment, 5(4), 355–360. doi:10.1177/107319119800500404
   PubMed Web of Science ®Google Scholar
• Silverstein, S. M., Berten, S., Olson, P., Paul, R., Willams, L. M., Cooper, N., & Gordon, E. (2007). Development and validation of a World-Wide-Web-based neurocognitive assessment battery: WebNeuro. Behavior Research Methods, 39(4), 940–949. doi:10.3758/BF03192989
   PubMed Web of Science ®Google Scholar
• Silverstein, S. M., Jaeger, J., Donovan-Lepore, A.-M., Wilkniss, S. M., Savitz, A., Malinovsky, I., … Marcello, S. (2010). A comparative study of the MATRICS and IntegNeuro cognitive assessment batteries. Journal of Clinical and Experimental Neuropsychology, 32(9), 937–952. doi:10.1080/13803391003596496
   PubMed Web of Science ®Google Scholar
• Smets, E. M., Garssen, B., Bonke, B., & De Haes, J. C. (1995). The Multidimensional Fatigue Inventory (MFI) psychometric qualities of an instrument to assess fatigue. Journal of Psychosomatic Research, 39(3), 315–325. doi:10.1016/0022-3999(94)00125-O
   PubMed Web of Science ®Google Scholar
• Spreen, O., & Strauss, E. (1998). A compendium of neuropsychological tests. New York: Oxford University Press.
   Google Scholar
• Strauss, E., Sherman, E., & Spreen, O. (2006). A compendium of neuropsychological tests: Administration, norms, and commentary (3rd ed.). Oxford: Oxford University Press.
   Google Scholar
• Trustram Eve, C., & de Jager, C. A. (2014). Piloting and validation of a novel self-administered online cognitive screening tool in normal older persons: The cognitive function test. International Journal of Geriatric Psychiatry, 29(2), 198–206. doi:10.1002/gps.3993
   PubMed Web of Science ®Google Scholar
• Tsushima, M., Tsushima, W., Tsushima, V., Lim, N., Madrigal, E., Jackson, C., & Mendler, M. H. (2013). Use of ImPACT to diagnose minimal hepatic encephalopathy: An accurate, practical, user-friendly internet-based neuropsychological test battery. Digestive Diseases and Sciences, 58(9), 2673–2681. doi:10.1007/s10620-013-2668-z
   PubMed Web of Science ®Google Scholar
• Van de Weijer-Bergsma, E., Kroesbergen, E. H., Prast, E. J., & Van Luit, J. E. (2015). Validity and reliability of an online visual-spatial working memory task for self-reliant administration in school-aged children. Behavioral Research Methods, 47(3), 708–719. doi:10.3758/s13428-014-0469-8
   PubMed Web of Science ®Google Scholar
• Van den Burg, W., Saan, R. J., & Deelman, B. G. (1985). 15-woordentest. Provisional manual. Groningen, The Netherlands: University Hospital, Department of Neuropsychology.
   Google Scholar
• van Hooijdonk, C., & Krahmer, E. (2008). Information modalities for procedural instructions: The influence of text, pictures, and film clips on learning and executing RSI exercises. IEEE Transactions on Professional Communication, 51, 50–62. doi:10.1109/TPC.2007.2000054
   Web of Science ®Google Scholar
• Verhage, F. (1964). Intelligentie en leeftijd: Onderzoek bij Nederlanders van twaalf tot zevenenzeventig jaar (Doctoral thesis) Van Gorcum, Assen.
   Google Scholar
• Wechsler, D. (1997).  WAIS-III: Wechsler adult intelligence scale. San Antonio, TX: Psychological Corporation.
   Google Scholar
• Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231–240. doi:10.1519/15184.1
   PubMed Web of Science ®Google Scholar
• Zigmond, A. S., & Snaith, R. P. (1983). The hospital anxiety and depression scale. Acta Psychiatrica Scandinavica, 67(6), 361–370. doi:10.1111/acp.1983.67.issue-6
   PubMed Web of Science ®Google Scholar