The impact of administration order in studies of computerized neurocognitive assessment tools (NCATs)

ABSTRACT

Computerized neurocognitive assessment tools (NCATs) have become a common way to assess postconcussion symptoms. As there is increasing research directly comparing multiple NCATs to each other, it is important to consider the impact that order of test administration may have on the integrity of the results. This study investigates the impact of administration order in a study of four different NCATs; Automated Neuropsychological Assessment Metrics (ANAM4), CNS Vital Signs (CNS-VS), CogState, and Immediate Post-Concussion Assessment and Cognitive Test (ImPACT). A total of 272 healthy active duty Service Members were enrolled into this study. All participants were randomly assigned to take two of the four NCATs with order of administration counterbalanced. Analyses attempted to investigate the effect of administration order alone (e.g., Time 1 versus Time 2), the effect of administration order combined with the impact of the specific NCAT received at Time 1, and only the impact of the Time 1 NCAT on Time 2 score variability. Specifically, independent samples t tests were used to compare Time 1 and Time 2 scores within each NCAT. Additional t tests compared Time 1 to Time 2 scores with Time 2 scores grouped by the NCAT received at Time 1. One-way analysis of variance (ANOVA) was used to compare only an NCAT’s Time 2 scores grouped by the NCAT received at Time 1. Cohen’s d effect sizes were calculated for all comparisons. The results from this study revealed statistically significant order effects for CogState and CNS-VS, though with effect sizes generally indicating minimum practical value, and marginal or absent order effects for ANAM4 and ImPACT with no clinically meaningful implications. Despite finding minimal order effects, clinicians should be mindful of the impact of administering multiple NCATs in a single session. Future studies should continue to be designed to minimize the potential effect of test administration order.

KEYWORDS:

• Computerized neurocognitive tests
• Computerized cognitive testing
• Computerized neurocognitive assessment tools
• Neuropsychological assessment
• Order of test administration
• Practice effects.

Acknowledgements

We would like to thank Mary Alice Dale, Karen Schwab, Felicia Qashu, James Wes McGee, Katie Toll, and Alex Fender for their contributions to the study.

Disclaimer

The views expressed herein are those of the author(s) and do not reflect the official policy of the Department of the Army, the Department of Defense, or the U.S. Government.

Disclosure statement

No potential conflict of interest was reported by the authors.