Benefits and challenges of using logistic regression to assess neuropsychological performance validity: Evidence from a simulation study

References

• Baker, D. A., Connery, A. K., Kirk, J. W., & Kirkwood, M. W. (2014). Embedded performance validity indicators within the California Verbal Learning Test, Children’s version. The Clinical Neuropsychologist, 28(1), 116–127. https://doi.org/10.1080/13854046.2013.858184
   PubMed Web of Science ®Google Scholar
• Bashem, J. R., Rapport, L. J., Miller, J. B., Hanks, R. A., Axelrod, B. N., & Millis, S. R. (2014). Comparisons of five performance validity indices in bona fide and simulated traumatic brain injury. The Clinical Neuropsychologist, 28(5), 851–875. https://doi.org/10.1080/13854046.2014.927927
   PubMed Web of Science ®Google Scholar
• Boone, K. B. (2017). Assessment of neurocognitive performance validity. In Textbook of clinical neuropsychology (pp. 39–50). Taylor & Francis.
   Google Scholar
• Bush, S. S., Ruff, R. M., Tröster, A. I., Barth, J. T., Koffler, S. P., Pliskin, N. H., Reynolds, C. R., & Silver, C. H. (2005). Symptom validity assessment: Practice issues and medical necessity NAN policy & planning committee. Archives of Clinical Neuropsychology, 20(4), 419–426. https://doi.org/10.1016/j.acn.2005.02.002
   PubMed Web of Science ®Google Scholar
• Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019). A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology, 110, 12–22. https://doi.org/10.1016/j.jclinepi.2019.02.004
   PubMed Web of Science ®Google Scholar
• Collins, G. S., Ogundimu, E. O., Cook, J. A., Manach, Y. L., & Altman, D. G. (2016). Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Statistics in Medicine, 35(23), 4124–4135. https://doi.org/10.1002/sim.6986
   PubMed Web of Science ®Google Scholar
• Davis, J. J., & Millis, S. R. (2014). Examination of performance validity test failure in relation to number of tests administered. The Clinical Neuropsychologist, 28(2), 199–214. https://doi.org/10.1080/13854046.2014.884633
   PubMed Web of Science ®Google Scholar
• Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science (New York, N.Y.), 243(4899), 1668–1674. https://doi.org/10.1126/science.2648573
   PubMed Web of Science ®Google Scholar
• Donders, J., & Strong, C.-A. H. (2011). Embedded effort indicators on the California verbal learning Test - Second Edition (CVLT-II): An attempted cross-validation. The Clinical Neuropsychologist, 25(1), 173–184. https://doi.org/10.1080/13854046.2010.536781
   PubMed Web of Science ®Google Scholar
• Dorociak, K. E., Schulze, E. T., Piper, L. E., Molokie, R. E., & Janecek, J. K. (2018). Performance validity testing in a clinical sample of adults with sickle cell disease. The Clinical Neuropsychologist, 32(1), 81–97. https://doi.org/10.1080/13854046.2017.1339830
   PubMed Web of Science ®Google Scholar
• Elstein, A. S. (2004). On the origins and development of evidence-based medicine and medical decision making. Inflammation Research, 53(S2), S184–S189. https://doi.org/10.1007/s00011-004-0357-2
   PubMedGoogle Scholar
• Green, P., Lees-Haley, P. R., & Allen III, L. M. (2003). The Word Memory Test and the validity of neuropsychological test scores. Journal of Forensic Neuropsychology, 2(3-4), 97–124.
   Google Scholar
• Greve, K. W., Bianchini, K. J., & Doane, B. M. (2006). Classification accuracy of the Test of Memory Malingering in traumatic brain injury: Results of a known-groups analysis. Journal of Clinical and Experimental Neuropsychology, 28(7), 1176–1190. https://doi.org/10.1080/13803390500263550
   PubMed Web of Science ®Google Scholar
• Harrell, F. (2015). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (Vol. 3). Springer.
   Google Scholar
• Harrell, F. (2020, September 15). Classification vs. prediction. Statistical Thinking. https://www.fharrell.com/post/classification/
   Google Scholar
• Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., Millis, S. R., & Conference Participants. (2009). American Academy of Clinical Neuropsychology Consensus Conference Statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23(7), 1093–1129. https://doi.org/10.1080/13854040903155063
   PubMed Web of Science ®Google Scholar
• Heinze, G. (2006). A comparative investigation of methods for logistic regression with separated or nearly separated data. Statistics in Medicine, 25(24), 4216–4226. https://doi.org/10.1002/sim.2687
   PubMed Web of Science ®Google Scholar
• Jeffreys, H. (1961). The theory of probability. OUP Oxford.
   Google Scholar
• King, E. N., & Ryan, T. P. (2002). A preliminary investigation of maximum likelihood logistic regression versus exact logistic regression. The American Statistician, 56(3), 163–170. https://doi.org/10.1198/00031300283
   Web of Science ®Google Scholar
• Larrabee, G. J. (2014). Minimizing false positive error with multiple performance validity tests: Response to Bilder, Sugar, and Hellemann (2014 this issue). The Clinical Neuropsychologist, 28(8), 1230–1242. https://doi.org/10.1080/13854046.2014.988754
   PubMed Web of Science ®Google Scholar
• Larrabee, G. J., Rohling, M. L., & Meyers, J. E. (2019). Use of multiple performance and symptom validity measures: Determining the optimal per test cutoff for determination of invalidity, analysis of skew, and inter-test correlations in valid and invalid performance groups. The Clinical Neuropsychologist, 33(8), 1354–1372. https://doi.org/10.1080/13854046.2019.1614227
   PubMed Web of Science ®Google Scholar
• Martin, P. K., Schroeder, R. W., & Odland, A. P. (2015). [Formula: See text]Neuropsychologists’ validity testing beliefs and practices: A survey of North American professionals. The Clinical Neuropsychologist, 29(6), 741–776. https://doi.org/10.1080/13854046.2015.1087597
   PubMed Web of Science ®Google Scholar
• McCaffrey, R. J., O’Bryant, S. E., Ashendorf, L., & Fisher, J. M. (2003). Correlations among the TOMM, Rey-15, and MMPI-2 validity scales in a sample of TBI litigants. Journal of Forensic Neuropsychology, 3(3), 45–53. https://doi.org/10.1300/J151v03n03_03
   Google Scholar
• Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
   Google Scholar
• Meyers, J. E., & Volbrecht, M. E. (2003). A validation of multiple malingering detection methods in a large clinical sample. Archives of Clinical Neuropsychology, 18(3), 261–276. https://doi.org/10.1093/arclin/18.3.261
   PubMed Web of Science ®Google Scholar
• Millis, S. R. (2008). What clinicians really need to know about symptom exaggeration, insufficient effort, and malingering: Statistical and measurement matters. In Neuropsychology of malingering casebook (pp. 39–56). Psychology Press.
   Google Scholar
• Nusinovici, S., Tham, Y. C., Yan, M. Y. C., Ting, D. S. W., Li, J., Sabanayagam, C., Wong, T. Y., & Cheng, C.-Y. (2020). Logistic regression was as good as machine learning for predicting major chronic diseases. Journal of Clinical Epidemiology, 122, 56–69. https://doi.org/10.1016/j.jclinepi.2020.03.002
   PubMed Web of Science ®Google Scholar
• Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373–1379. https://doi.org/10.1016/s0895-4356(96)00236-3
   PubMed Web of Science ®Google Scholar
• Perna, R. B., & Loughan, A. R. (2013). Children and the test of memory malingering: Is one trial enough? Child Neuropsychology, 19(4), 438–447. https://doi.org/10.1080/09297049.2012.731500
   PubMed Web of Science ®Google Scholar
• Rhoads, T., Resch, Z. J., Ovsiew, G. P., White, D. J., Abramson, D. A., & Soble, J. R. (2021). Every second counts: A comparison of four dot counting test scoring procedures for detecting invalid neuropsychological test performance. Psychological Assessment, 33(2), 133–141. https://doi.org/10.1037/pas0000970
   PubMed Web of Science ®Google Scholar
• Royston, P., Altman, D. G., & Sauerbrei, W. (2006). Dichotomizing continuous predictors in multiple regression: A bad idea. Statistics in Medicine, 25(1), 127–141. https://doi.org/10.1002/sim.2331
   PubMed Web of Science ®Google Scholar
• Schroeder, R. W., Buddin, W., Jr., Hargrave, D., VonDran, E., Campbell, E., Brockman, C., Heinrichs, R., & Baade, L. (2013). Efficacy of test of memory malingering Trial 1, Trial 2, the Retention Trial, and the Albany Consistency Index in a criterion group forensic neuropsychological sample. Archives of Clinical Neuropsychology, 28(1), 21–29. https://doi.org/10.1093/arclin/acs094
   PubMed Web of Science ®Google Scholar
• Schroeder, R. W., Martin, P. K., & Odland, A. P. (2016). [Formula: See text]Expert beliefs and practices regarding neuropsychological validity testing. The Clinical Neuropsychologist, 30(4), 515–535. https://doi.org/10.1080/13854046.2016.1177118
   PubMed Web of Science ®Google Scholar
• Schroeder, R. W., Olsen, D. H., & Martin, P. K. (2019). Classification accuracy rates of four TOMM validity indices when examined independently and jointly. The Clinical Neuropsychologist, 33(8), 1373–1387. https://doi.org/10.1080/13854046.2019.1619839
   PubMed Web of Science ®Google Scholar
• Sherman, E. M., Slick, D. J., & Iverson, G. L. (2020). Multidimensional malingering criteria for neuropsychological assessment: A 20-year update of the malingered neuropsychological dysfunction criteria. Archives of Clinical Neuropsychology, 35(6), 735–764. https://doi.org/10.1093/arclin/acaa019
   PubMed Web of Science ®Google Scholar
• Shura, R. D., Miskey, H. M., Rowland, J. A., Yoash-Gantz, R. E., & Denning, J. H. (2016). Embedded performance validity measures with postdeployment veterans: Cross-validation and efficiency with multiple measures. Applied Neuropsychology. Adult, 23(2), 94–104. https://doi.org/10.1080/23279095.2015.1014556
   PubMed Web of Science ®Google Scholar
• Smith, K., Boone, K., Victor, T., Miora, D., Cottingham, M., Ziegler, E., Zeller, M., & Wright, M. (2014). Comparison of credible patients of very low intelligence and non-credible patients on neurocognitive performance validity indicators. The Clinical Neuropsychologist, 28(6), 1048–1070. https://doi.org/10.1080/13854046.2014.931465
   PubMed Web of Science ®Google Scholar
• Soderstrom, I. R., & Leitner, D. W. (1997). The effects of base rate, selection ratio, sample size, and reliability of predictors on predictive efficiency indices associated with logistic regression models.
   Google Scholar
• Spencer, R. J., Waldron-Perrine, B., Drag, L. L., Pangilinan, P. H., Axelrod, B. N., & Bieliauskas, L. A. (2017). Neuropsychological test validity in Veterans presenting with subjective complaints of ‘very severe’ cognitive symptoms following mild traumatic brain injury. Brain Injury, 31(1), 32–38. https://doi.org/10.1080/02699052.2016.1218546
   PubMed Web of Science ®Google Scholar
• Sugarman, M. A., & Axelrod, B. N. (2015). Embedded measures of performance validity using verbal fluency tests in a clinical sample. Applied Neuropsychology: Adult, 22(2), 141–146. https://doi.org/10.1080/23279095.2013.873439
   PubMed Web of Science ®Google Scholar
• Sweet, J. J., Heilbronner, R. L., Morgan, J. E., Larrabee, G. J., Rohling, M. L., Boone, K. B., Kirkwood, M. W., Schroeder, R. W., Suhr, J. A., & Conference Participants. (2021). American Academy of Clinical Neuropsychology (AACN) 2021 consensus statement on validity assessment: Update of the 2009 AACN consensus conference statement on neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 35(6), 1053–1054. https://doi.org/10.1080/13854046.2021.1896036
   PubMed Web of Science ®Google Scholar
• van Smeden, M., Moons, K. G., de Groot, J. A., Collins, G. S., Altman, D. G., Eijkemans, M. J., & Reitsma, J. B. (2019). Sample size for binary logistic prediction models: Beyond events per variable criteria. Statistical Methods in Medical Research, 28(8), 2455–2474. https://doi.org/10.1177/0962280218784726
   PubMed Web of Science ®Google Scholar
• Van Walraven, C., & Hart, R. G. (2008). Leave’em alone-why continuous variables should be analyzed as such. Neuroepidemiology, 30(3), 138–139. https://doi.org/10.1159/000126908
   PubMed Web of Science ®Google Scholar
• Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., … Morey, R. D. (2018). Bayesian inference for psychology. Part II: Example applications with JASP. Psychonomic Bulletin & Review, 25(1), 58–76. https://doi.org/10.3758/s13423-017-1323-7
   PubMed Web of Science ®Google Scholar
• Webber, T. A., Critchfield, E. A., & Soble, J. R. (2020). Convergent, discriminant, and concurrent validity of nonmemory-based performance validity tests. Assessment, 27(7), 1399–1415. https://doi.org/10.1177/1073191118804874
   PubMed Web of Science ®Google Scholar
• Wedding, D., & Faust, D. (1989). Clinical judgment and decision making in neuropsychology. Archives of Clinical Neuropsychology, 4(3), 233–265.
   PubMedGoogle Scholar
• Whiteside, D. M., Caraher, K., Hahn-Ketter, A., Gaasedelen, O., & Basso, M. R. (2019). Classification accuracy of individual and combined executive functioning embedded performance validity measures in mild traumatic brain injury. Applied Neuropsychology. Adult, 26(5), 472–481. https://doi.org/10.1080/23279095.2018.1443935
   PubMed Web of Science ®Google Scholar
• Whiteside, D. M., Gaasedelen, O. J., Hahn-Ketter, A. E., Luu, H., Miller, M. L., Persinger, V., Rice, L., & Basso, M. R. (2015). Derivation of a cross-domain embedded performance validity measure in traumatic brain injury. The Clinical Neuropsychologist, 29(6), 788–803. https://doi.org/10.1080/13854046.2015.1093660
   PubMed Web of Science ®Google Scholar
• Wilhelm, S., & Manjunath, B. (2010). tmvtnorm: A package for the truncated multivariate normal distribution. Sigma, 2(2), 1–25.
   Google Scholar
• Wolfe, P. L., Millis, S. R., Hanks, R., Fichtenberg, N., Larrabee, G. J., & Sweet, J. J. (2010). Effort indicators within the California verbal learning test-II (CVLT-II). The Clinical Neuropsychologist, 24(1), 153–168. https://doi.org/10.1080/13854040903107791
   PubMed Web of Science ®Google Scholar