Symptom and performance validity in samples of adults at clinical evaluation of ADHD: a replication study using machine learning algorithms

References

• Aase, D. M., Soble, J. R., Shepard, P., Akagi, K., Schroth, C., Greenstein, J. E., Proescher, E., & Phan, K. L. (2021). Concordance of embedded performance and symptom validity tests and associations with mild traumatic brain injury and posttraumatic stress disorder among post-9/11 Veterans. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 36(3), 424–429. https://doi.org/10.1093/arclin/acaa053
   PubMed Web of Science ®Google Scholar
• Abramson, D. A., White, D. J., Rhoads, T., Carter, D. A., Hansen, N. D., Resch, Z. J., Jennette, K. J., Ovsiew, G. P., & Soble, J. R. (2021). Cross-validating the dot counting test among an adult ADHD clinical sample and analyzing the effect of ADHD subtype and comorbid psychopathology. Assessment, 10731911211050895. https://doi.org/10.1177/10731911211050895
   Google Scholar
• Adamou, M., Arif, M., Asherson, P., Aw, T.-C., Bolea, B., Coghill, D., Guðjónsson, G., Halmøy, A., Hodgkins, P., Müller, U., Pitts, M., Trakoli, A., Williams, N., & Young, S. (2013). Occupational issues of adults with ADHD. BMC Psychiatry, 13(1), 59. https://doi.org/10.1186/1471-244X-13-59
   PubMedGoogle Scholar
• Bali, R., & Sarkar, D. (2016). R machine learning by example: Understand the fundamentals of machine learning with R and build your own dynamic algorithms to tackle complicated real-world problems successfully. Community experience distilled. Packt Publishing. http://proquest.tech.safaribooksonline.de/9781784390846
   Google Scholar
• Bush, S. S., Ruff, R. M., Tröster, A. I., Barth, J. T., Koffler, S. P., Pliskin, N. H., Reynolds, C., & Silver, C. H. (2005). Symptom validity assessment: Practice issues and medical necessity NAN policy & planning committee. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 20(4), 419–426. https://doi.org/10.1016/j.acn.2005.02.002
   PubMed Web of Science ®Google Scholar
• Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1), 6. https://doi.org/10.1186/s12864-019-6413-7
   PubMed Web of Science ®Google Scholar
• Chicco, D., Tötsch, N., & Jurman, G. (2021). The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining, 14(1), 13. https://doi.org/10.1186/s13040-021-00244-z
   PubMed Web of Science ®Google Scholar
• Christiansen, H., Kis, B., Hirsch, O., Philipsen, A., Henneck, M., Panczuk, A., Pietrowsky, R., Hebebrand, J., & Schimmelmann, B. G. (2011). German validation of the Conners adult ADHD rating scales-self-report (CAARS-S) I: Factor structure and normative data. European Psychiatry: The Journal of the Association of European Psychiatrists, 26(2), 100–107. https://doi.org/10.1016/j.eurpsy.2009.12.024
   PubMed Web of Science ®Google Scholar
• Christiansen, H., Kis, B., Hirsch, O., Matthies, S., Hebebrand, J., Uekermann, J., Abdel-Hamid, M., Kraemer, M., Wiltfang, J., Graf, E., Colla, M., Sobanski, E., Alm, B., Rösler, M., Jacob, C., Jans, T., Huss, M., Schimmelmann, B. G., & Philipsen, A. (2012). German validation of the Conners Adult ADHD Rating Scales (CAARS) II: Reliability, validity, diagnostic sensitivity and specificity. European Psychiatry: The Journal of the Association of European Psychiatrists, 27(5), 321–328. https://doi.org/10.1016/j.eurpsy.2010.12.010
   PubMed Web of Science ®Google Scholar
• Christiansen, H., Hirsch, O., Philipsen, A., Oades, R. D., Matthies, S., Hebebrand, J., … Kis, B. (2013). German validation of the Conners adult ADHD rating scale-self-report: Confirmation of factor structure in a large sample of participants with ADHD. Journal of Attention Disorders, 17(8), 690–698. https://doi.org/10.1177/1087054711435680
   PubMed Web of Science ®Google Scholar
• Christiansen, H., Hirsch, O., Abdel-Hamid, M., & Kis, B. (2014). CAARS. Conners Skalen zu Aufmerksamkeit und Verhalten für Erwachsene. Huber.
   Google Scholar
• Conners, C. K., Erhardt, D., & Sparrow, E. (1999). Conner´s adult ADHD rating scales (CAARS). Technical manual. Multi-Health Systems.
   Google Scholar
• Czornik, M., Seidl, D., Tavakoli, S., Merten, T., & Lehrner, J. (2021). Motor reaction times as an embedded measure of performance validity: A study with a sample of Austrian early retirement claimants. Psychological Injury and Law, 41(1), 15. https://doi.org/10.1007/s12207-021-09431-z
   Google Scholar
• Dolgin, E. (2014). FDA clearance paves way for computerized ADHD monitoring. Nature Medicine, 20(5), 454–455. https://doi.org/10.1038/nm0514-454
   PubMed Web of Science ®Google Scholar
• Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87. https://doi.org/10.1145/2347736.2347755
   Web of Science ®Google Scholar
• Edebol, H., Helldin, L., & Norlander, T. (2012). Objective measures of behavior manifestations in adult ADHD and differentiation from participants with bipolar II disorder, borderline personality disorder, participants with disconfirmed ADHD as well as normative participants. Clinical Practice and Epidemiology in Mental Health, 8(1), 134–143. https://doi.org/10.2174/1745017901208010134
   PubMedGoogle Scholar
• Erdodi, L. A., Roth, R. M., Kirsch, N. L., Lajiness-O’neill, R., & Medoff, B. (2014). Aggregating validity indicators embedded in Conners’ CPT-II outperforms individual cutoffs at separating valid from invalid performance in adults with traumatic brain injury. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 29(5), 456–466. https://doi.org/10.1093/arclin/acu026
   PubMed Web of Science ®Google Scholar
• Erdodi, L. A., Sagar, S., Seke, K., Zuccato, B. G., Schwartz, E. S., & Roth, R. M. (2018). The Stroop test as a measure of performance validity in adults clinically referred for neuropsychological assessment. Psychological Assessment, 30(6), 755–766. https://doi.org/10.1037/pas0000525
   PubMed Web of Science ®Google Scholar
• Fuermaier, A. B. M., Tucha, L., Koerts, J., Aschenbrenner, S., Kaunzinger, I., Hauser, J., Tucha, L., Weisbrod, M., Lange, K. W., & Tucha, O. (2015). Cognitive impairment in adult ADHD–perspective matters! Neuropsychology, 29(1), 45–58. https://doi.org/10.1037/neu0000108
   PubMed Web of Science ®Google Scholar
• Fuermaier, A. B. M., Tucha, O., Koerts, J., Grabski, M., Lange, K. W., Weisbrod, M., … Tucha, L. (2016). The development of an embedded figures test for the detection of feigned attention deficit hyperactivity disorder in adulthood. PloS ONE, 11(10), e0164297. https://doi.org/10.1371/journal.pone.0164297
   PubMed Web of Science ®Google Scholar
• Fuermaier, A. B. M., Tucha, O., Koerts, J., Tucha, L., Thome, J., & Faltraco, F. (2021). Feigning ADHD and stimulant misuse among Dutch university students. Journal of Neural Transmission, 128(7), 1079–1084. https://doi.org/10.1007/s00702-020-02296-7
   PubMed Web of Science ®Google Scholar
• Green, P. (2003). Green’s word memory test. User’s manual.
   Google Scholar
• Harrison, A. G., & Edwards, M. J. (2010). Symptom exaggeration in post-secondary students: Preliminary base rates in a Canadian sample. Applied Neuropsychology, 17(2), 135–143. https://doi.org/10.1080/09084281003715642
   PubMed Web of Science ®Google Scholar
• Harrison, A. G., & Armstrong, I. T. (2020). Differences in performance on the test of variables of attention between credible vs. Noncredible individuals being screened for attention deficit hyperactivity disorder. Applied Neuropsychology. Child, 9(4), 314–322. https://doi.org/10.1080/21622965.2020.1750115
   PubMed Web of Science ®Google Scholar
• Hirsch, O., & Christiansen, H. (2017). Factorial structure and validity of the quantified behavior test plus (Qb+©). Assessment, 24(8), 1037–1049. https://doi.org/10.1177/1073191116638426
   PubMed Web of Science ®Google Scholar
• Hirsch, O., & Christiansen, H. (2018). Faking ADHD? Symptom validity testing and its relation to self-reported, observer-reported symptoms, and neuropsychological measures of attention in adults with ADHD. Journal of Attention Disorders, 22(3), 269–280. https://doi.org/10.1177/1087054715596577
   PubMed Web of Science ®Google Scholar
• Hoelzle, J. B., Ritchie, K. A., Marshall, P. S., Vogt, E. M., & Marra, D. E. (2019). Erroneous conclusions: The impact of failing to identify invalid symptom presentation when conducting adult attention-deficit/hyperactivity disorder (ADHD) research. Psychological Assessment, 31(9), 1174–1179. https://doi.org/10.1037/pas0000752
   PubMed Web of Science ®Google Scholar
• Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B. (2020). I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data. Neuroscience and Biobehavioral Reviews, 119(1), 456–467. https://doi.org/10.1016/j.neubiorev.2020.09.036
   PubMedGoogle Scholar
• Iglewicz, B., & Hoaglin, D. C. (1993). How to detect and handle outliers. ASQC basic references in quality control (Vol. 16). ASQC Quality Press.
   Google Scholar
• Jennette, K. J., Williams, C. P., Resch, Z. J., Ovsiew, G. P., Durkin, N. M., O’Rourke, J. J. F., Marceaux, J. C., Critchfield, E. A., & Soble, J. R. (2021). Assessment of differential neurocognitive performance based on the number of performance validity tests failures: A cross-validation study across multiple mixed clinical samples. The Clinical Neuropsychologist, 1–19. https://doi.org/10.1080/13854046.2021.1900398
   Web of Science ®Google Scholar
• Koerts, J., Tucha, L., Leenders, K. L., van Beilen, M., Brouwer, W. H., & Tucha, O. (2011). Subjective and objective assessment of executive functions in Parkinson’s disease. Journal of the Neurological Sciences, 310(1–2), 172–175. https://doi.org/10.1016/j.jns.2011.07.009
   PubMed Web of Science ®Google Scholar
• Kotsiantis, S. B., Zaharakis, I. D., & Pintelas, P. E. (2006). Machine learning: A review of classification and combining techniques. Artificial Intelligence Review, 26(3), 159–190. https://doi.org/10.1007/s10462-007-9052-3
   Web of Science ®Google Scholar
• Lantz, B. (2015). Machine learning with R: Learn how to use R to apply powerful machine learning methods and gain an insight into real-world applications Packt open source (2nd ed.). Packt Publ.
   Google Scholar
• Lis, S., Baer, N., Stein-en-Nosse, C., Gallhofer, B., Sammer, G., & Kirsch, P. (2010). Objective measurement of motor activity during cognitive performance in adults with attention-deficit/hyperactivity disorder. Acta Psychiatrica Scandinavica, 122(4), 285–294. https://doi.org/10.1111/j.1600-0447.2010.01549.x
   PubMed Web of Science ®Google Scholar
• Lunardon, N., Menardi, G., & Torelli, N. (2014). ROSE: A package for binary imbalanced learning. The R Journal, 6(1), 79–89. https://journal.r-project.org/archive/2014/RJ-2014-008/RJ-2014-008.pdf
   Google Scholar
• Marshall, P., Schroeder, R., O’Brien, J., Fischer, R., Ries, A., Blesi, B., & Barker, J. (2010). Effectiveness of symptom validity measures in identifying cognitive and behavioral symptom exaggeration in adult attention deficit hyperactivity disorder. The Clinical Neuropsychologist, 24(7), 1204–1237. https://doi.org/10.1080/13854046.2010.514290
   PubMed Web of Science ®Google Scholar
• Marshall, P. S., Hoelzle, J. B., Heyerdahl, D., & Nelson, N. W. (2016). The impact of failing to identify suspect effort in patients undergoing adult attention-deficit/hyperactivity disorder (ADHD) assessment. Psychological Assessment, 28(10), 1290–1302. https://doi.org/10.1037/pas0000247
   PubMed Web of Science ®Google Scholar
• Martin, P. K., & Schroeder, R. W. (2020). Base rates of invalid test performance across clinical non-forensic contexts and settings. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 35(6), 717–725. https://doi.org/10.1093/arclin/acaa017
   PubMed Web of Science ®Google Scholar
• Montesinos López, O. A. (2022). Multivariate statistical machine learning methods for genomic prediction. Springer International Publishing AG.
   Google Scholar
• Musso, M. W., & Gouvier, W. D. (2014). “Why is this so hard?” A review of detection of malingered ADHD in college students. Journal of Attention Disorders, 18(3), 186–201. https://doi.org/10.1177/1087054712441970
   PubMed Web of Science ®Google Scholar
• Nelson, J. M., & Lovett, B. J. (2019). Assessing ADHD in college students: Integrating multiple evidence sources with symptom and performance validity data. Psychological Assessment, 31(6), 793–804. https://doi.org/10.1037/pas0000702
   PubMed Web of Science ®Google Scholar
• Ord, A. S., Miskey, H. M., Lad, S., Richter, B., Nagy, K., & Shura, R. D. (2021). Examining embedded validity indicators in Conners continuous performance test-3 (CPT-3). The Clinical Neuropsychologist, 35(8), 1426–1441. https://doi.org/10.1080/13854046.2020.1751301
   PubMed Web of Science ®Google Scholar
• Orrù, G., Mazza, C., Monaro, M., Ferracuti, S., Sartori, G., & Roma, P. (2020). The development of a short version of the SIMS using machine learning to detect feigning in forensic assessment. Psychological Injury and Law, 13(6), 235. https://doi.org/10.1007/s12207-020-09389-4
   Google Scholar
• Phillips, M. S., Wisinger, A. M., Lapitan-Moore, F. T., Ausloos-Lozano, J. E., Bing-Canar, H., Durkin, N. M., … Soble, J. R. (2022). Cross-validation of multiple embedded performance validity indices in the Rey auditory verbal learning test and brief visuospatial memory test-revised in an adult attention deficit/hyperactivity disorder clinical sample. Psychological Injury and Law, 56(6), 601. https://doi.org/10.1007/s12207-022-09443-3
   Google Scholar
• Rabiner, D. L. (2013). Stimulant prescription cautions: Addressing misuse, diversion and malingering. Current Psychiatry Reports, 15(7), 375. https://doi.org/10.1007/s11920-013-0375-2
   PubMed Web of Science ®Google Scholar
• Rhoads, T., Neale, A. C., Resch, Z. J., Cohen, C. D., Keezer, R. D., Cerny, B. M., Jennette, K. J., Ovsiew, G. P., & Soble, J. R. (2021). Psychometric implications of failure on one performance validity test: A cross-validation study to inform criterion group definition. Journal of Clinical and Experimental Neuropsychology, 43(5), 437–448. https://doi.org/10.1080/13803395.2021.1945540
   PubMed Web of Science ®Google Scholar
• Rhys, H. (2020). Machine learning with R, the tidyverse, and mlr (1st ed.). Manning Publications.
   Google Scholar
• Rostain, A. L. (2006). Addressing the misuse and abuse of stimulant medications on college campuses. Current Psychiatry Reports, 8(5), 335–336. https://doi.org/10.1007/s11920-006-0032-0
   PubMedGoogle Scholar
• Schmand, B., & Lindeboom, J. (2005). Amsterdamer Kurzzeitgedächtnistest (AKGT) [Amsterdam Short Term Memory Test]. PITS B. V.
   Google Scholar
• Scimeca, L. M., Holbrook, L., Rhoads, T., Cerny, B. M., Jennette, K. J., Resch, Z. J., Obolsky, M. A., Ovsiew, G. P., & Soble, J. R. (2021). Examining Conners continuous performance test-3 (CPT-3) embedded performance validity indicators in an adult clinical sample referred for ADHD evaluation. Developmental Neuropsychology, 46(5), 347–359. https://doi.org/10.1080/87565641.2021.1951270
   PubMed Web of Science ®Google Scholar
• Sherman, E. M. S., Slick, D. J., & Iverson, G. L. (2020). Multidimensional malingering criteria for neuropsychological assessment: A 20-Year update of the malingered neuropsychological dysfunction criteria. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 35(6), 735–764. https://doi.org/10.1093/arclin/acaa019
   PubMed Web of Science ®Google Scholar
• Sibley, M. H., Rohde, L. A., Swanson, J. M., Hechtman, L. T., Molina, B. S. G., Mitchell, J. T., … Stehli, A. (2018). Late-onset ADHD reconsidered with comprehensive repeated assessments between ages 10 and 25. The American Journal of Psychiatry, 175(2), 140–149. https://doi.org/10.1176/appi.ajp.2017.17030298
   PubMed Web of Science ®Google Scholar
• Sibley, M. H. (2021). Empirically-informed guidelines for first-time adult ADHD diagnosis. Journal of Clinical and Experimental Neuropsychology, 43(4), 340–351. https://doi.org/10.1080/13803395.2021.1923665
   PubMed Web of Science ®Google Scholar
• Soble, J. R., Alverson, W. A., Phillips, J. I., Critchfield, E. A., Fullen, C., O’Rourke, J. J. F., Messerly, J., Highsmith, J. M., Bailey, K. C., Webber, T. A., & Marceaux, J. C. (2020). Strength in numbers or quality over quantity? Examining the importance of criterion measure selection to define validity groups in performance validity test (PVT) research. Psychological Injury and Law, 13(1), 44–56. https://doi.org/10.1007/s12207-019-09370-w
   Web of Science ®Google Scholar
• Söderström, S., Pettersson, R., & Nilsson, K. W. (2014). Quantitative and subjective behavioural aspects in the assessment of attention-deficit hyperactivity disorder (ADHD) in adults. Nordic Journal of Psychiatry, 68(1), 30–37. https://doi.org/10.3109/08039488.2012.762940
   PubMed Web of Science ®Google Scholar
• Stevens, A., Bahlo, S., Licha, C., Liske, B., & Vossler-Thies, E. (2016). Reaction time as an indicator of insufficient effort: Development and validation of an embedded performance validity parameter. Psychiatry Research, 245(1), 74–82. https://doi.org/10.1016/j.psychres.2016.08.022
   PubMedGoogle Scholar
• Suhr, J., Hammers, D., Dobbins-Buckland, K., Zimak, E., & Hughes, C. (2008). The relationship of malingering test failure to self-reported symptoms and neuropsychological findings in adults referred for ADHD evaluation. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 23(5), 521–530. https://doi.org/10.1016/j.acn.2008.05.003
   PubMed Web of Science ®Google Scholar
• Suhr, J. A., Buelow, M., & Riddle, T. (2011). Development of an Infrequency Index for the CAARS. Journal of Psychoeducational Assessment, 29(2), 160–170. https://doi.org/10.1177/0734282910380190
   Web of Science ®Google Scholar
• Sullivan, B. K., May, K., & Galbally, L. (2007). Symptom exaggeration by college adults in attention-deficit hyperactivity disorder and learning disorder assessments. Applied Neuropsychology, 14(3), 189–207. https://doi.org/10.1080/09084280701509083
   PubMed Web of Science ®Google Scholar
• Sweet, J. J., Heilbronner, R. L., Morgan, J. E., Larrabee, G. J., Rohling, M. L., Boone, K. B., Kirkwood, M. W., Schroeder, R. W., & Suhr, J. A. (2021). American Academy of Clinical Neuropsychology (AACN) 2021 consensus statement on validity assessment: Update of the 2009 AACN consensus conference statement on neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 35(6), 1053–1106. https://doi.org/10.1080/13854046.2021.1896036
   PubMed Web of Science ®Google Scholar
• Toplak, M. E., West, R. F., & Stanovich, K. E. (2013). Practitioner review: Do performance-based measures and ratings of executive function assess the same construct? Journal of Child Psychology and Psychiatry, and Allied Disciplines, 54(2), 131–143. https://doi.org/10.1111/jcpp.12001
   PubMed Web of Science ®Google Scholar
• Tucha, L., Fuermaier, A. B. M., Koerts, J., Groen, Y., & Thome, J. (2015). Detection of feigned attention deficit hyperactivity disorder. Journal of Neural Transmission, 122(Suppl 1), S123–34. https://doi.org/10.1007/s00702-014-1274-3
   Google Scholar
• Ulberstad, F. (2012). QbTest technical manual. Qbtech AB.
   Google Scholar
• Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019). Machine learning algorithm validation with a limited sample size. PloS ONE, 14(11), e0224365. https://doi.org/10.1371/journal.pone.0224365
   PubMed Web of Science ®Google Scholar
• Wallace, E. R., Garcia-Willingham, N. E., Walls, B. D., Bosch, C. M., Balthrop, K. C., & Berry, D. T. R. (2019). A meta-analysis of malingering detection measures for attention-deficit/hyperactivity disorder. Psychological Assessment, 31(2), 265–270. https://doi.org/10.1037/pas0000659
   PubMed Web of Science ®Google Scholar
• White, D. J., Ovsiew, G. P., Rhoads, T., Resch, Z. J., Lee, M., Oh, A. J., & Soble, J. R. (2022). The Divergent Roles of Symptom and Performance Validity in the Assessment of ADHD. Journal of Attention Disorders, 26(1), 101–108. https://doi.org/10.1177/1087054720964575
   PubMed Web of Science ®Google Scholar