Threats to external validity in the neuroprediction of substance use treatment outcomes

References

• Finney JW, Moos RH. The long-term course of treated alcoholism: II. Predictors and correlates of 10-year functioning and mortality. J Stud Alcohol. 1992;53:142–53. doi:10.15288/jsa.1992.53.142.
   PubMed Web of Science ®Google Scholar
• Decker KP, Peglow SL, Samples CR, Cunningham TD. Long-Term outcomes after residential substance use treatment: relapse, morbidity, and mortality. Mil Med. 2017;182:e1589–e1595. doi:10.7205/MILMED-D-15-00560.
   PubMed Web of Science ®Google Scholar
• Satre DD, Mertens JR, Areán PA, Weisner C. Five-Year alcohol and drug treatment outcomes of older adults versus middle-aged and younger adults in a managed care program. Addiction. 2004;99:1286–97. doi:10.1111/j.1360-0443.2004.00831.x.
   PubMed Web of Science ®Google Scholar
• Brownell KD, Marlatt GA, Lichtenstein E, Wilson GT. Understanding and preventing relapse. Am Psychol. 1986;41:765–82. doi:10.1037/0003-066X.41.7.765.
   PubMed Web of Science ®Google Scholar
• McLellan AT, Lewis DC, O’Brien CP, Kleber HD. Drug dependence, a chronic medical illness: implications for treatment, insurance, and outcomes evaluation. Jama. 2000;284:1689–95. doi:10.1001/jama.284.13.1689.
   PubMed Web of Science ®Google Scholar
• Proctor SL, Herschman PL. The continuing care model of substance use treatment: what works, and when is “enough,” “enough?”. Psychiatr J. 2014;2014:692423. doi:10.1155/2014/692423.
   PubMedGoogle Scholar
• Chang G, Martin KB, Tang M, Fleming JA. Inpatient hospitalization for substance use disorders one year after residential rehabilitation: predictors among US veterans. Am J Drug Alcohol Abuse. 2016;42:56–62. doi:10.3109/00952990.2015.1088863.
   PubMed Web of Science ®Google Scholar
• Oliver JA, McClernon FJ. Precision medicine in addiction research: where has the time gone? Addiction. 2017;112:2096–97.
   PubMed Web of Science ®Google Scholar
• Reske M, Paulus MP. Predicting treatment outcome in stimulant dependence. Ann N Y Acad Sci. 2008;1141:270–83. doi:10.1196/annals.1441.011.
   PubMed Web of Science ®Google Scholar
• Forster SE, Dickey MW, Forman SD. Regional cerebral blood flow predictors of relapse and resilience in substance use recovery: a coordinate-based meta-analysis of human neuroimaging studies. Drug Alcohol Depend. 2018;185:93–105. doi:10.1016/j.drugalcdep.2017.12.009.
   PubMed Web of Science ®Google Scholar
• Moeller SJ, Paulus MP. Toward biomarkers of the addicted human brain: using neuroimaging to predict relapse and sustained abstinence in substance use disorder. Prog Neuro-Psychopharmacol Biol Psychiatr. 2018;80:143–54. doi:10.1016/j.pnpbp.2017.03.003.
   PubMed Web of Science ®Google Scholar
• Garrison KA, Potenza MN. Neuroimaging and biomarkers in addiction treatment. Curr Psychiatry Rep. 2014;16:513. doi:10.1007/s11920-014-0513-5.
   PubMed Web of Science ®Google Scholar
• Volkow ND, Baler RD. Brain imaging biomarkers to predict relapse in alcohol addiction. JAMA Psychiat. 2013;70:661–63. doi:10.1001/jamapsychiatry.2013.1141.
   PubMed Web of Science ®Google Scholar
• Goldstein RZ, Volkow ND. Dysfunction of the prefrontal cortex in addiction: neuroimaging findings and clinical implications. Nat Rev Neurosci. 2011;12:652–69. doi:10.1038/nrn3119.
   PubMed Web of Science ®Google Scholar
• Braus DF, Wrase J, Grüsser S, Hermann D, Ruf M, Flor H, Mann K, Heinz A. Alcohol-Associated stimuli activate the ventral striatum in abstinent alcoholics. J Neural Transm. 2001;108:887–94. ( Vienna, Austria : 1996). doi:10.1007/s007020170038.
   PubMed Web of Science ®Google Scholar
• Grüsser SM, Wrase J, Klein S, Hermann D, Smolka MN, Ruf M, Weber-Fahr W, Flor H, Mann K, Braus DF, et al. Cue-Induced activation of the striatum and medial prefrontal cortex is associated with subsequent relapse in abstinent alcoholics. Psychopharmacology. 2004;175:296–302. doi:10.1007/s00213-004-1828-4.
   PubMed Web of Science ®Google Scholar
• Gabrieli JDE, Ghosh SS, Whitfield-Gabrieli S. Prediction as a humanitarian and pragmatic contribution from human cognitive neuroscience. Neuron. 2015;85:11–26. doi:10.1016/j.neuron.2014.10.047.
   PubMed Web of Science ®Google Scholar
• Kranzler HR, Smith RV, Schnoll R, Moustafa A, Greenstreet-Akman E. Precision medicine and pharmacogenetics: what does oncology have that addiction medicine does not? Addiction. 2017;112:2086–94. doi:10.1111/add.13818.
   PubMed Web of Science ®Google Scholar
• Volkow ND. Personalizing the treatment of substance use disorders. Am J Psychiatry. 2020;177:113–16. doi:10.1176/appi.ajp.2019.19121284.
   PubMed Web of Science ®Google Scholar
• Marhe R, Luijten M, van de Wetering BJ, Smits M, Franken IH. Individual differences in anterior cingulate activation associated with attentional bias predict cocaine use after treatment. Neuropsychopharmacology. 2013;38:1085–93. doi:10.1038/npp.2013.7.
   PubMed Web of Science ®Google Scholar
• Luo SX, Martinez D, Carpenter KM, Slifstein M, Nunes EV. Multimodal predictive modeling of individual treatment outcome in cocaine dependence with combined neuroimaging and behavioral predictors. Drug Alcohol Depend. 2014;143:29–35. doi:10.1016/j.drugalcdep.2014.04.030.
   PubMed Web of Science ®Google Scholar
• Forster SE, Finn PR, Brown JW. Neural responses to negative outcomes predict success in community-based substance use treatment. Addiction. 2017;112:884–96. doi:10.1111/add.13734.
   PubMed Web of Science ®Google Scholar
• Seo D, Lacadie CM, Tuit K, Hong K-I, Constable RT, Sinha R. Disrupted ventromedial prefrontal function, alcohol craving, and subsequent relapse risk. JAMA Psychiat. 2013;70:727–39. doi:10.1001/jamapsychiatry.2013.762.
   PubMed Web of Science ®Google Scholar
• Kukull WA, Ganguli M. Generalizability: the trees, the forest, and the low-hanging fruit. Neurology. 2012;78:1886–91. doi:10.1212/WNL.0b013e318258f812.
   PubMed Web of Science ®Google Scholar
• Blanco C, Olfson M, Okuda M, Nunes EV, Liu SM, Hasin DS. Generalizability of clinical trials for alcohol dependence to community samples. Drug Alcohol Depend. 2008;98:123–28. doi:10.1016/j.drugalcdep.2008.05.002.
   PubMed Web of Science ®Google Scholar
• Humphreys K, Weingardt KR, Harris AHS. Influence of subject eligibility criteria on compliance with National Institutes of Health guidelines for inclusion of women, minorities, and children in treatment research. Alcohol Clin Exp Res. 2007;31:988–95. doi:10.1111/j.1530-0277.2007.00391.x.
   PubMed Web of Science ®Google Scholar
• Moberg CA, Humphreys K. Exclusion criteria in treatment research on alcohol, tobacco and illicit drug use disorders: a review and critical analysis. Drug Alcohol Rev. 2017;36:378–88. doi:10.1111/dar.12438.
   PubMed Web of Science ®Google Scholar
• Okuda M, Hasin DS, Olfson M, Khan SS, Nunes EV, Montoya I, Liu SM, Grant BF, Blanco C. Generalizability of clinical trials for cannabis dependence to community samples. Drug Alcohol Depend. 2010;111:177–81. doi:10.1016/j.drugalcdep.2010.04.009.
   PubMed Web of Science ®Google Scholar
• Reese ED, Kane LF, Paquette CE, Frohlich F, Daughters SB. Lost in translation: the gap between neurobiological mechanisms and psychosocial treatment research for substance use disorders. Curr Addicti Rep. 2021;8:440–51. doi:10.1007/s40429-021-00382-8.
   Google Scholar
• Andersson HW, Wenaas M, Nordfjærn T. Relapse after inpatient substance use treatment: a prospective cohort study among users of illicit substances. Addict Behav. 2019;90:222–28. doi:10.1016/j.addbeh.2018.11.008.
   PubMed Web of Science ®Google Scholar
• Najt P, Fusar-Poli P, Brambilla P. Co-Occurring mental and substance abuse disorders: a review on the potential predictors and clinical outcomes. Psychiatry Res. 2011;186:159–64. doi:10.1016/j.psychres.2010.07.042.
   PubMed Web of Science ®Google Scholar
• McCabe SE, West BT, Jutkiewicz EM, Boyd CJ. Multiple DSM-5 substance use disorders: a national study of US adults. Hum Psychopharmacol. 2017;32:10.1002/hup.2625. doi:10.1002/hup.2625.
   PubMed Web of Science ®Google Scholar
• Fisher JA, Kalbaugh CA. Challenging assumptions about minority participation in US clinical research. Am J Public Health. 2011;101:2217–22. doi:10.2105/AJPH.2011.300279.
   PubMed Web of Science ®Google Scholar
• Choy T, Baker E, Stavropoulos K. Systemic racism in EEG research: considerations and potential solutions. Affective Sci. 2021;3:14–20.
   PubMedGoogle Scholar
• Chakraborty BH, Steinhauer SR. Reporting of minority participation rates and racial differences in schizophrenia and psychophysiological research: improving but still not adequate. Schizophr Res. 2010;123:90–91. doi:10.1016/j.schres.2010.06.006.
   PubMed Web of Science ®Google Scholar
• SAMHSA. National Survey of Substance Abuse Treatment Services (N-SSATS): 2019, data on substance abuse treatment facilities. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2020.
   Google Scholar
• Weisberg HI, Hayden VC, Pontes VP. Selection criteria and generalizability within the counterfactual framework: explaining the paradox of antidepressant-induced suicidality? Clin Trials. 2009;6:109–18. doi:10.1177/1740774509102563.
   PubMed Web of Science ®Google Scholar
• Susukida R, Crum RM, Hong H, Stuart EA, Mojtabai R. Comparing pharmacological treatments for cocaine dependence: incorporation of methods for enhancing generalizability in meta-analytic studies. Int J Methods Psychiatr Res. 2018;27:e1609. doi:10.1002/mpr.1609.
   PubMed Web of Science ®Google Scholar
• Reske M. What neuroimaging has and has not yet added to our understanding of addiction. 2013;108:1357–59. doi:10.1111/add.12175.
   Google Scholar
• Whelan R, Garavan H. When optimism hurts: inflated predictions in psychiatric neuroimaging. Biol Psychiatry. 2014;75:746–48. doi:10.1016/j.biopsych.2013.05.014.
   PubMed Web of Science ®Google Scholar
• Scheinost D, Noble S, Horien C, Greene AS, Lake EM, Salehi M, Gao S, Shen X, O’Connor D, Barron DS, et al. Ten simple rules for predictive modeling of individual differences in neuroimaging. NeuroImage. 2019;193:35–45. doi:10.1016/j.neuroimage.2019.02.057.
   PubMed Web of Science ®Google Scholar
• Yip SW, Kiluk B, Scheinost D. Toward addiction prediction: an overview of cross-validated predictive modeling findings and considerations for future neuroimaging research. Biol Psychiatr Cogn Neurosci Neuroimaging. 2020;5:748–58. doi:10.1016/j.bpsc.2019.11.001.
   PubMedGoogle Scholar
• CLARITY. Tool to assess risk of bias in longitudinal symptom research studies aimed at the general population. Hamilton (Ontario): CLARITY Group at McMaster University; 2018.
   Google Scholar
• Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;339:b2535.
   Google Scholar
• Motschman CA, Gass JC, Wray JM, Germeroth LJ, Schlienz NJ, Munoz DA, Moore FE, Rhodes JD, Hawk LW, Tiffany ST. Selection criteria limit generalizability of smoking pharmacotherapy studies differentially across clinical trials and laboratory studies: a systematic review on varenicline. Drug Alcohol Depend. 2016;169:180–89. doi:10.1016/j.drugalcdep.2016.10.018.
   PubMed Web of Science ®Google Scholar
• Robinson SM, Sobell LC, Sobell MB, Leo GI. Reliability of the timeline followback for cocaine, cannabis, and cigarette use. Psychol Addictive Behav. 2014;28:154–62. doi:10.1037/a0030992.
   PubMed Web of Science ®Google Scholar
• Sobell LC, Sobell MB. Timeline follow-back: a technique for assessing self-reported alcohol consumption. In: Litten RZ, Allen JP, eds. Measuring alcohol consumption: psychosocial and biochemical methods. Totowa (NJ): Humana Press; 1992. p. 41–72.
   Google Scholar
• McLellan AT, Luborsky L, Woody GE, O’Brien CP, Druley KA. Predicting response to alcohol and drug abuse treatments. Role of psychiatric severity. Arch Gen Psychiatr. 1983;40:620–25. doi:10.1001/archpsyc.1983.04390010030004.
   PubMed Web of Science ®Google Scholar
• Dutra L, Stathopoulou G, Basden SL, Leyro TM, Powers MB, Otto MW. A meta-analytic review of psychosocial interventions for substance use disorders. Am J Psychiatry. 2008;165:179–87. doi:10.1176/appi.ajp.2007.06111851.
   PubMed Web of Science ®Google Scholar
• Jakubczyk A, Ilgen MA, Kopera M, Krasowska A, Klimkiewicz A, Bohnert A, Blow FC, Brower KJ, Wojnar M. Reductions in physical pain predict lower risk of relapse following alcohol treatment. Drug Alcohol Depend. 2016;158:167–71. doi:10.1016/j.drugalcdep.2015.11.020.
   PubMed Web of Science ®Google Scholar
• Roncero C, Palma-Álvarez RF, Díaz-Morán S, Grau-López L, Rodríguez-Cintas L, Ros-Cucurull E, Álvarez AI, Casas M, Daigre C. Cocaine relapse and health-related quality of life: a 23 weeks study. Actas Esp Psiquiatr. 2019;47:37–44.
   PubMed Web of Science ®Google Scholar
• Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, Donohue MR, Foran W, Miller RL, Hendrickson TJ, et al. Reproducible brain-wide association studies require thousands of individuals. Nature. 2022;603:654–60. doi:10.1038/s41586-022-04492-9.
   PubMed Web of Science ®Google Scholar
• Tanaka SC, Yamashita A, Yahata N, Itahashi T, Lisi G, Yamada T, Ichikawa N, Takamura M, Yoshihara Y, Kunimatsu A, et al. A multi-site, multi-disorder resting-state magnetic resonance image database. Sci Data. 2021;8:227. doi:10.1038/s41597-021-01004-8.
   PubMed Web of Science ®Google Scholar
• Tian D, Zeng Z, Sun X, Tong Q, Li H, He H, Gao J-H, He Y, Xia M. A deep learning-based multisite neuroimage harmonization framework established with a traveling-subject dataset. NeuroImage. 2022;257:119297. doi:10.1016/j.neuroimage.2022.119297.
   PubMed Web of Science ®Google Scholar
• Jovicich J, Barkhof F, Babiloni C, Herholz K, Mulert C, van Berckel BNM, Frisoni GB. Harmonization of neuroimaging biomarkers for neurodegenerative diseases: a survey in the imaging community of perceived barriers and suggested actions. Alzheimer’s Dementia. 2019;11:69–73. ( Amsterdam, Netherlands).
   Web of Science ®Google Scholar
• Carter CS, Barch DM, Bullmore E, Breiling J, Buchanan RW, Butler P, Cohen JD, Geyer M, Gollub R, Green MF, et al. Cognitive neuroscience treatment research to improve cognition in schizophrenia II: developing imaging biomarkers to enhance treatment development for schizophrenia and related disorders. Biol Psychiatry. 2011;70:7–12. doi:10.1016/j.biopsych.2011.01.041.
   PubMed Web of Science ®Google Scholar
• Poldrack RA, Huckins G, Varoquaux G. Establishment of best practices for evidence for prediction: a review. JAMA Psychiat. 2020;77:534–40. doi:10.1001/jamapsychiatry.2019.3671.
   PubMed Web of Science ®Google Scholar
• Schnack HG, Kahn RS. Detecting neuroimaging biomarkers for psychiatric disorders: sample size matters. Fron Psychiatr. 2016;7:50. doi:10.3389/fpsyt.2016.00050.
   PubMed Web of Science ®Google Scholar
• Vabalas A, Gowen E, Poliakoff E, Casson AJ, Hernandez-Lemus E. Machine learning algorithm validation with a limited sample size. PloS One. 2019;14:e0224365. doi:10.1371/journal.pone.0224365.
   PubMed Web of Science ®Google Scholar
• Varma S, Simon RJBB. Bias in error estimation when using cross-validation for model selection. BMC Bioinform. 2006;7:1–8.
   PubMed Web of Science ®Google Scholar
• Zhai T, Gu H, Yang Y. Cox regression based modeling of functional connectivity and treatment outcome for relapse prediction and disease subtyping in substance use disorder. Front Neurosci. 2021;15:1520. doi:10.3389/fnins.2021.768602.
   Web of Science ®Google Scholar
• Shen X, Finn ES, Scheinost D, Rosenberg MD, Chun MM, Papademetris X, Constable RT. Using connectome-based predictive modeling to predict individual behavior from brain connectivity. Nat Protoc. 2017;12:506–18. doi:10.1038/nprot.2016.178.
   PubMed Web of Science ®Google Scholar
• Jollans L, Boyle R, Artiges E, Banaschewski T, Desrivières S, Grigis A, Martinot J-L, Paus T, Smolka MN, Walter H, et al. Quantifying performance of machine learning methods for neuroimaging data. NeuroImage. 2019;199:351–65. doi:10.1016/j.neuroimage.2019.05.082.
   PubMed Web of Science ®Google Scholar
• Nielsen AN, Barch DM, Petersen SE, Schlaggar BL, Greene DJ. Machine learning with neuroimaging: evaluating its applications in psychiatry. Biol Psychiatr. 2020;5:791–98. doi:10.1016/j.bpsc.2019.11.007.
   Web of Science ®Google Scholar
• Janssen RJ, Mourão-Miranda J, Schnack HG. Making individual prognoses in psychiatry using neuroimaging and machine learning. Biol Psychiatr. 2018;3:798–808. doi:10.1016/j.bpsc.2018.04.004.
   Web of Science ®Google Scholar
• Tejavibulya L, Rolison M, Gao S, Liang Q, Peterson H, Dadashkarimi J, Farruggia MC, Hahn CA, Noble S, Lichenstein SD, et al. Predicting the future of neuroimaging predictive models in mental health. Mol Psychiatry. 2022. doi:10.1038/s41380-022-01635-2.
   PubMed Web of Science ®Google Scholar
• Thomas Insel MD, Cuthbert BsPh.D., Marjorie Garvey MB, Robert Heinssen BCHsPh.D., Pine DS, Kevin Quinn MDsPh.D., Sanislow CsPh.D., Wang P, M.D.D.R.P.H. National Institute of Mental Health, Bethesda, MdResearch Domain Criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry. 2010;167:748–51. doi:10.1176/appi.ajp.2010.09091379.
   PubMed Web of Science ®Google Scholar
• Lesko CR, Buchanan AL, Westreich D, Edwards JK, Hudgens MG, Cole SR. Generalizing study results: a potential outcomes perspective. Epidemiology. 2017;28:553–61. doi:10.1097/EDE.0000000000000664.
   PubMed Web of Science ®Google Scholar
• Meier MH, Hall W, Caspi A, Belsky DW, Cerdá M, Harrington HL, Houts R, Poulton R, Moffitt TE. Which adolescents develop persistent substance dependence in adulthood? using population-representative longitudinal data to inform universal risk assessment. Psychol Med. 2016;46:877–89. doi:10.1017/S0033291715002482.
   PubMed Web of Science ®Google Scholar
• Jacobs B, Beems T, Stulemeijer M, van Vugt AB, van der Vliet TM, Borm GF, Vos PE. Outcome prediction in mild traumatic brain injury: age and clinical variables are stronger predictors than CT abnormalities. J Neurotrauma. 2009;27:655–68. doi:10.1089/neu.2009.1059.
   Web of Science ®Google Scholar
• Fleisher AS, Sun S, Taylor C, Ward CP, Gamst AC, Petersen RC, Jack CR, Aisen PS, Thal LJ. Volumetric MRI vs clinical predictors of Alzheimer disease in mild cognitive impairment. Neurology. 2008;70:191. doi:10.1212/01.wnl.0000287091.57376.65.
   PubMed Web of Science ®Google Scholar
• Marcoulides GA, Drezner Z. Tabu search variable selection with resource constraints. Commun Stat - Simul Comput. 2004;33:355–62. doi:10.1081/SAC-120037240.
   Web of Science ®Google Scholar
• Forster SE, Forman SD, Gancz NN, Siegle GJ, Dickey MW, Steinhauer SR. Electrophysiological predictors and indicators of contingency management treatment response: rationale and design for the ways of rewarding abstinence project (WRAP). Contemp Clin Trials Commun. 2021;23:100796. doi:10.1016/j.conctc.2021.100796.
   PubMed Web of Science ®Google Scholar