A Stakeholder-Informed Ethical Framework to Guide Implementation of Suicide Risk Prediction Models Derived from Electronic Health Records

REFERENCES

• Barak-Corren, Y., Castro, V. M., Javitt, S., Hoffnagle, A. G., Dai, Y., Perlis, R. H., … Reis, B. Y. (2017). Predicting suicidal behavior from longitudinal electronic health records. The American Journal of Psychiatry, 174(2), 154–162. doi:10.1176/appi.ajp.2016.16010077
   PubMed Web of Science ®Google Scholar
• Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., … Skopp, N. A. (2019). Prediction models for suicide attempts and deaths: A systematic review and simulation. JAMA Psychiatry, 76(6), 642–651. doi:10.1001/jamapsychiatry.2019.0174
   PubMed Web of Science ®Google Scholar
• Cath, C. (2018). Governing artificial intelligence: Ethical, legal and technical opportunities and challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2133), 20180080. doi:10.1098/rsta.2018.0080
   PubMed Web of Science ®Google Scholar
• Coley, R. Y., Johnson, E., Simon, G. E., Cruz, M., & Shortreed, S. M. (2021). Racial/Ethnic disparities in the performance of prediction models for death by suicide after mental health visits. JAMA Psychiatry, 78(7), 726–734. doi:10.1001/jamapsychiatry.2021.0493
   PubMed Web of Science ®Google Scholar
• Czeisler, M. É., Lane, R. I., Petrosky, E., Wiley, J. F., Christensen, A., Njai, R., … Rajaratnam, S. M. W. (2020). Mental health, substance use, and suicidal ideation during the COVID-19 pandemic - United States, June 24–30, 2020. MMWR. Morbidity and Mortality Weekly Report, 69(32), 1049–1057. doi:10.15585/mmwr.mm6932a1
   PubMed Web of Science ®Google Scholar
• Eaneff, S., Obermeyer, Z., & Butte, A. J. (2020). The case for algorithmic stewardship for artificial intelligence and machine learning technologies. JAMA, 324(14), 1397–1398. doi:10.1001/jama.2020.9371
   PubMed Web of Science ®Google Scholar
• Education Development Center Inc. Zero suicide in health and behavioral health care. Retrieved from https://zerosuicide.sprc.org/
   Google Scholar
• Fonseka, T. M., Bhat, V., & Kennedy, S. H. (2019). The utility of artificial intelligence in suicide risk prediction and the management of suicidal behaviors. The Australian and New Zealand Journal of Psychiatry, 53(10), 954–964. doi:10.1177/0004867419864428
   PubMed Web of Science ®Google Scholar
• Friese, S. (2018). User’s manual for ATLAS.ti 8.0. Berlin: ATLAS.ti Scientific Software Development GmbH. Retrieved from http://www.atlasti.com/uploads/media/atlasti_v6_manual.pdf.
   Google Scholar
• Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. (2017). Opportunities and challenges in developing risk prediction models with electronic health records data: A systematic review. Journal of the American Medical Informatics Association : JAMIA, 24(1), 198–208. doi:10.1093/jamia/ocw042
   PubMed Web of Science ®Google Scholar
• Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. doi:10.1016/j.jbi.2008.08.010
   PubMed Web of Science ®Google Scholar
• Hedegaard, H., Curtin, S. C., & Warner, M. (2020). Increase in suicide mortality in the United States, 1999–2018. NCHS Data Brief, (362), 1–8.
   Google Scholar
• Joyce, D. W., & Geddes, J. (2020). When deploying predictive algorithms, are summary performance measures sufficient? JAMA Psychiatry, 77(5), 447–448. doi:10.1001/jamapsychiatry.2019.4484
   PubMed Web of Science ®Google Scholar
• Kessler, R. C., Bauer, M. S., Bishop, T. M., Demler, O. V., Dobscha, S. K., Gildea, S. M., … Bossarte, R. M. (2020). Using administrative data to predict suicide after psychiatric hospitalization in the Veterans Health Administration System. Frontiers in Psychiatry, 11, 390. doi:10.3389/fpsyt.2020.00390
   PubMed Web of Science ®Google Scholar
• Kessler, R. C., Stein, M. B., Petukhova, M. V., Bliese, P., Bossarte, R. M., Bromet, E. J., … Ursano, R. J, Army STARRS Collaborators (2017). Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Molecular Psychiatry, 22(4), 544–551. doi:10.1038/mp.2016.110
   PubMed Web of Science ®Google Scholar
• Kessler, R. C., Warner, C. H., Ivany, C., Petukhova, M. V., Rose, S., Bromet, E. J., … Ursano, R. J, Army STARRS Collaborators (2015). Predicting suicides after psychiatric hospitalization in US Army soldiers: The Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry, 72(1), 49–57. doi:10.1001/jamapsychiatry.2014.1754
   PubMed Web of Science ®Google Scholar
• Kline-Simon, A. H., Sterling, S., Young-Wolff, K., Simon, G., Lu, Y., Does, M., & Liu, V. (2020). Estimates of workload associated with suicide risk alerts after implementation of risk-prediction model. JAMA Network Open, 3(10), e2021189. doi:10.1001/jamanetworkopen.2020.21189
   PubMed Web of Science ®Google Scholar
• Lawrie, S. M., Fletcher-Watson, S., Whalley, H. C., & McIntosh, A. M. (2019). Predicting major mental illness: Ethical and practical considerations. BJPsych Open, 5(2), e30. doi:10.1192/bjo.2019.11
   PubMed Web of Science ®Google Scholar
• Linthicum, K. P., Schafer, K. M., & Ribeiro, J. D. (2019). Machine learning in suicide science: Applications and ethics. Behavioral Sciences & the Law, 37(3), 214–222. doi:10.1002/bsl.2392
   PubMed Web of Science ®Google Scholar
• McCarthy, J. F., Bossarte, R. M., Katz, I. R., Thompson, C., Kemp, J., Hannemann, C. M., … Schoenbaum, M. (2015). Predictive modeling and concentration of the risk of suicide: Implications for preventive interventions in the US Department of Veterans Affairs. American Journal of Public Health, 105(9), 1935–1942. doi:10.2105/ajph.2015.302737
   PubMed Web of Science ®Google Scholar
• McKernan, L. C., Clayton, E. W., & Walsh, C. G. (2018). Protecting life while preserving liberty: Ethical recommendations for suicide prevention with artificial intelligence. Frontiers in Psychiatry, 9, 650–650. doi:10.3389/fpsyt.2018.00650
   PubMed Web of Science ®Google Scholar
• Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020). From what to how: An initial review of publicly available AI ethics tools, methods and research to translate principles into practices. Science and Engineering Ethics, 26(4), 2141–2168. doi:10.1007/s11948-019-00165-5
   PubMed Web of Science ®Google Scholar
• Nundy, S., Montgomery, T., & Wachter, R. M. (2019). Promoting trust between patients and physicians in the era of artificial intelligence. JAMA, 322(6), 497. doi:10.1001/jama.2018.20563
   PubMed Web of Science ®Google Scholar
• Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science (New York, N.Y.), 366(6464), 447–453. doi:10.1126/science.aax2342
   PubMed Web of Science ®Google Scholar
• Reger, G. M., McClure, M. L., Ruskin, D., Carter, S. P., & Reger, M. A. (2019). Integrating predictive modeling into mental health care: An example in suicide prevention. Psychiatric Services (Washington, D.C.), 70(1), 71–74. doi:10.1176/appi.ps.201800242
   PubMed Web of Science ®Google Scholar
• Richards, J. E., Whiteside, U., Ludman, E. J., Pabiniak, C., Kirlin, B., Hidalgo, R., & Simon, G. (2019). Understanding why patients may not report suicidal ideation at a health care visit prior to a suicide attempt: A qualitative study. Psychiatric Services (Washington, D.C.), 70(1), 40–45. doi:10.1176/appi.ps.201800342
   PubMed Web of Science ®Google Scholar
• Simon, G. E., Johnson, E., Lawrence, J. M., Rossom, R. C., Ahmedani, B., Lynch, F. L., … Shortreed, S. M. (2018). Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. The American Journal of Psychiatry, 175(10), 951–960. doi:10.1176/appi.ajp.2018.17101167
   PubMed Web of Science ®Google Scholar
• Simon, G. E., Matarazzo, B. B., Walsh, C. G., Smoller, J. W., Boudreaux, E. D., Yarborough, B. J. H., … Schoenbaum, M. (2021). Reconciling statistical and clinicians’ predictions of suicide risk. Psychiatric Services (Washington, D.C.), 72(5), 555–562. doi:10.1176/appi.ps.202000214
   PubMed Web of Science ®Google Scholar
• Su, C., Aseltine, R., Doshi, R., Chen, K., Rogers, S. C., & Wang, F. (2020). Machine learning for suicide risk prediction in children and adolescents with electronic health records. Translational Psychiatry, 10(1), 413. doi:10.1038/s41398-020-01100-0
   PubMed Web of Science ®Google Scholar
• Tran, T., Luo, W., Phung, D., Harvey, R., Berk, M., Kennedy, R. L., & Venkatesh, S. (2014). Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments. BMC Psychiatry, 14, 76. doi:10.1186/1471-244x-14-76
   PubMed Web of Science ®Google Scholar
• Tucker, R. P., Tackett, M. J., Glickman, D., & Reger, M. A. (2019). Ethical and practical considerations in the use of a predictive model to trigger suicide prevention interventions in healthcare settings. Suicide & Life-Threatening Behavior, 49(2), 382–392. doi:10.1111/sltb.12431
   PubMed Web of Science ®Google Scholar
• Walsh, C. G., Chaudhry, B., Dua, P., Goodman, K. W., Kaplan, B., Kavuluru, R., … Subbian, V. (2020). Stigma, biomarkers, and algorithmic bias: Recommendations for precision behavioral health with artificial intelligence. JAMIA Open, 3(1), 9–15. doi:10.1093/jamiaopen/ooz054
   PubMedGoogle Scholar
• Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017). Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science, 5(3), 457–469. doi:10.1177/2167702617691560
   Web of Science ®Google Scholar
• Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2018). Predicting suicide attempts in adolescents with longitudinal clinical data and machine learning. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 59(12), 1261–1270. doi:10.1111/jcpp.12916
   PubMed Web of Science ®Google Scholar
• Whiting, D., & Fazel, S. (2019). How accurate are suicide risk prediction models? Asking the right questions for clinical practice. Evidence-Based Mental Health, 22(3), 125–128. doi:10.1136/ebmental-2019-300102
   PubMed Web of Science ®Google Scholar
• Yarborough, B. J. H., & Stumbo, S. P. (2021). Patient perspectives on acceptability of, and implementation preferences for, use of electronic health records and machine learning to identify suicide risk. General Hospital Psychiatry, 70, 31–37. doi:10.1016/j.genhosppsych.2021.02.008
   PubMed Web of Science ®Google Scholar