Can machine learning complement traditional medical device surveillance? A case study of dual-chamber implantable cardioverter–defibrillators

Abstract

Background

Machine learning methods may complement traditional analytic methods for medical device surveillance.

Methods and results

Using data from the National Cardiovascular Data Registry for implantable cardioverter–defibrillators (ICDs) linked to Medicare administrative claims for longitudinal follow-up, we applied three statistical approaches to safety-signal detection for commonly used dual-chamber ICDs that used two propensity score (PS) models: one specified by subject-matter experts (PS-SME), and the other one by machine learning-based selection (PS-ML). The first approach used PS-SME and cumulative incidence (time-to-event), the second approach used PS-SME and cumulative risk (Data Extraction and Longitudinal Trend Analysis [DELTA]), and the third approach used PS-ML and cumulative risk (embedded feature selection). Safety-signal surveillance was conducted for eleven dual-chamber ICD models implanted at least 2,000 times over 3 years. Between 2006 and 2010, there were 71,948 Medicare fee-for-service beneficiaries who received dual-chamber ICDs. Cumulative device-specific unadjusted 3-year event rates varied for three surveyed safety signals: death from any cause, 12.8%–20.9%; nonfatal ICD-related adverse events, 19.3%–26.3%; and death from any cause or nonfatal ICD-related adverse event, 27.1%–37.6%. Agreement among safety signals detected/not detected between the time-to-event and DELTA approaches was 90.9% (360 of 396, k=0.068), between the time-to-event and embedded feature-selection approaches was 91.7% (363 of 396, k=−0.028), and between the DELTA and embedded feature selection approaches was 88.1% (349 of 396, k=−0.042).

Conclusion

Three statistical approaches, including one machine learning method, identified important safety signals, but without exact agreement. Ensemble methods may be needed to detect all safety signals for further evaluation during medical device surveillance.

Keywords:

• implanted cardioverter–defibrillator
• methodology
• surveillance

Acknowledgments

The authors acknowledge Ms Erin Singleton, MPH, who assisted in project management as an employee of the Yale–New Haven Hospital Center for Outcomes Research and Evaluation; Ms Julia Eichenfield, MPH, who provided background research during the course of her summer student employment at the Yale–New Haven Hospital Center for Outcomes Research and Evaluation; and Dr Jerome Kassirer, who provided comments on an earlier draft of this manuscript without compensation for his effort. The National Cardiovascular Data Registry (NCDR) ICD Registry is an initiative of the American College of Cardiology Foundation, with partnering support from the Heart Rhythm Society. The views expressed in this manuscript represent those of the authors and do not necessarily represent the official views of the NCDR or its associated professional societies identified at cvquality.acc.org/ncdr. This project was jointly funded by the US Food and Drug Administration (FDA) and Medtronic Inc to develop methods for postmarket surveillance of medical devices. Members of the sponsoring organizations contributed directly to the project, participating in study conception and design, analysis and interpretation of data, and critical revision of the manuscript; the authors made the final decision to submit the manuscript for publication. In addition, the project was approved by but did not receive financial support from the American College of Cardiology’s NCDR. The NCDR research committee reviewed the final manuscript prior to submission, but otherwise had no role in the design, conduct, or reporting of the study. JPC, FAM, and RES receive support from the American College of Cardiology for roles within the NCDR. NRD is supported by grant K12 HS023000-03 from the Agency for Healthcare Research and Quality. JVF is supported by grant K23 HL118147-01 from the National Heart, Lung, and Blood Institute. SLTM is supported by grant U01FD004493 (Medical Device Epidemiology Network Methodology Center) from the FDA. IR is supported by an Early Career Fellowship cofunded by the National Health and Medical Research Council and the National Heart Foundation of Australia.

Author contributions

JSR, JB, and CSP had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors contributed toward data analysis, drafting and revising the paper and agree to be accountable for all aspects of the work.

Disclosure

JSR receives support from the US FDA as part of the Centers for Excellence in Regulatory Science and Innovation program and from the Laura and John Arnold Foundation to support the Collaboration on Research Integrity and Transparency at Yale. JSR, NRD, HMK, and GMG receive research support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing. JSR and GMG receive research support from the Blue Cross Blue Shield Association to better understand medical technology evidence generation. JSR, JPC, NRD, SXL, SLTM, IR, HMK, and CSP work under contract to the Centers for Medicare and Medicaid Services to develop and maintain performance measures that are used for public reporting. JVF receives salary support from the American College of Cardiology NCDR, and modest consulting fees from Janssen Pharmaceuticals. RK is an employee of Medtronic Inc. DMD is an employee of the FDA. HMK chairs a cardiac scientific advisory board for United Health, is a participant/participant representative of the IBM Watson Health Life Sciences Board, is a member of the Advisory Board for Element Science and the Physician Advisory Board for Aetna, and is the founder of Hugo, a personal health-information platform. The authors report no other conflicts of interest in this work.