Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products

ABSTRACT

Introduction

Computer aided detection and diagnosis (CADe and CADx) products are an emerging branch of medical device industry. However, limited technical standard has been developed for product verification and validation. It will be helpful to investigate the current practice of preclinical and clinical evaluation of approved products and provide insights for future standardization.

Areas covered

Document review was conducted on 56 products approved by the United States Food and Drug Administration, including Summary of Safety and Effectiveness Data, 510(k) decision and de novo decision summaries. Key parameters describing product characteristics, preclinical studies and clinical studies were collected. Evaluation strategies for CADe/CADx products were analyzed and assessed.

Expert opinion

Preclinical studies were widely adopted in the verification of CADe/CADx products. Standalone performance testing was a common procedure, but the selection of testing dataset and performance metrics showed significant variability and flexibility among manufacturers. Clinical studies were reported by all class III products and some class II products, and Multi-Reader Multi-Case design was commonly used. However, statistical analysis and presentation/interpretation of results was oftentimes incomplete. To resolve above issues, systematic development of standards of CADe/CADx is encouraged, which can be implemented at different aspects through the product lifecycle.

KEYWORDS:

• Computer aided detection
• computer aided diagnosis
• deep learning
• medical device regulation
• clinical study

Article highlights

• CADe/CADx products are increasingly approved as medical device, boosted by development of modern artificial intelligence algorithms like deep learning

• Fifty-six software products intended for the use of detection, diagnosis, measurement, contouring, enhancement, triage, or follow-up of lesions approved by the United States Food and Drug Administration between 2001 and 2018 were reviewed

• The role of CADe/CADx is shifting from second reader, concurrent reader to first reader, gradually changing the standard clinical workflow

• Verification and clinical evaluation of CADe/CADx products shows considerable variation in study design, methodology, and reporting, which may bring challenges to compare products and implement postmarket regulation

• Standardization is encouraged to promote verification and validation of CADe/CADx products, which should cover the product lifecycle and develop in different aspects

Acknowledgments

The authors want to show their gratitude for Dr. Haiping Ren, who participated in the early stage of technical discussion and provided helpful information on regulation strategy of artificial intelligence medical device.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Additional information

Funding

This paper was funded by National Key Research and Development Program of China 2019YFC0118801.