Comparison of 21 artificial intelligence algorithms in automated diabetic retinopathy screening using handheld fundus camera

Abstract

Background

Diabetic retinopathy (DR) is a common complication of diabetes and may lead to irreversible visual loss. Efficient screening and improved treatment of both diabetes and DR have amended visual prognosis for DR. The number of patients with diabetes is increasing and telemedicine, mobile handheld devices and automated solutions may alleviate the burden for healthcare. We compared the performance of 21 artificial intelligence (AI) algorithms for referable DR screening in datasets taken by handheld Optomed Aurora fundus camera in a real-world setting.

Patients and methods

Prospective study of 156 patients (312 eyes) attending DR screening and follow-up. Both papilla- and macula-centred 50° fundus images were taken from each eye. DR was graded by experienced ophthalmologists and 21 AI algorithms.

Results

Most eyes, 183 out of 312 (58.7%), had no DR and mild NPDR was noted in 21 (6.7%) of the eyes. Moderate NPDR was detected in 66 (21.2%) of the eyes, severe NPDR in 1 (0.3%), and PDR in 41 (13.1%) composing a group of 34.6% of eyes with referable DR. The AI algorithms achieved a mean agreement of 79.4% for referable DR, but the results varied from 49.4% to 92.3%. The mean sensitivity for referable DR was 77.5% (95% CI 69.1–85.8) and specificity 80.6% (95% CI 72.1–89.2). The rate for images ungradable by AI varied from 0% to 28.2% (mean 1.9%). Nineteen out of 21 (90.5%) AI algorithms resulted in grading for DR at least in 98% of the images.

Conclusions

Fundus images captured with Optomed Aurora were suitable for DR screening. The performance of the AI algorithms varied considerably emphasizing the need for external validation of screening algorithms in real-world settings before their clinical application.

KEY MESSAGES

• What is already known on this topic? Diabetic retinopathy (DR) is a common complication of diabetes. Efficient screening and timely treatment are important to avoid the development of sight-threatening DR. The increasing number of patients with diabetes and DR poses a challenge for healthcare.

• What this study adds? Telemedicine, mobile handheld devices and artificial intelligence (AI)-based automated algorithms are likely to alleviate the burden by improving efficacy of DR screening programs. Reliable algorithms of high quality exist despite the variability between the solutions.

• How this study might affect research, practice or policy? AI algorithms improve the efficacy of screening and might be implemented to clinical use after thorough validation in a real-life setting.

Keywords:

• Diabetes
• diabetic retinopathy
• screening
• artificial intelligence
• handheld fundus camera

Author contributions

All authors (NH, AMK, PH, PO, AK and JW) contributed to the study conception, design, analysis and interpretation of the data. The first draft of the manuscript was written by NH, and all authors revised it critically for intellectual content and approved the final version to be published. All authors agree to be accountable for all aspects of the work.

Disclosure statement

The authors report no conflict of interest. Petri Huhtinen, PhD, is an employee of Optomed.

Data availability statement

The data that support the findings of this study are available from the corresponding author, NH, upon reasonable request.

Additional information

Funding

This work was supported by PPSHP VTR under Grant Number (K71769) and Finnish Eye Foundation under Grant Number (00082022).