Evaluation of a smartphone photoscreening app to detect refractive amblyopia risk factors in children aged 1–6 years

Abstract

Purpose

To determine the specificity and sensitivity of a smartphone app (GoCheckKids [GCK] used as a photoscreening tool on the iPhone 7 to detect refractive amblyopia risk factors in children aged 1–6 years.

Participants and methods

A prospective, multicenter, 10-month evaluation of children aged 1–6 years old who underwent photoscreening with the GCK app to detect amblyopia risk factors. The first acceptable quality photograph of each study subject was evaluated by trained technicians using GCK’s proprietary automated image processing algorithm to analyze for amblyopia risk factors. Trained graders, masked to the cycloplegic clinical data, remotely reviewed photographs taken with the app and compared results to the gold standard pediatric ophthalmology examinations using the 2013 American Association for Pediatric Ophthalmology & Strabismus amblyopia risk factor guidelines. Primary outcome was the ability of the GCK app to identify amblyopia risk factors compared to the cycloplegic refraction.

Results

There were 287 patient images analyzed. The overall sensitivity and specificity in detecting amblyopia risk factors were 76% and 85%, respectively using manual grading. The overall automated grading results had a sensitivity and sensitivity in detecting amblyopia risk factors of 65% and 83%, respectively.

Conclusion

The GCK smartphone app is a viable photoscreening device for the detection of amblyopia risk factors in children aged 1–6 years.

Video abstract

Point your SmartPhone at the code above. If you have a QR code reader the video abstract will appear. Or use:

http://youtu.be/VOOwxXdOif0

Keywords:

• Apple iPhone
• pediatric
• instrument-based screening
• lazy eye
• blindness
• cost-effective
• childfriendly

Introduction

Amblyopia is the leading cause of monocular visual impairment for children in the US, a leading cause of childhood vision impairment worldwide,^1–5 and affects about 1%–4% of preschool-aged children in the US.⁶ Left untreated, amblyopia can result in permanent vision loss,⁷ but can be effectively treated if detected early.^8,9 Some have shown that photoscreening in children under the age of 3 years can be a reliable method to detect amblyogenic risk factors.¹⁰ Various medical organizations recommend vision screening, but actual screening rates remain low as a result of perceived and real barriers, including parental perceptions that the procedure is too time-consuming or that children are uncooperative.^11,12

Instrument-based photoscreening has been shown to identify risk factors that are likely to lead to amblyopia or poor vision.¹³ Automated photoscreening may be a means to evaluate an even greater number of children without placing an undue burden on the healthcare system or those charged with pediatric care.

The American Academy of Pediatrics (AAP) and the United States Preventative Services Task Force (USPSTF) recommend photoscreening for children under the age of 5 years, but differ on the preferred age to initiate screening, with the former suggesting 1 year and the latter suggesting 3 years.^7,14 The American Association for Pediatric Ophthalmology & Strabismus (AAPOS) recommends beginning instrument-based testing in children between the ages of 1 and 3 years, and places an additional emphasis on higher specificity in younger children and higher sensitivity in older children.⁹ Published recommendations concur that screening techniques should have high sensitivity and specificity.^15–17

This study evaluated the GoCheckKids (GCK, Gobiquity Mobile Health, Scottsdale, AZ, USA) smartphone photo-screening app’s ability to accurately identify children at risk for amblyopia by comparing smartphone photorefraction measurements from four smartphone platforms. One is the Nokia Lumia 1020 (Nokia, Espoo Finland), which has an actual flash. Three others are recent iOS iPhone models 6, 6s, and 7 (Apple, Cupertino, CA, USA) all of which utilize an LED “flash.” Images obtained from these are compared to standard cycloplegic refraction. (Since the time of this study, Gobiquity has chosen to commercialize only the iPhone 7 app due to rapidly changing technologies; as such the remainder of this paper will concentrate only on details and results from the iPhone 7 arm of the study).

Participants and methods

Participants

This single visit study was conducted at four private pedi-atric ophthalmology practices located in Alaska, Arizona, California, and Pennsylvania. Children aged 1–6 years old were enrolled in the iPhone 7 arm of a multi device study. We recruited one group aged 1–3 years as well as those 3–6 years to parallel USPSTF age classification. Written informed consent was given by a parent or legal guardian. The study was approved by the Salus Institutional Review Board (Austin, TX). This study was registered at ClinicalTrials.gov (NCT03447171); as mentioned above, this is a subset analysis from that study.

Exclusion criteria included any history of ocular surgery (strabismus, congenital cataract, or congenital glaucoma surgery) that could potentially affect photorefraction and/or any condition that would preclude the ability of the investigator to obtain a reliable photo image/measurement (eg, cataracts, corneal opacity) or cycloplegic refraction measurement.

All children who met the enrollment criteria were screened using the GoCheckKids app on the iPhone 7 by an ophthalmic technician. The app provides a template to optimize alignment and focal distance for the child, and then generates a flash image similar to conventional iPhone but with the mandatory red-eye reduction pre-flash overridden. The technician reviewed the image to ensure the child was looking directly at the camera before accepting the image. Images were taken prior to the installation of any cycloplegic drops. Following photoscreening, all children underwent a complete eye exam with cycloplegic refraction.

There was no anticipated early withdrawal, as this was a single-visit study with no treatment regimen or follow-up required. Both the technicians securing the images and those evaluating them were masked to gold standard clinical data.

GoCheckKids Mobile Smartphone app

GCK’s Mobile Smartphone photoscreening app is a U.S. Food and Drug Administration-cleared Class IIe medical device consisting of the native camera with flash configured for eccentric photorefraction and a software app to capture and process images. It is intended for vision screening in children from the ages of 6 months to 6 years. This study employed an Apple iPhone 7 in portrait mode (flash-horizontal and to the right of the lens when viewed by the patient).

Image quality assessment

Following image acquisition by the ophthalmic technician, images were processed using the GCK app’s proprietary Image Processing Algorithm (IPA), which identified and created estimates of eye metrics (such as the first Purkinje image location, pupil diameter, crescent width and limbus diameter) and calculated a photorefraction value. These values were compared to the apps referral criteria and the results were immediately displayed as “risk factors identified,” “no risk factors identified,” or “not gradable.” If a not gradable result is displayed the user is encouraged to “retake” the image, or schedule a rescreening appointment.

All images were later reviewed manually, identifying the same eye metrics using calipers, by trained ophthalmic imaging specialists. All images were saved on a remote server that was Health Insurance Portability and Accountability Act (HIPPA)-compliant. Automated results and manual results were compared and are reported in the analysis.

Statistical analysis

Referral criteria for amblyopic risk factors identified after the mobile app use was established from photorefractive measurement thresholds for hyperopia, myopia, and anisometropia. The gold standard referral criteria for cycloplegic refraction is established by the 2013 AAPOS guidelines.⁹ Comparison of the two methods yielded sensitivity, specificity, Positive Predictive Value, and Negative Predictive Value. This study was designed for two-tailed 95% CI, and maximum CI width of 10%.

The study was conducted in accordance with the protocol, Good Clinical Practice Standards, and applicable regulatory requirements, particularly as they pertain to children.

Results

A total of 287 patients were enrolled for evaluation. The majority of patients were Caucasian (52.6%), followed by Hispanic (16%) and Asian (13.2%). There were slightly more female patients than male patients (50.5% vs 49.5%, respectively). Additional patient demographics are included in Table 1.

Table 1 Patient demographics

	Age 1–3 (n = 132)		Age 4–6 (n = 155)		Total (N = 287)
	n	%	n	%	n	%
Race/Ethnicity
African American	7	5.3	6	3.9	13	4.5
Asian	19	14.4	19	12.3	38	13.2
Asian/Caucasian	0	0.00	3	1.9	3	1.1
Asian/Hispanic	0	0.00	1	0.7	1	0.4
Asian/other	1	0.8	1	0.7	2	0.7
Caucasian	74	56.1	77	49.7	151	52.6
Caucasian/Hispanic	1	0.8	1	0.7	2	0.7
Hispanic	17	12.9	29	18.7	46	16.0
Indian	0	0.00	1	0.7	1	0.4
Native American	7	5.3	4	2.6	11	3.8
Pacific Islander	3	2.3	3	1.9	6	2.1
Other	3	2.3	10	6.5	13	4.5
Gender
Male	68	51.5	74	47.7	142	49.5
Female	64	48.5	81	52.3	145	50.5

Image quality assessment

There were 32/287 rejected images from the manual grading group (11.1%), and 37/287 rejected images in the auto grading group (12.9%). In the auto grading group, the leading reason to reject an image was poor gaze (lack of fixation, not strabismus), while in the manual group, the leading reason to reject an image was that the image was considered blurry (Table 2). As stated, if an image was considered “not gradable,” the user was encouraged to “retake,” or schedule a re-screening.

Table 2 Reasons for rejecting captured images

Image rejection reason	iPhone 7 portrait image errors (auto)	iPhone 7 portrait image errors (manual)
Blurry image	3	14
Bad gaze	21	11
Unable to identify pupil	6	5
Test distance error	7	2

The manual grading results for the iPhone 7 had a sensitivity and specificity of 76% and 85% (95% CI 71%–81% and 80%–90%), respectively. The automated grading results on the iPhone 7 had a sensitivity and specificity of 65% and 83% (95% CI 62%–68% and 80%–86%), respectively. The pre-screening probability for our patients was 38% (see Table 3).

Table 3 Combined sensitivity and specificity

	Manual grading, age 1–6	Auto grading, age 1–6
Sensitivity	76%	65%
Specificity	85%	83%
Positive Predictive Value	0.76	0.69
Negative Predictive Value	0.85	0.80

False positives and false negatives

For the manual and automated grading groups, there were 24 and 27 false positives respectively. False negatives were 23 and 33 respectively for manual and automated grading with 12 and 16 respectively due to astigmatism and 11 and 17 due to accommodated hyperopia.

All single-illuminator eccentric photorefractors have a “null zone” that exists between low hyperopia and low myopia refractive values. This null zone is tuned by the manufacturer for the application, based on distance to the subject.^18–20 An accommodating eye will focus (or partially accommodate) at the plane of the camera, the reflected light returns close to the flash source, and no crescent appears in the image of the subjects’ pupil.²¹ When this phenomenon occurs in an eye with hyperopia exceeding the 2013 AAPOS referral criteria, the photorefractor produces measurements below its risk threshold and no referral recommendation is generated. In this study, those images were classified as a false negative.

For the purposes of this study, a false positive occurred when the photorefractor produced measurements above its risk threshold but the child’s cycloplegic refraction was below or exactly at the 2013 AAPOS referral guideline. In this study, we evaluated two separate categories: astigmatism and refractive crescent measurements.

The majority of individual false positives occurred when the same type of refractive error was detected by both photorefraction and cycloplegic refraction, but the severity was above threshold for photoscreening, but below threshold for cycloplegic refraction. In these cases the app overestimated the refractive value. In the manual grading group, an additional two false positives were attributed to extraneous reflections produced by exam room lighting (artifact), and one instance of partial anterior embryotoxon in one eye at 12–3 o’clock. In the auto grading group, there was one instance each of esotropia and blurry image.

There were no reported adverse events.

Discussion

This study evaluated the ability of a smartphone photoscreening app to detect amblyopic risk factors in children aged 1–6 years. The gold standard for detecting refractive errors in children is cycloplegic refraction,²² performed by trained eye care specialists, which requires instilling medication and cooperation on the part of the child.^22,23 Insufficient access and numbers of trained eye care professionals, as well as costs, limit the practicality of exclusively relying on comprehensive eye exams to conduct mass vision screenings in young children.^22,23 In today’s society, smartphones are ubiquitous and relatively inexpensive devices that may help clinicians and other allied healthcare professionals minimize current barriers to screening vision and detection of amblyopia. In this study, the GCK smartphone app had NPV of 80% or greater, performing well in its ability to recognize patients that did not require comprehensive exam by an eye care professional.

The GCK app provides immediate, onsite automated grading and also centralized, subsequent expert manual grading. Different Current Procedural Terminology (CPT) codes can therefore be employed – 99174 for manual grading and 99177 for automated grading. We had image quality issues with each grading method, but better sensitivity with manual grading. The pediatrician considering instrument–based amblyopia screening should balance these issues when selecting which grading paradigm to employ.

A systematic literature review was unable to find a screening test other than instrument-based objective screening that was consistently associated with both high sensitivity and specificity.²⁴ Retinal Birefringence Screening – a highly spe-cialized form of “photo-screening” – has high validation for amblyopia and strabismus.²⁵ Other photoscreening devices have specificity and sensitivity ranges comparable to the GCK.^{15–17,26,27} In this study, the GCK app had a sensitivity of 76% and a specificity of 85%. Peterseim et al²⁸ found a sensitivity of 76% and a specificity of 67% in detecting amblyopia risk factors with the GCK app and a modified Nokia Lumia 1020 cellphone. Differences in how subjects were managed between the two studies, and inclusion of strabismus detection, might account for the differences in specificity. In our study, any image considered “non gradable” was excluded from statistical analysis with a recommendation to re-screen or reschedule the child. In the Peterseim et al study, however, any image considered “not gradable” was included in the analysis as a vision fail. An additional relevant difference between the two studies was the platform on which the app was being evaluated. This study used the Apple iPhone 7, whereas Peterseim et al used a Nokia smartphone.

The AAP has recommended instrument-based vision screening in infants as young as 1 year, with appropriate action to follow, if positive.²⁹ If not identified early, treatments for amblyopia may be ineffective.³⁰ Estimated costs to screen and treat children diagnosed with amblyopia are approximately $1.24 billion annually (presuming about 4.4 million children are screened and about 3% of the children are treated).³⁰

An advantage to the GCK app is the ease with which clinicians can use the application. With the prevalence of smart-phone technology even in more remote areas of the world, virtually everyone has the ability to use this app, meaning children in more rural areas can be properly screened for amblyopia risk factors without having to travel great distances.

There are limitations to this study, however. We did not evaluate the GCK app for its ability to detect strabismus, also an AAPOS amblyopia risk factor, for example. Additionally, only a single flash (portrait) image was acquired which limits the ability to detect astigmatism, that requires both a portrait and landscape image. It is possible that the results garnered in this study would not be borne out with much larger cohorts, or in all populations. Similarly, this study evaluated the GCK app on one smartphone device, the iPhone 7. As cellphone manufacturers continue to improve their technologies, this app may function differently on future smartphones.

It is possible that accommodation in hyperopic patients caused the refractive crescent measurement to fall below the referable risk threshold. Also, because the app only measured refractive error in one meridian, it may have underestimated the amount of astigmatism. Lastly, Internet access is necessary, which may preclude this app’s secondary follow-up image interpretation use in the most remote regions of the world. The next generation smartphone app includes this capability.

Conclusion

The GoCheckKids study demonstrated a specificity and sensitivity for identifying refractive amblyopia risk factors with the iPhone 7 app that is within the range of other photoscreening methods. This smartphone photoscreener is a viable photoscreening app for the detection of amblyopia risk factors in children aged 1–6 years.

Acknowledgments

John Tamkin, PhD, a consultant for Gobiquity, provided technical assistance. Michelle Dalton, ELS, provided medical writing assistance. The study was supported by Gobiquity Mobile Health, Inc.

Disclosure

The study was supported by Gobiquity Mobile Health, Inc. The authors report no other conflicts of interest in this work.