A guide to optometrists for appraising and using artificial intelligence in clinical practice

ABSTRACT

Artificial intelligence systems are becoming increasingly available as diagnostic aids for optometric practice. These perform well but are often ‘black-box’ systems offering little or no insight into how a decision was reached. While there is potential for artificial intelligence to improve patient outcomes, clinicians without training in computer science may find it difficult to ascertain whether these technologies are suitable for their practice, or how they should be used. This review provides an overview of how artificial intelligence systems work in optometry, their strengths, weaknesses, and regulatory considerations. A checklist is provided for appraising a system, covering regulatory approvals, ascertaining what the system can and cannot do, how it can be used in practice, whether it is suitable for the clinical population, and whether the outputs can be explained. Artificial intelligence has the potential to improve accuracy and efficiency in many areas of optometry if used correctly, and should be embraced by clinicians as an assistive tool.

GRAPHICAL ABSTRACT

KEYWORDS:

• Artificial intelligence
• deep learning
• diagnostic imaging
• machine learning
• optometry

What is artificial intelligence?

Artificial intelligence (AI) is a term used to describe a variety of software systems that enable machines to make decisions that appear ‘intelligent’ to the human observer. ¹ This may involve rule-based systems designed by humans or be based on an algorithm that the machine itself has developed through a process known as machine learning. AI may be utilised to improve workflows or usability of devices, such as automatic focus systems on cameras,^2,3 or to make more complex decisions, such as the detection and classification of diseases.^4–6 As the scope of optometric practice and the utilisation of imaging technologies expand, AI will, in turn, become more influential in clinical decision-making, and therefore it is important that clinicians develop a better understanding of the processes underpinning AI and the inherent advantages and limitations of this technology.

This review will extend the work of Charng et al.,⁷ to explain the similarities and differences between the ways humans and machines approach clinical problems, explore the strengths and weaknesses of AI, and understand how AI can be used to assist practitioners in making high quality clinical decisions. By understanding the limitations of AI systems, or the biases that these systems may have, clinicians can modify their own diagnostic decision making to take advantage of the benefits of AI-derived information and avoid the pitfalls of over-reliance on these systems.

AI is already being used in eyecare in various ways. Key examples include in image acquisition and noise reduction in ocular coherence tomography (OCT)^8,9 and OCT angiography.¹⁰ AI is also used to aid in technical tasks such as OCT retinal layer segmentation^11,12 and image co-registration.¹³ These AI systems have become so accurate and useful that their adoption in clinical practice has occurred both inconspicuously and ubiquitously. More recently, AI systems have been developed to detect the optic disc and macula in fundus images,^14–16 and further developed to provide clinical decision support or even to make diagnoses autonomously.^5,17–19

Whilst this decision-making form of AI is viewed by some as a threat to the profession, others recognise that these tools have substantial potential to improve care delivery for screening, diagnostic confidence, and practitioner efficiency^20,21 This review will provide a basic understanding of how AI works in the context of clinical diagnostic assistance. To illustrate this, consider an AI system which classifies fundus photos as to whether or not they contain referrable diabetic retinopathy; this example will be referred to throughout the paper.

Basic description of AI algorithms

When viewed from a clinical perspective, AI systems can be categorised as ‘white-box’, ‘grey-box’, or ‘black-box’ based on how easily the algorithm can be interpreted.²² A white-box system is based on clear rules that can be understood by experts in the field. For example, the decision tree depicted in Figure 1 is used to grade diabetic retinopathy from retinal photographs. Such algorithms are regarded as white-box systems as the logic behind each decision is clearly defined. While these are used by clinicians, they can also be used by AI systems provided the data required to inform each decision can be extracted; the algorithm would need to include a method for detecting the presence of microaneurysms for example, or for the operator to provide this information explicitly.

Figure 1. A decision tree for diabetic retinopathy grading.

In contrast, the neural network for diabetic retinopathy grading depicted in Figure 2 is far more difficult to interrogate, with data passing through multiple computational nodes with different importance or ‘weightings’ attributed to each. The algorithm examines the photo as a large collection of red, green and blue pixel values, identifying ‘features’ in the data in order to arrive at a disease classification. A feature is any pattern in the data, from those clearly associated with the pathophysiology such as a group of red pixels representing haemorrhages, to obscure compositions such as the difference in green value between two distant pixel locations. Understanding exactly how these features are used to determine disease severity is impossible, hence the attribution of such models as black-box systems.

Figure 2. Example convolutional neural network for diabetic retinopathy grading. Data flows from left to right. Orange and blue lines show connections with higher or lower weighting, respectively, indicating a higher or lower impact on the calculations for the following node. N = node.

Grey-box systems are a hybrid of these designs, whereby only part of the system logic can be understood.²³ An example system may use a decision tree to determine the presence and location of haemorrhages in a retinal photograph, then use a neural network to classify each haemorrhage as diabetic, hypertensive or indicative of some other retinopathy.

Explainability is a term used in AI theory to qualify the extent to which the decision-making process of the model can be comprehended by humans.²⁴ By definition, white-box systems are fully explainable and black-box systems are not, though processes have been developed to approximate the inner workings of the system, as detailed below. The majority of AI systems used in optometry involve image processing, such as detecting diabetic retinopathy,^5,17,19 papilloedema¹⁸ or glaucoma⁵ risk factors from fundus photographs, with the most accurate results generally obtained with black-box systems because they are not constrained by a need to limit detection to image features which would be apparent to a human operator.²²

However, the opaque nature of black-box systems has created some barriers to the adoption of this technology in healthcare, presumably because clinicians are suspicious of decision-making outcomes that they cannot verify using human logic.²⁵ While tools have been developed to improve the explainability of these systems, some basic knowledge of AI is needed to understand these tools. Neural networks, a type of AI algorithm used extensively in eyecare will be the focus of this review, however the concepts discussed are relevant to many AI technologies.

Neural networks

Neural networks consist of layers of computational units quite like the layers of neurons in the human visual cortex. A neural network is a mathematical model whereby one or more outputs, usually classifications or predictions, are calculated based on input data. These calculations are performed by passing these data through an arrangement of intermediate or ‘hidden’ layers of nodes which, like neurons in the brain, combine input signals to create an output signal. This process is explained in detail below. An example arrangement is shown in Figure 2, where data are passed through three hidden layers (vertically aligned), each consisting of five nodes.

More complex neural networks, such as those in commercial or research use, have tens or even hundreds of layers with hundreds or thousands of nodes per layer, with the exact architecture determined by the programmer depending on the specifics of the task. Such complexity makes it impossible to understand the influence of each node in arriving at a final decision.

A network with only one hidden layer is called a shallow neural network or a perceptron, while those with more than one layer are known as deep neural networks. The use of deep networks is known as deep learning. Perceptrons are good for solving linearly separable problems, such as dividing two distinct clusters on a two-dimensional graph, but cannot solve more elaborate problems such as detecting retinopathy in an image.²⁶ Generally, more complex networks can solve more difficult problems but have higher memory and processing requirements. They are also harder to train.

The term ‘hidden layer’ derives from their opaque nature. Unlike a decision tree, the user is not privy to the data as it passes through these layers, or how these data are processed; data are entered into the network and one or more outputs are obtained. Hiding these intermediate data is an unfortunate necessity of the algorithm design because the interim data held in these intermediate steps are not interpretable. The role of these layers is to extract features and patterns from the input data, allowing predictions to be made based on these extractions. For example, AI which finds cats and dogs in photos may extract the shape of the eyes and ears and use these to differentiate the two animal classes.

Feature extraction occurs as data pass through nodes in the hidden layers. Each node is connected to at least one input from the previous layer, and at least one output in the following layer. Each connection has an associated parameter, known as a ‘weight’, which determines how these data influence the output value of the next node.

For illustrative purposes, imagine a neural network that determines the risk of diabetic retinopathy based on clinical risk factors, such as glycated haemoglobin levels and duration of diabetes. A node may have two input values: 9.8 representing glycated haemoglobin level, and 14.8 representing years since diagnosis of diabetes. If glycated haemoglobin level is given a weighting of 2.0 and years since diagnosis 0.5, the weighted sum would be 19.6 + 7.4 = 27 which would be passed on to the next layer. Such a model would place higher importance on glycated haemoglobin levels despite the smaller value. This is more complex in practice, with the resultant sum undergoing further processing via an ‘activation function’ which amplifies or diminishes this value based on thresholds or other mathematical processes, allowing the algorithm to learn.²⁷

More layers or nodes do not necessarily mean better performance–different designs will be better at detecting some features to the detriment of others. For example, He et al.²⁸ found that for classifying images in the CIFAR-10 dataset,²⁹ a training image bank of 60,000 images of everyday objects (e.g. animals and vehicles), increasing the number of layers of a ResNet²⁸ neural network while keeping the number of nodes per layer, training epochs and other hyperparameters unchanged, improved performance up to 110 layers, but was considerably worse when using 1202 layers. The number of layers and nodes per layer to use is based on the type and size of the input data, complexity of the problem, size of the training data, and other factors. Developing the appropriate architecture is a complex topic outside of the scope of this article, but this has been addressed in detail elsewhere.^30,31

There are variations in the way each node processes data as they flow through the network, and how data flow from layer to layer. In image processing, additional filtering such as image convolutions are often employed,³² whereby image data are analysed taking into account the surrounding pixels. Additional connections between non-adjacent hidden layers may also be added to improve accuracy and reduce overall complexity.²⁸ There are many other variations, and high-performance systems combine multiple techniques to obtain the best results.

Training and testing

Creating artificial intelligence involves more than just deciding on the network architecture. While the right design is important to enable features and patterns to be extracted, the algorithm needs to ‘learn’ how to do this. Much like humans, AI learns through exposure. This involves sample data and a process to modify the model to maximise accuracy. Both the data and training process shape how well AI will perform with real-life situations.

Data

To train AI, large sets of sample data are required which have been pre-labelled (annotated) by one or more human experts, such as fundus photos from patients with diabetes, which have been graded as referrable or not. These expert labels are known as the ‘ground truth’. Datasets need to be large–often hundreds of thousands of images–and be diverse (age, sex, ethnicity, disease severity, co-morbidities) to include as many different presentations as possible. Ideally these data are obtained from multiple camera systems with varying resolutions and image quality to more closely reflect data used in clinical practice. Complex problems tend to require larger datasets.³³

A diverse, large dataset gives the best opportunity to expose the AI to different clinical presentations. This is imperative to performing well with real-world data–a concept enshrined in the term ‘generalisability’. If a population is not represented in the training data, the AI may not perform well with that population when deployed in the clinic. For example, if a system is trained entirely on images with a particular colour profile or in a homogenous ethnic group, then the features that are extracted may not perform well on another camera system or a patient with different ethnicity.

Moreover, the dataset must comprise a broad range of presentations of referrable and non-referrable disease. For example, if there were very few examples of severe non-proliferative diabetic retinopathy and a preponderance of proliferative disease with vitreous haemorrhage, then the AI system will train to extract features more representative of the proliferative disease and may under-detect cases of severe non-proliferative diabetic retinopathy.

Training process

Once suitable sample data have been obtained, the AI needs to be exposed to each sample to find common patterns and features. Initially, all model connection weights are given random values. Then, one at a time or in batches, samples are passed into the AI. If the output does not match the ground truth, then the training algorithm modifies the weights using techniques like the backpropagation method,³⁴ where only the weights which had the highest impact on the prediction are changed. This continues until all samples have been passed through the system, referred to as an ‘epoch’. Training can involve hundreds or even thousands of epochs to obtain optimum performance. The network is said to converge when these modifications no longer improve the model performance, and training ceases.

Training AI in this way carries a risk of overfitting, where performance is good for the sample data set but is poor on novel data. Model overfitting is a common cause of poor real-world performance of some AI systems. To overcome this, the data may be split into three subsets: training, test, and validation sets.^35,36 Training data are used to train the system with connection weight modification occurring as described above. After each epoch, performance is assessed using the test dataset, which has not been assessed by the AI and is not used to modify weights. If the performance is significantly better with training data compared to test data, this can indicate overfitting. Once the model has converged, a final assessment against the validation set is performed to estimate performance with data that the model has not been exposed to previously.

There are many additional factors to be considered to optimally train a network. For example, the number of epochs, how much to change each connection weights at each step, and the algorithm used to modify weights can all affect performance. These, and any other parameters defined before training begins, are collectively known as hyperparameters. Tuning these values is sufficiently difficult that hyperparameter optimisation, AI developed specifically to tune hyperparameters, is now being used as part of the design process.³⁷ Hyperparameter tuners are part of the training algorithm and do not continue to tune the model after training is complete.

Performance metrics

Accuracy is defined as the portion of predictions which match the ground truth when tested against the validation set. When an AI has a reported accuracy of, say, 80%, this by itself does not describe how well the system performs. This 80% encompasses the overall accuracy, but some predictions may be more accurate than others. A better way to assess this is with sensitivity (true positive rate: the probability of a ‘yes’ response being correct) and specificity (true negative rate: the probability of a ‘no’ response being correct).

To illustrate, consider two different systems developed to detect referable diabetic retinopathy. Metrics for these systems are presented in Table 1, showing accuracy for ‘yes’ (referrable) and ‘no’ (not referrable) separately. System X is accurate 100% of the time for non-referrable images (specificity), but only 60% of the time for referrable (sensitivity). System Y is 80% accurate for both ‘yes’ (sensitivity) and ‘no’ (specificity). Both systems have an overall accuracy of 80% (the average of sensitivity and specificity). System Y’s higher sensitivity may be desirable in a screening program, whereas System X may be preferred as an assistive diagnostic tool.

Table 1. Confusion matrices for two theoretical AI systems. Sensitivity is the fraction of actual yes responses correctly predicted as yes (X:60%, Y:80%). Specificity is the fraction of actual no responses correctly predicted as no (X:100%, Y:80%).

	System X		System Y
	Actual: Yes	Actual: No	Actual: Yes	Actual: No
Predicted: Yes	3	0	4	1
Predicted: No	2	5	1	4
Sensitivity	60%		80%
Specificity	100%		80%

Whilst it may be difficult to achieve 100% for both sensitivity and specificity, parameters can be tweaked to seek different mixes of these metrics. When AI provides an output, this is not a binary yes/no response. Instead, a confidence level between 0 and 1 is produced, known as an output score. For example, an output score of 0.6 indicates a 60% confidence that referrable retinopathy is present in the image. This is considered a ‘yes’ response if it is greater than a predefined threshold, whose value will affect sensitivity and specificity. While it may seem logical to interpret any score over 0.5 as yes and smaller values as no, this threshold could be changed to, say, 0.3 to increase sensitivity, or 0.7 to increase specificity.

Improving sensitivity will generally worsen specificity, and vice versa. There are various ways to determine an appropriate threshold, with receiver operating characteristic curves being one common technique as shown in Figure 3. These graphs show (1 – specificity) on the x-axis, and sensitivity on the y-axis. The line represents how these statistics change for different threshold values, starting with a threshold of 1.0 in the bottom left corner through to 0.0 in the top right. Analysing how the threshold affects these statistics allows an optimal value to be chosen.

Figure 3. Grad-CAM heatmaps showing areas of importance when detecting referable diabetic retinopathy, using two published neural network architectures. A: InceptionResNetV2.³⁸ B: InceptionV3.³⁹ Image from the DDR dataset.⁴⁰ Scale is the impact of this pixel on the final output relative to other pixels in the image. C: receiver operating characteristics curves for the two neural networks: blue is network A, orange is network B.

Furthermore, by calculating the area under the curve, different AI systems can be compared, with higher area values generally considered superior. In this example, the area under the curve is 0.869 for network A (blue) and 0.894 for network B (orange), indicating that network B performs better overall. Note that network B has a higher true positive rate than network A for higher thresholds and worse for lower thresholds. This means network A may be preferable in screening programs, where it is better to catch as many people with disease as possible, even though on further testing a number are found not to have the disease. It is important to note that the sensitivity and specificity thresholds of commercial AI systems are not able to be changed by the clinician.

Explainability

Given the complex and opaque nature of AI systems, it can be difficult to understand how a decision was made, leading to a situation where the clinician is faced with a diagnostic dilemma of accepting or rejecting the AI recommendation with little comparative data. This can be problematic when decisions have the potential to affect patient care.

Techniques have been developed that improve the explainability of AI systems by highlighting image regions which are most influential in the prediction of the AI. The most common approach is the generation of a colour-coded or intensity-graded ‘heatmap’, where the ‘hottest’ regions in the image are those that contributed most to the AI system output.²⁰ Whilst not providing all model parameters or detailing the inner workings, heatmaps provide a method to visually explain the areas of the image that contributed most to the decision made by the AI system.

Various techniques exist to generate these heatmaps. Occlusion-based explanations⁴¹ run the same image through the AI multiple times, occluding or obscuring a different section of the image each time. By analysing how these occlusions affect the prediction, important regions of the image can be identified. The Grad-CAM technique⁴² produces similar results using an efficient but complex process: by capturing error information from the last hidden layer containing image convolutions and tracing these values back to image pixels, regions with the biggest contribution to the final output can be highlighted.

It is important to note that AI systems may find salience in areas that may not be deemed salient by human clinical experts. It is useful for the clinician to consider disparities between regions they deem salient and those deemed important by the AI, as this may help to understand cases where the clinician disagrees with the output. Just because human experts may not recognise an area to be salient does not necessarily relegate this to a flaw in the system; it may be that the system is detecting features that human observers do not.

An example of this is the existence of AI systems that can determine the sex of the participant from a fundus photo.^43–45 Prior to these studies no anatomical retinal differences between the sexes were known. Differences in temporal artery morphology and red saturation of the retina, amongst other characteristics, are now understood.^43,44

While explainability methods such as heatmaps are fundamental to understanding and trusting AI outputs, they are not available in all systems. A lack of incorporation of explainability, such as heatmaps, can erode trust in the system and limits opportunities for the clinician to understand those areas of the image that are most contributory to the classification.

Different AI systems may provide vastly different heatmaps despite producing similar outputs, suggesting that different features or regions are influencing the decisions made by the systems. For example, Figure 3 shows Grad-CAM heatmaps for two AI systems trained to detect referable diabetic retinopathy. The image depicts proliferative retinopathy with neovascularisation at the inferonasal aspect of the optic disc and multiple haemorrhages elsewhere. Both systems detect referrable retinopathy with 80 ± 2% accuracy, and both grade the image as referable with 73% confidence. Despite these similarities, the heatmaps suggest the two systems placed higher importance on different regions. While a single heatmap is relevant only to a single image analysis, repeated heatmap patterns can reveal insights into response patterns of an AI system. For example, if the top portion of the image is never highlighted it may suggest the algorithm is ignoring this area.

For image processing, heatmaps are the dominant method that is used for explainability. Other techniques are being developed but are not widely used as they require changes to the AI algorithm itself, potentially affecting performance. One example is neural-backed decision trees which use neural networks to separate inputs into categories represented in a decision tree,⁴⁶ as shown in Figure 4. While remaining a black-box system, this method shows how objects are categorised using a multi-level approach. There are no limits to how many levels can be implemented. This could highlight where an AI shows high confidence in a broader category but low confidence when classifying a specific object, allowing results to be interpreted accordingly.

Figure 4. Neural-backed decision tree produced by the nbdt software available at (https://github.Com/alvinwan/neural-backed-decision-trees). This example categorises an image into one of ten categories used in the CIFAR-10 dataset.⁴⁷ The green path shows the branches calculated by the algorithm, with the confidence of each decision shown as a percentage. Image ‘Fishing boat Langkawi’ by Bernard Spragg is marked with CC0 1.0.

Caveats

Artificial intelligence is a powerful tool for clinical practice, provided that the limitations of the technology are acknowledged and understood. Arguably the most important factor is fitness for purpose. When using a system designed to detect diabetic retinopathy, the user cannot expect it to detect glaucoma, macular degeneration, or any other pathology. Such a system is blind to other conditions, with ocular emergencies such as retinal detachment being completely ignored. The system may also misclassify diseases with similar presentations; hypertensive retinopathy may be considered diabetic retinopathy if the AI was not specifically trained to detect this condition. This is especially important when such systems are used in screening programs or by clinicians not trained in the identification of these pathologies.

A known issue for healthcare in general is implicit bias,^48,49 limiting patient-clinician relationships and resulting in poorer quality of care for some patients. As datasets used to train AI systems are often curated by expert clinicians, these biases may be introduced into these data and ‘learned’ by the AI.⁵⁰ For example, a clinician may subconsciously grade diabetic retinopathy in a heavily pigmented fundus as less severe than the same presentation in a lightly pigmented fundus. If these data are used in a training dataset, the AI will be ‘taught’ to misclassify heavily pigmented fundus images in the same manner. Similarly, if the clinician tends to overlook subtle lesions, then AI will do the same.

If there is insufficient clinical information to correctly classify an image, the dataset will contain erroneous data and impede the training process. Consequently, the ground truth may not always be true. Errors can be reduced by having ground truth labels created by multiple experts, more experienced experts, and by defining an adjudication procedure if there are differences in opinion.^51,52

Ground truth errors aside, AI users need to consider the generalisability of the system to ensure applicability to the target population. For example, cup/disc ratio is a known risk factor for glaucoma,⁵³ and it has been reported that the Australian Aboriginal population have larger cup/disc ratios than Caucasians.^54,55 A system trained with data from Caucasians may perform poorly for Aboriginal populations if cup/disc ratio, rather than optic nerve rim volume or some other parameter, is determined by the AI to be an associated feature. Where training data demographics are not available, such a system would need to be validated for the target population.

Assuming these concerns have been addressed, it must be noted that AI can only make predictions based on data that it is provided. For example, numerous factors can influence the glaucoma risk of a patient: optic disc parameters,^53,56 intraocular pressure,⁵³ corneal thickness, visual fields,⁵³ retinal nerve fibre layer thickness,^57,58 family history, comorbid obstructive sleep apnoea syndrome,^59,60 and others. Clinicians will consider all of these data when making a diagnosis. AI may perform poorly in glaucoma detection if it is only provided a subset of these data. As a case in point, an AI system is unlikely to detect pre-perimetric glaucoma if it was trained with visual field data from advanced glaucoma with classic arcuate defects.

These limitations are likely to be noted in systems that have been subjected to peer review; however, commercial systems may use proprietary software which has not been subject to scrutiny from the scientific community, and training data may not be available. Such systems should be independently validated to verify system performance. Regulatory bodies oversee these systems when used as medical devices, which may reduce these concerns but does not completely resolve them. As an example, Tufail et al.⁶¹ demonstrated how some USA Food and Drug Administration-approved systems for diabetic retinopathy screening performed worse than expected in some populations, with one flagging all disc-centred images as diseased.

Regulations

In many countries including Australia, software may require regulatory oversight when used in healthcare. The Australian Therapeutic Goods Administration defines software as a medical device if its intended use is:

• Diagnosis, prevention, monitoring, prediction, prognosis or treatment of a disease, injury or disability

• Compensation for an injury or disability

• Investigation of the anatomy or of a physiological process

• To control conception.⁶²

Software used for screening is also covered by this definition. Like other therapeutics, software as medical devices must be included in the Australian Register of Therapeutic Goods before they can be supplied in Australia. This is also true in many other countries. Devices are classified based on risk to individuals or public health, from Class I (lowest risk) to Class III (highest risk). Higher classifications attract more regulatory oversight and higher clinical standards.

Regulating software as medical devices is an evolving field with different countries applying their own regulatory requirements. Challenges include adaptive algorithms that continue to learn based on the real-world data to which they are exposed. These systems differ from their inspected state as soon as they are used, risking the introduction of errors or biases. Such systems require a total product lifecycle regulation method¹⁹ for managing quality, safety, and performance.

Artificial intelligence in optometry

Use in practice

AI-based systems have already been developed and approved for numerous medical fields, including in eyecare,¹⁹ and this trend is set to continue. These systems should be viewed as clinical tools and not a substitute for thorough examinations and clinical judgement. Non-AI assistive technologies are already present in clinical equipment, such as colour coding seen on modern OCT printouts. Misdiagnoses and unnecessary referrals from ‘red’ or ‘green disease’ are well documented,⁶ stemming from misuse or misunderstanding of these indicators. AI systems carry the same risks of misuse but can be valuable if used correctly. Clinicians should:

1. acknowledge the AI prediction;

2. check the input for errors that could lead to invalid results such as misalignment or clipping; and

3. verify the finding with further testing as appropriate, including heatmap assessment if available

No AI system is perfect, and erroneous results will be produced from time to time. Awareness of the limitations of a given system, including sensitivity and specificity metrics, is vital to the effective use of AI in the clinic. Such information, including device indications and limitations, are required to be included in product information available from regulatory bodies such as the Therapeutic Goods Administration (https://www.tga.gov.au) or the Food and Drug Administration (https://www.fda.gov).

Things to look for

Given the complexity of AI systems and use of computer science and mathematical jargon in the AI literature, deciding if AI could and should be used in a clinic can be challenging. A guide to interpreting AI literature has been produced by Ting et al.,³³ however commercial systems are unlikely to make the code or training data available for scrutiny, and the underlying technology is likely to be modified from the academic literature. The following steps can help appraise an AI system for clinical use.

1. Does the system have regulatory approval in your location? This includes prototypes used in clinical trials. Systems without current regulatory approval should not be used in clinical practice. Approved devices are listed on the websites of regulatory bodies such as the Australian Register of Therapeutic Goods (https://www.tga.gov.au/resources/artg) and the Food and Drug Administration (https://www.fda.gov).

2. What is the system designed to do? There can be a difference between what a system is intended to be used for and what you could potentially do with it. For example, a device designed to measure pupil size while determining contact lens parameters should not be used to screen for anisocoria. While off-label use of medical devices is a clinical decision, AI is engineered for a specific task and is unlikely to perform well outside this scope. Regulatory bodies such as the Australian Register of Therapeutic Goods (https://www.tga.gov.au/resources/artg) and the Food and Drug Administration (https://www.fda.gov) list the intended use(s) of all registered medical devices, including software.

3. How will the device be used in the clinic? The sensitivity and specificity of the system may impact how useful the device is to the practice. AI with a high sensitivity and low specificity may be excellent for screening but create too many false positives unless clinicians are prepared to overrule the AI recommendation. These data may be available from regulatory bodies such as the Therapeutic Goods Administration (https://www.tga.gov.au) or the Food and Drug Administration (https://www.fda.gov). Where these data are not available, contact the manufacturer for further information.

4. Is the target population represented in the test data? This is especially important if the device is to be used with minority groups. Where representation cannot be shown, results should be viewed cautiously until performance can be determined with the target group. These data may be available from regulatory bodies or from the manufacturer. If data cannot be obtained, clinicians should look for validation studies in the literature.

5. Does the system explain the outputs? This requirement depends on the cost and risk of erroneous findings and will vary from practice to practice. AI used to detect referrable diabetic retinopathy benefits from heatmap outputs (Figure 3) as the validity of the decision can be assessed by the clinician to avoid unnecessary referrals (‘red disease’). Conversely, AI may be used by dispensing staff in pre-test to assess suitability for contact lenses, where suitable patients are given a pamphlet to read in the waiting room. Explanations are less important in this case as the optometrist will need to undertake a full assessment before fitting, hence the cost and risk of an incorrect result is low.

The future

AI technology is in its infancy, and the systems can be extremely complex and computationally demanding, sometimes limiting their creation to those with specific training and access to expensive computing equipment. However, advances in AI and cloud-based computing have led to code-free deep learning, a technique for creating AI without writing code via a computer accessed remotely.^43,52,63 The performance of these systems is comparable to state-of-the-art AI,⁶³ allowing clinicians without computer science training to participate in clinical AI research.⁵² While this has clear advantages, there are some notable drawbacks. A computer scientist trained in AI will be aware of, and work to mitigate, the limitations and pitfalls described above. This may not be the case for a novice in the field. The increasing availability of large, open datasets for AI research means some concerns relating to training data may be avoided,⁵² yet others like ethnic diversity in training, test and validation data sets still require careful consideration.

Clinician-driven AI will allow clinicians to explore AI systems with the knowledge of exactly how these may be used in the clinic. There is a risk, however, of such bespoke systems being used in the clinic, as a medical device, without obtaining regulatory approval or external validation. Clinicians need to be aware that code-free deep learning is for research purposes only and should not be used for routine clinical care decision making.

In response to concerns about the use of AI in healthcare, the Food and Drug Administration has developed an action plan for artificial intelligence and machine learning-based software as a medical device.⁶⁴ This action plan seeks to improve transparency, ensure high standards of machine learning practices, and address concerns of bias. Future AI systems will be subject to ongoing performance monitoring due to their dynamic nature. This should lead to systems which are trusted by both the clinician and patient.

Conclusion

Artificial intelligence is a powerful tool for eye healthcare providers. The utility of AI stems from its complexity and it is important to understand the limitations arising from this. As with any assistive technology, misuse can lead to misdiagnosis and mismanagement. However, when used as designed, the technology presents new opportunities for earlier diagnosis and effective management. A well-designed and appropriately used AI system may detect subtleties which have been overlooked, or flag an area to re-examine. It is up to the clinician, not the computer, to make the final judgement.

A broad understanding of how AI works is helpful in appraising an AI system or interpreting results, however in-depth knowledge is not required. Using AI effectively in clinical practice requires the system to have regulatory approval, to be designed for the intended use, have appropriate sensitivity and specificity for the needs of the clinician, and be usable with the target population. Methods for explaining outputs such as heatmaps may also be beneficial. Off-label use of AI systems is not recommended as performance is likely to be poor.

Optometrists should embrace AI systems in clinical practice as an assistive tool and not an alternative to a comprehensive examination. Effective use of AI in optometry has the potential to improve accuracy and efficiency of many areas of optometric practice, benefiting both the clinician and the patient.

Disclosure statement

P van Wijngaarden is a co-founder of Enlighten Imaging, an early- stage medical technology company devoted to retinal imaging and image analysis. The other authors report there are no competing interests to declare.