Utilizing artificial intelligence in endoscopy: a clinician’s guide

ABSTRACT

Introduction

Artificial intelligence (AI) that surpasses human ability in image recognition is expected to be applied in the field of gastrointestinal endoscopes. Accordingly, its research and development (R &D) is being actively conducted. With the development of endoscopic diagnosis, there is a shortage of specialists who can perform high-precision endoscopy. We will examine whether AI with excellent image recognition ability can overcome this problem.

Areas covered

Since 2016, papers on artificial intelligence using convolutional neural network (CNN in other word Deep Learning) have been published. CNN is generally capable of more accurate detection and classification than conventional machine learning. This is a review of papers using CNN in the gastrointestinal endoscopy area, along with the reasons why AI is required in clinical practice. We divided this review into four parts: stomach, esophagus, large intestine, and capsule endoscope (small intestine).

Expert opinion

Potential applications for the AI include colorectal polyp detection and differentiation, gastric and esophageal cancer detection, and lesion detection in capsule endoscopy. The accuracy of endoscopic diagnosis will increase if the AI and endoscopist perform the endoscopy together.

KEYWORDS:

• Artificial intelligence
• gastric cancer
• helicobacter pylori
• esophageal squamous cell carcinoma
• colon polyp
• narrow band imaging
• magnified endoscopy
• esophagogastroduodenoscopy
• colonoscopy
• capsule endoscopy

1. Introduction

With convolutional neural network (CNN) technology, high-speed computers, and abundant digital data, the image recognition capability of artificial intelligence (AI) has surpassed that of humans. In medical practice, diagnostic imaging support is the area most suited for the practical use of AI.

This paper discusses the latest research findings on CNN-based imaging diagnostic AI for detecting gastrointestinal tract lesions. A CNN is a type of machine learning model associated with accuracy and is the de facto standard in the field of image recognition. Therefore, we focused on CNN-based, clinically relevant, and physician-oriented studies in this review. A summary of the contemporary and future AI applications for diagnosing diseases affecting the stomach, esophagus, large intestine, and small intestine (using capsule endoscopy) is presented.

2. AI to diagnose stomach disease

2.1. Characteristics of stomach disease and the necessity for AI-based systems

Esophagogastroduodenoscopy (EGD) is the standard procedure used to diagnose diseases of the stomach. However, EGD performance significantly varies among endoscopists owing to long-term specialized training requirements and experience to detect and diagnose early stage gastric cancers and precursor lesions [1]. Several studies have reported that the rate of undetected upper gastrointestinal (GI) cancers over the previous 3 years was 9.8%–25.8% [2–4], mostly because of various endoscopic errors. To address this problem, the imaging diagnostic capability of AI has also been applied to EGD (Figure 1), even though many clinicians believe that AI application would be more difficult for the stomach than for other areas of the GI tract because of the anatomical features of the stomach and because stomach diseases often involve mucosal inflammation, making AI imaging accuracy difficult.

Figure 1. Representative images of gastric cancer detected by artificial intelligence.

(a) A reddish and slightly depressed lesion of gastric cancer appears on the greater curvature of the lower gastric body.(b) The yellow rectangular frame was marked by artificial intelligence as a possible lesion to indicate the extent of a suspected gastric cancer lesion. An endoscopist manually marked the location of the cancer using a green rectangular frame. [0–IIc, 10 mm, tub1, T1a(M)].

Figure 2. Representative images of esophageal cancer detected by artificial intelligence.

(a) Reddish irregular area of esophageal squamous cell carcinoma (ESCC) on the right wall in white light imaging. The endoscopist marked the lesion using a green square, and an artificial intelligence (AI) diagnosing system surrounded the lesion using a white square to diagnose esophageal cancer. Because both squares were matched perfectly, we know the AI diagnosing system could detect ESCC correctly.(b) In narrow band imaging, the same lesion appears as a Brownish area. The AI diagnosing system detected ESCC in the same way.

2.2. Malignant disease diagnosis

Table 1 provides an overview of studies in which image recognition by AI was applied to the stomach. Many clinical AI studies on the stomach have focused on detecting and diagnosing gastric cancer. Hirasawa et al. first reported that AI-based detection may be useful in gastric cancer. Their system was trained using 13,584 high-resolution white light images (WLIs) of gastric cancer, narrowband images (NBIs), and chromoendoscopy using indigo carmine [5]. A validation set of 2296 images showed a sensitivity of 92.2%; however, the positive predictive value (PPV) was only 30.6%, often misdiagnosing non-cancerous lesions as cancer. Ishioka et al. evaluated the performance of the same CNN when applied to videos, and the results were similar to those with still images [6], with a sensitivity of 94.1%. However, PPV was not calculated as only gastric cancers that had previously been diagnosed were targeted. In another study that evaluated gastric cancer detection, Wu et al. reported on AI performance with a validation set of 100 gastric cancers and 100 nonmalignant images, obtaining sensitivity, specificity, and accuracy values of 94.0%, 91.0, and 92.5, respectively [7]. Luo et al. developed a GI AI diagnosis system (GRAIDS) to demonstrate its capability in detecting upper and lower GI cancers, including esophageal cancers [8]. The accuracy for identifying GI cancer ranged from 91.5% to 97.7% with seven validation sets used in their multicenter study. GRAIDS performance was compared with that of endoscopists; GRAIDS achieved a diagnostic sensitivity similar to the expert endoscopists (94.2% vs 94.5%) and superior sensitivity compared with competent (85.8%) and trainee (72.2%) endoscopists.

Table 1. Use of artificial intelligence in stomach field.

	Reference (Year)	Application	Procedure	CNN archi-tecture	Training dataset	Validation dataset	AUC	Accuracy (%)	Sensitivity (%)	Specificity (%)
Malignant disease	Hirasawa (2018) [5]	Detection of GC	WLI, CE, NBI	SSD	51,558 images (abnormal 13,584)	Abnormal 2,296 images	n/a	n/a	92.2	n/a
	Ishioka (2019) [6]	Detection of GC	WLI, CE, NBI	SSD	51,558 images (abnormal 13,584)	Abnormal 68 videos	n/a	n/a	94.1	n/a
	Wu (2019) [7]^†	Detection of GC	WLI, CE, NBI	VGG ResNet	9,151 images (abnormal 3,170)	200 images (abnormal 100)	n/a	92.5	94.0	91.0
	Kanesaka (2018) [9]	Diagnosis of early GC	ME-NBI	SVM	126 images (abnormal 66)	81 images (abnormal 61)	n/a	96.3	96.7	95
	Li (2019) [10]	Diagnosis of early GC	ME-NBI	Inception-v3	2088 images (abnormal 1,702)	341 images (abnormal 170)	n/a	90.91	91.18	90.64
	Horiuchi (2019) [11]	Diagnosis of early GC	ME-NBI	GoogLeNet	2,570 images (abnormal 1,492)	258 images (abnormal 151)	0.85	85.3	95.4	71.0
	Zhu (2019) [12]	Diagnosis of invasion depth of GC	WLI	ResNet	abnormal 790 images	abnormal 203 images	0.94	89.16	76.47	95.56
Non-neoplastic disorder	Shichijo (2017) [13]	Diagnosis of H. pylori infection	WLI, CE, NBI	GoogLeNet	32,208 ‣unclassified/‣classified images from 1,768 patients (abnormal 753)	11,481 images from 397 patients (abnormal 72)	‣0.89 ‣0.93	‣83.1 ‣87.7	‣81.9 ‣88.9	‣83.4 ‣87.4
	Itoh (2018) [14]	Diagnosis of H. pylori infection	WLI	GoogLeNet	149 images (abnormal 70)	30 images (abnormal 15)	0.956	n/a	86.7	86.7
	Nakashima (2018) [15]	Diagnosis of H. pylori infection	‣WLI ‣BLI ‣LCI	GoogLeNet	162 images (abnormal 75)	60 images (abnormal 30)	‣0.66 (WLI) ‣0.96 (BLI-bright) ‣0.95 (LCI)	n/a	‣66.7 (WLI) ‣96.7 (BLI-bright) ‣96.7 (LCI)	‣60 (WLI) ‣86.7 (BLI-bright) ‣83.3 (LCI)
	Guimarães (2020) [16]	Diagnosis of H. pylori infection	WLI	ImageNET	200 images (abnormal 100)	70 images (abnormal 30)	0.981	92.9	100	87.5
Anatomical classification of stomach	Takiyama (2018) [18]	Classification of gastric location	WLI	GoogLeNet	27,335 EGD images (Stomach 20,202)	17,081 EGD images (Stomach 13,048)	0.99	n/a	‣96.9 (U-stomach) ‣95.9 (M-stomach) ‣96.0 (L-stomach)	‣98.5 (U-stomach) ‣98.0 (M-stomach) ‣98.8 (L-stomach)
	Wu (2019) [7]^††	Classification of gastric location	WLI	VGG ResNet	24,549 images	170 images	n/a	‣90.0 (into 10 parts) ‣65.9 (into 26 parts)	n/a	n/a
	Wu (2019) [19]	Monitoring blind spots	WLI	VGG DenseNet	34,513 images	107 videos	n/a	Average 90.02 (range for each site 70.21–100)	Average 87.5 (range for each site 63.4–100)	Average 95.02 (range for each site 75–100)

^{†, ††}Same article

Abbreviations: CNN, convolutional neural network; GC, gastric cancer; WLI, white light image; CE, Chromoendoscopy; NBI, narrow band imaging; SSD, Single Shot MultiBox Detector; ME-MBI, magnifying endoscopy with narrow band imaging; H. pylori, helicobacter pylori; BLI, blue light imaging; LCI, linked color imaging; EGD, Esophagogastroduodenoscopy

Kanesaka et al. identified early gastric cancer on magnified NBIs with an accuracy of 96% [9]. Other studies subsequently reported that CNN technology was efficient for early gastric cancer diagnoses using magnified NBIs [10,11].

With regard to gastric cancer characterization, Zhu et al. applied an AI-based system to evaluate invasion depth using WLIs, with an overall accuracy of 89.16%, significantly higher than that of endoscopists [12].

2.3. Diagnosis of non-neoplastic disorders

AI has also been applied to evaluate non-neoplastic disorders. Shichijo et al. detected and diagnosed Helicobacter pylori gastritis on WLIs, NBIs, and chromoendoscopy images, with a sensitivity of 81.9% and a specificity of 83.4% [13].

Itoh et al. also demonstrated that AI could detect and diagnose H. pylori gastritis on WLIs with a sensitivity and specificity of 86.7% [14]. Nakashima et al. optimized these results using blue-light (BL) imaging and linked color (LC) imaging; the areas under the curve (AUCs) in the receiver operating characteristics (ROC) analysis of WLIs, BL images, and LC images were 0.66, 0.96, and 0.95, respectively [15]. Guimarães et al. also reported that CNN technology could be used to diagnose H. pylori gastritis on WLIs, with an accuracy of 92.9% [16]. Machine learning can also be applied to assess gastroduodenal ulcers and predict the risk of recurrent bleeding [17]; AI identified patients with recurrent ulcer bleeding after 1 year, with an AUC of 0.775 and an overall accuracy of 84.3%.

2.4. Anatomical classification of the stomach

AI systems have also been tested to classify the stomach anatomically. For instance, Takiyama et al. developed a CNN to classify the anatomical location of the upper digestive tract [18], using 27,335 classified EGD images, including 20,202 stomach images, as the training dataset. The AUC of the ROC values was 0.99 for the stomach. In a study on gastric cancer detection, Wu et al. reported that AI could monitor blind spots in the stomach [7]. For their study, they collected images of 26 typical sites in the stomach and used them to train the CNN. If the AI recognized the sites, classified into either 10 or 26 parts, the sites were defined as ‘observed.’ In this test, the AI achieved an accuracy of 90% or 65.9%, respectively, which was better than that of the endoscopists.

Wu et al. subsequently developed the AI system WISENSE (now renamed as ENDOANGEL) to perform a real-time stomach site check [19]. They conducted a randomized controlled trial on 324 patients and reported that using the AI during EGD resulted in a 15% reduction in blind spots. Chen D et al. most recently reported the performance of ENDOANGEL in another clinical trial [20]. They compared the blind spot rate of three EGD types (unsedated ultrafine, sedated conventional, and unsedated) with and without the AI system. They reported that the rate in conventional EGD was the lowest among the three groups and that AI had its largest additional effect in the sedated conventional EGD group.

2.5. Perspectives

Many studies have focused on AI use in the stomach, particularly in the detection and diagnosis of gastric cancer. Some investigations have used magnifying endoscopic images. Further AI studies on the stomach are expected to contribute to the detection and diagnosis of gastric cancer, because a diagnosis of H. pylori infection allows the evaluation of the risk of gastric cancer, while the anatomical classification of the stomach is helpful in the detection of gastric cancer because it ensures that all sites of the stomach are checked. AI systems can assist endoscopists in making accurate diagnoses and will hopefully improve survival rates by detecting gastric cancer at an early stage.

Most of the studies on AI have been reported in Japan and China. The reports in China have mainly assessed patients with gastric cancer at various stages, which provided valuable information on recognizing both early and advanced cancer. However, reports from Japan have focused more on early gastric cancer. Specifically, these reports focused mostly on small, flat lesions, which may be difficult to identify even for expert endoscopists. The use of AI-based diagnostic systems could be more beneficial for detecting smaller lesions or lesions more difficult to identify versus larger lesions, especially since endoscopists rarely miss larger advanced gastric cancerous lesions.

Early diagnosis of H. pylori infection can potentially contribute to premalignancy evaluations. For example, scoring systems for premalignant conditions using AI-based diagnosis of H. pylori infection status (present, eradicated, or non-eradicated), the degree of gastritis, and the degree of intestinal metaplasia might be useful. As another challenge, AI-based diagnostic systems trained by images of the normal stomach mucosa with/without gastric cancer, with/without multiple gastric cancer, with/without advanced gastric cancer, or with/without lymph node metastasis might also be useful for evaluating premalignant conditions. AI could be helpful in providing an appropriate endoscopic surveillance system by identifying candidates a high risk of gastric cancer for gastric cancer.

Furthermore, AI systems may have potential use in other fields. For example, some promising studies have used AI to diagnose non-neoplastic disorders or cancer invasion depth. However, there are currently more investigations on the detection and diagnosis of gastric cancer using AI. The application of AI-based diagnostic systems for gastric cancer invasion by using still images is thought to be difficult because diagnoses require observing tumor shapes that change depending on the degree of air-insufflation. If it is possible to train AI and validate the systems with video WL and EUS images, it may contribute to invasion depth assessments.

The study designs thus far have been mostly retrospective and involved still images, although some studies have used videos. Only a few studies have been conducted as clinical trials; therefore, more real-time clinical trials are needed to confirm the application of AI models in daily clinical practice.

AI has great potential in the field of gastroenterology. Accurate diagnosis using AI may allow for optical biopsies, which could reduce unnecessary biopsies or endoscopic resections, decreasing the risk of bleeding in patients and reducing health expenditures. AI may also help to standardize diagnostic approaches, which are subjective and vary among clinicians and may thus allow high-quality methods to be used, even in institutions with few expert endoscopists.

When AI technology is applied to a routine EGD, our role as endoscopists is to prepare optimal stomach conditions to provide an environment to establish correct diagnoses. The process involves inserting a scope smoothly for relieving discomfort and to resect the tumor safely. The focus should be on improving endoscopic skills even when using AI-based diagnostic systems.

Use of AI in a stomach field requires three factors for clinical usage: detection, classification, and blind spot monitoring. In the near future, when AI models are fully trained to assess all gastric neoplastic and non-neoplastic diseases, including rare ones, and when clinicians confirm the feasibility of the above three factors in a real-life setting, AI-based diagnostic systems may make a more important contribution to the endoscopic diagnosis of stomach diseases.

The use of AI systems for stomach disease evaluations can be summarized as follows: (1) many AI studies of the stomach have focused on the detection and diagnosis of gastric cancer; (2) research in this field has involved magnifying NBIs; (3) studies on the diagnosis of non-neoplastic lesions, particularly H. pylori infection, have also demonstrated promising results; (4) anatomical classification is used to monitor blind spots, with clinical trials being conducted in this field; and (5) the use of AI systems in the stomach requires more clinical trials with video images to confirm their applicability in daily clinical practice.

3. AI to diagnose esophageal disease

3.1. Characteristics of esophageal disease and the necessity for AI-based systems

Esophageal cancer is the seventh most common cancer worldwide [21]. It has two distinct histological types: esophageal squamous cell carcinoma (ESCC), which is common in Asia, the Middle East, Africa, South America, and Japan, and esophageal adenocarcinoma (EAC), which is increasing in the United States and Europe [22]. Diagnosed at an advanced stage, both types of esophageal cancer require highly invasive treatment, and their prognosis is poor. Therefore, early detection is of great importance. However, it is difficult to diagnose early stage esophageal cancer using conventional endoscopy with WLIs. Iodine staining has been used to detect ESCC in high-risk patients. However, it is associated with chest discomfort and increased procedure time [23]. NB imaging is a revolutionary technology involving image-enhanced endoscopy; it can detect superficial ESCC more frequently [24–27]. NB imaging is superior to iodine staining because it is easy to use, involves pushing one button in a moment, and causes no discomfort to patients. However, NB imaging has insufficient sensitivity for detecting ESCC (53%) when used by inexperienced endoscopists [28]; therefore, new effective technologies are needed to detect superficial ESCC.

3.2. ESCC diagnosis

Horie et al. reported a CNN-based AI system that can detect both ESCC and EAC [29] (Figure 2). They trained a CNN system using 8428 endoscopic images of esophageal cancer, including 365 cases of ESCC and 32 of EAC. They then validated the AI system using a different validation data set consisting of 1118 images from 97 patients, including 47 with esophageal cancer and 50 without esophageal cancer. The AI-based diagnostic system analyzed 1118 images in 27 patients and detected cancer in 98% (46/47) of cases. Notably, the system detected all seven lesions less than 10 mm in diameter. The sensitivity of the system for each image was 77%, with a specificity of 79%, a PPV of 39%, and a negative predictive value (NPV) of 95%. Moreover, the system could differentially diagnose superficial and advanced cancer with an accuracy of 98%. Although the PPV was not high, the authors mentioned that deep learning regarding normal structures and benign lesions in the esophagus would improve the PPV.

Ohmori et al. developed a CNN-based AI system that was trained using both non-magnified and magnified endoscopic images of ESCC, non-cancerous lesions, and normal esophagus tissue [30]. The accuracy of the AI system for diagnosing ESCC was comparable to that of experienced endoscopists. The system also achieved a high PPV (76%) for detection using non-magnified images and in the differentiation of ESCC using magnified images.

Cai et al. developed a deep neural network (DNN)-based AI system, trained using 2428 images, to detect ESCC [31]. The system was superior to both experienced and inexperienced endoscopists in detecting ESCC in endoscopic images, with higher sensitivity and specificity values. Interestingly, the sensitivity of endoscopists in detecting ESCC was greater in the image set with a rectangular frame displayed by the AI system to indicate ESCC. Guo et al. trained an AI system using 6473 NBIs and validated both NBIs and video datasets of non-magnified and magnified NBIs [32]. The sensitivity for detecting ESCC in the video datasets was 100% in both the non-magnified and magnified NBIs.

3.3. Pharyngeal cancer diagnosis

Pharyngeal cancer often occurs in patients with ESCC. Recent advances in NB imaging and increased awareness among endoscopists have led to the increased detection of superficial pharyngeal cancer (SPC) during screening upper endoscopy [24,33]. Detected at an early stage, pharyngeal cancer can be treated using endoscopic resection, which ensures safe, effective, and minimally invasive treatment with good outcomes [34–36]. However, most reports on the detection and treatment of SPC are from Japan. These techniques are not prevalent worldwide, and an efficient detection system for SPC is necessary. Tamashiro et al. used a CNN-based AI system to detect SPC; the system was trained using 5403 images of pharyngeal cancer and detected all pharyngeal cancer lesions in the validation datasets, including small lesions [37]. Thus, this AI system could prevent SPCs from being missed in Japan and worldwide.

3.4. Depth of invasion diagnosis

Tokai et al. used a CNN-based AI system to diagnose the invasion depth of ESCC, distinguishing EP-SM1 lesions from those deeper than SM2 in superficial ESCC [38]. The accuracy score of this system exceeded that of 12 out of 13 board-certified endoscopists, and its AUC was greater than that of all endoscopists. Nakagawa et al. also reported good outcomes in a CNN-based AI system for diagnosing the invasion depth of ESCC using both non-magnified and magnified images [39].

3.5. Endocytoscopic diagnosis

Intrapapillary capillary loops (IPCLs) are microvessels that were first characterized using magnifying endoscopy for ESCC [40,41]. They are now used as established markers to diagnose ESCC. Everson et al. reported the AI classification of IPCL patterns in patients with ESCC [42]. Their CNN system could differentiate abnormal and normal IPCL patterns with 93.7% accuracy on magnified NBIs.

The endocytoscopic system (ECS) provides a magnifying endoscopic examination that enables surface epithelial cells to be observed using methylene blue staining [43–45]. The optical magnification power of the ECS is 500× [44], which can be increased to 900× using digital magnification in the video processor. ECS has been used to diagnose benign and malignant lesions with high accuracy [45]. Kumagai et al. used a CNN-based AI system to diagnose ESCC using ECS images [46]. They trained the system using 4715 ECS images of the esophagus. The AI system correctly diagnosed 92.6% of ESCC cases and its overall accuracy was 90.9% in ESC images. They concluded that AI could be used to support endoscopists in diagnosing ESCC based on ECS images, without reference to biopsy-based histology.

3.6. EAC diagnosis

There are limited reports on EAC diagnosis. Ebigbo et al. developed a CNN-based AI system to detect EAC. Their system was trained using 148 images and achieved high sensitivity (92%) and specificity (100%) compared with 11 of the 13 endoscopists [47]. They also validated the system in real time concurrently with the endoscopic examination using 62 images from 14 patients assessed using the AI system. All images were validated pathologically by resection or biopsy and differentiated the normal esophagus from EAC with a sensitivity of 83.7% and a specificity of 100% [48]. One recent study by Hashimoto et al. developed a CNN system that was trained by 916 images of EAC caused by Barrett’s esophagus (BE) indicating high-grade dysplasia and T1 cancer. This system was validated by 458 images with a reported accuracy of 95.4% [49]. The two other studies were reported by de Groof et al. who developed a computer-aided detection (CAD) system using a hybrid ResNet-UNet model system that included 1704 images of BE neoplasia and 669 images of nondysplastic BE [50]. This CAD system achieved higher accuracy than any of the individual 53 international nonexpert endoscopists in the validation process using still images. de Groof et al. evaluated a CAD system during live endoscopic procedures in 20 patients; they obtained WLIs at every 2-cm level of BE and the CAD system reported the results immediately [51]. They reported that 9 of 10 patients with neoplastic lesions were correctly diagnosed with a 90% accuracy rate per level in the analysis.

3.7. Perspectives

Although only a few reports have described AI-based diagnostic systems for esophageal cancer, the number of reports is now rising sharply. Most of these reports have both trained the AI and validated the results using still images, except for one recent report that validated ESCC using videos [32]. Interestingly, two studies on EAC attempted to perform real time detection analysis during endoscopic procedures by still images, which may be useful for assessing conditions such as EAC, because the area of BE is limited. However, detection using videos is more useful as detection can be done promptly without the need to capture too many pictures. However, the detection in videos has to be analyzed well before endoscopists could use during endoscopy. To further investigate whether AI is useful to prevent missed diagnosis of cancers, validation videos should not be limited to slow observation videos but should include fast videos, However, this high accuracy is in high-quality video and is less accurate in videos containing low-quality images.. For detecting superficial cancers, AI systems can be valuable supportive tools for inexperienced endoscopists as well as experienced endoscopists. Because only specialists diagnose invasion depth and use magnifying endoscopy or ECS, Since the invasion depth diagnosis is related to treatment policy decision support, the accuracy of AI needs to exceed that of specialists, which require further improvement. . In addition to endoscopic images, a patient’s information such as past history and laboratory data may be integrated into the AI system to assess disease risk and accurately identify the diagnosis beyond the capabilities of human endoscopists.

4. AI to diagnose colorectal polyps

4.1. Deep learning-based CAD/diagnostic system for colorectal polyps

One of the main tasks of colonoscopy is to detect colorectal polyps. To this end, the roles of AI can be divided into detection and diagnosis of polyps. Recently, many studies have been conducted to develop a computer-assisted detection (CADe) or diagnosis (CADx) system that can be used in real time during colonoscopy.

Colorectal polyps are missed during examination for several reasons: insufficient preparation of the colorectum, the anatomical structure of the colorectum itself (haustra), peristaltic movement, instrumental issues (old scopes or monitors), and lack of expertise, fatigue, or careless mistakes on the part of physicians.

Colorectal polyps, especially adenomas and sessile serrated lesions (SSLs), are thought to be precancerous lesions, and missed polyps can be a cause of interval cancer [52,53]. Therefore, many efforts have been made to decrease the polyp missing rate. In this regard, technological progress in endoscopic devices may have improved the quality of colonoscopy, and image-enhanced endoscopy such as NB imaging has led to an increase in the polyp detection rate and adenoma detection rate (ADR) [54]. In particular, the Endocuff Vision, which is mounted onto the distal tip of the colonoscope and stretches the colorectal fold, has been shown to increase ADR [55], and a recent study showed that a per-oral methylene blue formulation increased ADR [56].

However, even these technologies may not compensate for physician-related factors. In this regard, CADe systems may be a promising technology for next-generation endoscopy. Many studies have been conducted to develop a CADe system, and most indicate that deep learning-based CADe systems are the most promising technology.

4.2. Colorectal polyp diagnosis

Recently, several studies have been conducted on deep learning-based CADe systems of colorectal polyps. Apart from the deep-learning algorithms, there are three major differences between the CADe systems used: (1) training and testing images, which can be static and/or video, and can vary in number; (2) type of image mode included – conventional WLIs and/or enhanced images as well as magnified or unmagnified images; and (3) output styles – whether the CADe system localizes polyps or simply confirms the existence of polyps within the images.

Table 2 shows the studies on AI-based CADe systems [57–64]. The results of each study cannot simply be compared, but taken together, the reports demonstrate that AI-based CADe systems have approximately 95% accuracy in detecting colorectal polyps in static images. However, the accuracy decreases in images from videos that include low-quality images. Thus, to make better CADe systems, low-quality images may need to be included in the training images, not only high-quality images. Additionally, the CADe systems should be tested on video images because they are closer to the actual situations in which the CADe systems would be used. Furthermore, even retrospective evaluation using video images is insufficient, and prospective studies in real-time settings should be conducted to evaluate the actual performance of the CADe system.

Table 2. Use of artificial intelligence in colonic polyp.

	Reference (year)	Study design	Training set	Testing set	Output style	AUC	Accuracy (%)	Sensitivity (%)	Specificity (%)	Miscellaneous
CADe systems	Urban 2018 [57]	Retrospective	‣ 8641 static images ‣ 9 videos (video 1) ‣ WLI and NBI ‣ Normal magnification	‣ 1330 static images ‣ 9 videos (= 44947 frames, video 1) ‣ 11 videos (video 2) ‣ WLI and NBI ‣ Normal magnification	Localize	0.974	96.4%	n/a	n/a	CNN overlay led to the identification of 45 polyps – nine more than the 36 identified by experts in video 1, with 81 false positive polyps, and to the identification of 68 polyps vs. 73 identified by experts in video 2, with 46 false positive polyps.
	Misawa 2018 [58]	Retrospective	‣ 73 videos with 155 polyps (= 63,125 polyp positive frames, 133,496 polyp negative frames) were divided into short videos; 2/3 polyp positive short videos and 4/5 polyp negative short videos were used as the training set; the rest were the testing set. ‣ WLI only		Alert on frame	n/a	76.5%	90%	63.3%
	Wang 2018 [59]	Retrospective	‣ 3634 static images with polyps and 1911 static images without polyps	‣ 5541 static images with polyps, and 21,572 static images without polyps (set A) ‣ 29 public videos with polyps (set B) ‣ 54 videos without polyps (set C)	Localize	‣ 0.984 (set A)		‣94.4% (set A) ‣88.2% (image-based analysis) ‣100% (set B) ‣ (set C)	‣95.9% (set A) ‣95.4% (set C)
	Yamada 2019 [60]	Retrospective	‣ 4087 static images of polyps ‣ videos (891 frames of polyps and 134,983 frames of non-cancerous lesions)	‣ 705 static images with polyps, and 4135 static images without polyps ‣ 77 videos	Localize	‣ 0.975 (static images)		‣ 97.3% (static images) ‣ 74% (frame-based analysis)	‣ 99% (static images) ‣ 94.5% (frame-based analysis)	FNR 26.1%, FPR 1.2% in polyp positive videos and FPR 5.5% in polyp negative videos for frame-based analysis
	Becq 2019 [61]	Prospective (Non-real time manner)	Written above (Wang et al. 2018)	50 videos	Localize	n/a	n/a	98.8%	n/a	PPV 40.6% for polyp detection; PDR for the endoscopist was 62% vs. 82% for the CADe system.
	Klare 2019 [62]	Prospective (real-time manner)	Not addressed (“KoloPol” was used)	55 routine colonoscopies	Localize	n/a	n/a	75.3%	n/a	PDRs 50.9% and ADRs 29.1% (55/73) for polyp-based analysis
	Wang 2019 [63]	Prospective randomized	Written above (Wang et al. 2018)	536 with standard CS vs. 522 CS with CAD system	Localize	n/a	n/a	n/a	n/a	ADR was significantly increased in the group with the CAD system (29.1% vs. 20.3%)
	Wang 2020 [64]	Double-blind randomized	Written above (Wang et al. 2018)	478 sham system vs. 484 CAD system	Localize	n/a	n/a	n/a	n/a	ADR was significantly higher in the group with the CAD system (32% vs. 28%)
CADx systems	Byrne 2017 [78]	Retrospective	‣ 223 videos of polyp (= 60,089 frames, 29% NICE type 1, 53% NICE type 2, 18% no polyps) ‣ NBI only ‣ Normal and near focus	40 videos (as a validation set) 125 videos with 51 HPs and 74 ADs (as a testing set)	NICE type 1 vs. type 2	0.95	94%	98%	83%	PPV 90%, and NPV 97% for adenoma
	Komeda 2017 [79]	Retrospective	‣ 1200 images from videos ‣ WLI, NBI, and CE	‣ Cross-validation ‣ 10 videos	Adenomatous vs. Non- adenomatous	n/a	75.1%	n/a	n/a	70% of the polyps were correct in colonoscopy videos.
	Chen 2018 [80]	Prospective (Non-real time manner)	‣ Static images of 1476 NPs and 681 HPs ‣ NBI only	‣ 188 images of NPs and 96 images of HPs ‣ < 5 mm in size ‣ Static image ‣ NBI only	Neoplastic vs. hyperplastic	n/a	90.1%	96.3%	78.1%	PPV 89.6%, and NPV 91.5%
	Song 2020 [81]	Retrospective	‣ 12,480 static images from 624 polyps (48 HPs, 76 SSLs, 393 BAs, 62 MSMCs, 45 DSMCs) ‣ NBI near focus images	‣ 182 polyps (15 HPs, 24 SSLs, 106 BAs, 20 MSMCs, 17 DSMCs) (set 1) ‣ 363 polyps (30 HPs, 70 SSLs, 206 BAs, 28 MSMCs, 29 DSMCs) (set 2)	SSL vs. BA/MSMC vs. DSMC	‣ 0.93 (SSL)(set 1) ‣ 0.86 (BA/MSMC)(set 1) ‣ 0.91 (DSMC)(set 1) ‣ 0.95 (SSL)(set 2) ‣ 0.89 (BA/MSMC)(set 2) ‣ 0.89 (DSMC)(set 2)	81.3% (set 1) 82.4% (set 2)	‣ 82.1% (SSL)(set 1) ‣ 84.1% (BA/MSMC)(set 1) ‣ 58.5% (DSMC)(set 1) ‣ 74.0% (SSL)(set 2) ‣ 88.5% (BA/MSMC)(set 2) ‣ 62.1% (DSMC)(set 2)	‣ 93.7% (SSL)(set 1) ‣ 88.5% (BA/MSMC)(set 1) ‣ 93.3% (DSMC)(set 1) ‣ 93.5% (SSL)(set 2) ‣ 72.1% (BA/MSMC)(set 2) ‣ 96.7% (DSMC)(set 2)
	Zachariah 2020 [82]	Retrospective	‣ 5278 static images (3310 adenomatous and 1968 serrated polyps) ‣ WLI and NBI	‣ 634 static images	Adenomatous vs. serrated (HPs and SSLs)	n/a	94%	96%	90%	PPV 94%, and NPV 93%

AUC, area under the curve; CNN, convolutional neural network; WLI, white light image; NBI, narrow band image; FPR, false positive rate; FNR, false negative rate; PDR, polyp detection rate; ADR, adenoma detection rate; CS, colonoscopy; HP, hyperplastic polyp; AD, adenoma; CE, chromoendoscopy; NP neoplastic polyp; PPV, positive predictive value; NPV, negative predictive value; SSL, sessile serrated lesion; BA, basic adenoma; MSMC, mucosal or superficial submucosal cancer; DSMC, deep submucosal cancer

Table 3. Deep learning-based systems for capsule endoscopy.

	Reference (Year)	Classification or detection	Capsule system	CNN archi-tecture	No. of training/testing images	AUC	Accu-racy	Sensi-tivity	Speci-ficity
Developing CNN system	Zou (2015) [90]	Localization	n/a	AlexNet	6000/1500	n/a	95.5%	n/a	n/a
	Segui (2016) [92]	Scene classification	PillCam® SB2	n/a	100,000/20,000	n/a	96.0%	n/a	n/a
	Yuan (2017) [65]	Polyp	n/a	SSAEIM	4000 (abnormal 1000)/n/a	n/a	98.0%	n/a	n/a
	Zhou (2017) [66]	Celiac disease	PillCam® SB2	GoogLeNet	400 (abnormal 200) /n/a	n/a	n/a	100% per patient	100% per patient
	He (2018) [67]	Hookworm	n/a	n/a	441,624 (abnormal 4828)/11-fold cross-validation	0.895	88.5%	84.6%	88.6%
	Jia (2016) [91]	Bleeding	n/a	AlexNet	8200 (abnormal 2050)/1800 (abnormal 800)	n/a	0.9955 (F1 score)	99.2%	n/a
	Li (2017) [68]	Bleeding	n/a	LeNet AlexNet GoogLeNet VGG-Net	9672 (abnormal 312)/2418 (abnormal 78)	n/a	96.82 (F-measure) 98.72 (F-measure)99.36 (F-measure) 98.72 (F-measure)	96.20% 98.72% 98.73% 98.72%	99.91% 99.96% 100% 99.96%
	Aoki (2019) [89]	Bleeding	PillCam® SB2/SB3	ResNet	27,847 (abnormal 6503)/10,208 (abnormal 208)	0.9998	99.89%	96.63%	99.96%
	Aoki (2019) [69]	Erosion/ulcer	PillCam® SB2/SB3	SSD	5360 (abnormal 5360)/10,440 (abnormal 440)	0.958	90.8%	88.2%	90.9%
	Fan (2019) [70]	Erosion ulcer	PillCam®	AlexNet	12,910 (abnormal 5910)/8250 (abnormal 2250)	0.9863 0.9891	95.34% 95.16%	93.67% 96.80%	95.98% 94.79%
	Alaskar (2019) [71]	Ulcer*	n/a	AlexNet GoogLeNet	336 (abnormal 256)/105 (abnormal 80)	1 1	100% 100%	100% 100%	100% 100%
	Klang (2019) [72]	Ulcer	PillCam® SB3	Xception	17,640 (abnormal 7391)/5-fold cross-validation	0.99	95.4%–96.7%	92.5%–97.1%	96.0%–98.1%
	Leenhardt (2019) [67]	Angioectasia	PillCam® SB3	Semantic Segmen-tation Algorithm	600 (abnormal 300)/600 (abnormal 300)	n/a	n/a	100%	96.0%
	Tsuboi (2019) [73]	Angioectasia	PillCam® SB2/SB3	SSD	2237 (abnormal 2237)/10,448 (abnormal 488)	0.998	98.4%	98.8%	98.4%
	Iakovidis (2018) [99]	Various lesions**	MiroCam®	WCNN	852 (abnormal 423)/344 (abnormal 172)	n/a	89.2%	92.4%	85.8%
	Ding (2019)^† [92]	Various lesions***	Ankon CE system	ResNet	158,235 (abnormal n/a)/113,268,334 (abnormal n/a)	n/a	n/a	95.4%	96.99%
	Saito (2020) [93]	Various lesions****	PillCam® SB2/SB3	SSD	30584 (abnormal 30584) /17507 (abnormal 7507)	0.911	84.5%	90.7%	79.8%
	Reference (Year)	Class	Capsule system	CNN model	No. of testing videos	Read-ing time by only physicians	Reading time by physi-cians using CNN	Sensi-tivity per lesion	Sensi-tivity per video
Test using full-videos	Ding (2019)^†† [92]	Various lesions***	Ankon CE system	ResNet	5000	96.6 min	5.9 min	99.90%	99.88%
	Aoki (2019) [99]	Erosion/ulcer	PillCam® SB3	SSD	20	12.2 min (ex-pert) 207 min (trainee)	3.1 min (expert) 5.2 min (trainee)	87% (expert) 55% (trainee)	n/a

^†,††Same article.

*Esophageal and gastric ulcers

**Vascular anomalies (small bowel angioectasia, lymphangiectasias, and blood in the lumen), polypoid anomalies (lymphoid nodular hyperplasia, lymphoma, and Peutz-Jeghers polyps), and inflammatory anomalies (ulcers, aphthae, mucosal breaks with surrounding erythema, cobblestone mucosa, luminal stenoses and/or fibrotic strictures, and mucosal/villous edema).

***Clinically significant abnormal lesions (inflammation, ulcer, polyps, protruding lesion, vascular disease, bleeding, parasite, and diverticulum) and normal variants (lymphangiectasia, lymphoid hyperplasia).

Abbreviations: CNN, convolutional neural network; SSAEIM, stacked sparse autoencoder with image manifold constraint; SSD, Single Shot MultiBox Detector; WCNN, Weakly-supervised CNN; CE capsule endoscopy

In these studies, both sensitivity and specificity or PPV are important because these CADe systems are supposed to be used during colonoscopy; too many false positives can be annoying and may cause clinicians to overlook real events. Indeed, the CADe system created by Wang et al. showed good sensitivity, although the same value was fairly low in a prospective study using the same system [74].

Wang et al. recently conducted a prospective randomized controlled trial and subsequent double-blind randomized controlled trial using their own CADe system [63,64]. These studies demonstrated that the CADe system was efficient; the ADR was significantly higher in the CADe system group than in the standard colonoscopy group (29.1% vs. 20.3%, p < 0.001) in the former study, and the CADe system also showed higher ADR than the sham system (34% vs. 28%; p = 0.030) in the latter study. The latter study used a sham system to create a double-blinded scenario, although a randomized tandem colonoscopy study may have been more appropriate to evaluate the benefits of the CADe system.

Although these studies show that CADe systems are promising for the detection and diagnosis of colorectal polyps, the system’s performance for small polyps or flat lesions is not clear, and further evaluation may be necessary.

4.3. Computer-assisted diagnostic systems for colorectal polyps

Recently, because of advances in histopathological knowledge, improvement in instruments, and progress of new image-enhanced technologies, expert colonoscopists may be able to perform optical diagnosis of colorectal polyps with high accuracy in both colorectal polyp classification and prediction of colorectal cancer depth.

In terms of the optical classification of polyps, because histopathological examination after removal of diminutive polyps is costly and time-consuming, the American Society for Gastrointestinal Endoscopy recommends ‘diagnose and leave’ and ‘resect and discard’ strategies. To achieve these strategies, optical classification with high accuracy is necessary. However, special skills and training are required to reach a sufficient level [74]. The European Society of Gastrointestinal Endoscopy advises that optical diagnosis should be performed in a strictly controlled environment by experienced, trained, and audited endoscopists [75]. Therefore, CADx systems with specialist-level diagnostic skills would be useful to support trainees and help reduce unnecessary biopsy or histopathological examinations.

The prediction of cancer depth is also an important issue. Owing to recent improvements in endoscopic devices, endoscopic submucosal dissection (ESD) is widely performed in cases of large benign colorectal polyps or submucosal colorectal cancers. However, ESD of submucosal cancers with deep invasion is technically difficult and can cause lymph node metastases in 10%–15% of cases. Thus, these cancers should be treated using radical surgery.

In Japan, examination of pit patterns using magnifying chromoendoscopy or the Japan NBI Expert Team classification is recommended to diagnose the classes of polyps and predict deep submucosal invasion [76]. In Western countries, the NBI International Colorectal Endoscopic classification has been validated as a useful method for diagnosing polyps or predicting deep submucosal lesions [77]. However, these classifications have considerable inter-observer differences and require expertise. Therefore, CADx systems may be a highly promising technology for predicting the histopathology of polyps and the presence of deep submucosal lesions. Recently, deep learning-based CADx systems have also been developed.

Five AI-based CADx systems using standard colonoscopy have been reported (Table 1) [78–82]. Four of these were developed to perform binary classification: hyperplastic polyps (serrated polyps or non-adenomatous polyps) vs. adenomatous polyps [78,79,82]. Two of them used only NBIs to differentiate adenoma from non-adenoma, with accuracy values of 90%–95% and NPVs of more than 90%, which meet the PIVI requirement [78–80]. Two studies used WLIs to develop the CADx systems; in one of these, the accuracy was only 75% to differentiate adenoma from non-adenoma. However, the number of training images used was only 1200 [79]. In another study using both WLIs and NBIs, the accuracy was 92% using WLIs, which was comparable to that using NBIs [82].

Another approach involves the use of an endocytoscope. Kudo et al. developed a CADx system for endocytoscopy, referred to as endocytoscopy Endo-BRAIN® [83,84]. Endocytoscopy has two important abilities: 1) in vivo observation of cell nuclei and crypt structure using methylene blue staining and (2) observation of microvessels using NBI mode. In this retrospective study using static endocytoscopy images from 100 patients, the CADx system showed 98% accuracy, 96.9% sensitivity, 100% specificity, 100% PPV, and 94.6% NPV for the differentiation of neoplastic vs. non-neoplastic polyps [84]. Although endocytoscopy is still not widely used in clinical settings, these results show the promising role of Endo-BRAIN®.

Song et al. developed a CADx system to classify polyps into three classes: SSLs, benign adenoma/mucosal or superficial submucosal cancers, and deep submucosal cancers [81]. The overall diagnostic accuracy of this system was 81%–82% higher than that of trainees and comparable to that of experts. Furthermore, the diagnostic accuracy of the trainees improved from 64%–72% to 83%–84% with the assistance of the CADx system.

Although these results show robust performance from CADx systems, it must be noted that physicians’ technical expertise is important to diagnose polyps properly; in cases of small polyps with uniform surface patterns and/or vascular patterns, it is not difficult to diagnose the class of polyps. The problem occurs when the polyp is large and has a non-uniform surface pattern/vascular pattern. Such polyps can conceal advanced adenoma or cancer in only a small part of the tumor. Magnifying colonoscopy or endocytoscopy is more useful for diagnosing polyps; however, endoscopists must still show the surface of the most advanced lesions appropriately. Thus, prospective studies in real-time should be conducted in the near future to evaluate the performance of CADx systems in real clinics.

4.4. Perspectives

In conclusion, CADe and CADx systems for diagnosing colorectal polyps have been progressing dramatically in recent years, and previous studies have shown robust performance of these systems. Furthermore, the combination of CADe/CADx systems with other technologies, such as full-spectrum colonoscopy, may produce synergistic effects [85].

With regard to colorectal polyps, the accuracy of AI is about the same as that of human physicians. Furthermore, clinical studies and prospective trials have reported its applicability in actual clinical situations. Validations have been performed with previous still images and videos. Future studies should develop standards that can compare the excellent points of each AI, establish guidelines for its common use, and achieve cost-effectiveness to enable insurance coverage.

In clinical practice, large tumors are rarely missed during colonoscopy; therefore, AI may be less useful in finding large tumorous lesions. However, for large tumors, AI may be useful for predicting the efficacy of anticancer agents in combination with clinical course information and the efficacy of radiotherapy for tumors in combination with clinical information.

The large intestine has a length of 1.5 m or more, and the folds must be separated by an endoscopist during examination. With current AI systems, we have not been able to eliminate the obstacles in viewing the tissue among the folds, and because the large intestine is a winding tube, the prospects of being able to automate endoscopic operations are limited. Therefore, for AI application in these fields, new breakthroughs other than CNNs will be needed.

Although several issues remain to be resolved, including regulatory approval and legal issues, these technologies will likely be introduced within clinics in the near future.

5. AI in capsule endoscopy

5.1. Characteristics of capsule endoscopy and the necessity of automatic detection systems

Capsule endoscopy has marked a revolution for investigating various small bowel abnormalities such as mucosal breaks, angioectasia, protruding lesions, and presence of bleeding. (Figure 3). Capsule endoscopy uses a tiny wireless camera to capture pictures of the GI tract. A capsule endoscopy camera is placed inside a vitamin-size capsule that patients swallow. As the capsule travels through the GI tract, the camera takes thousands of pictures that are transmitted to a recorder. At the time of this publication, around 10 types of capsule endoscopies from five companies are commercially available. Although it is relatively simple to perform during patient examination, capsule endoscopy has major drawbacks for physicians. The greatest characteristic of capsule endoscopy is the automatic capture of the GI tract. A review of around 50,000 images per patient is time-consuming for physicians. In addition, physicians fear the risk of oversight because abnormalities may appear in a few frames. Indeed, trainee endoscopists have been reported to overlook more abnormalities than expert endoscopists [86]. A computer-aided supporting system would help reduce the reading time and the oversights by automatically detecting GI tract abnormalities.

Figure 3. Representative images of various abnormalities captured by capsule endoscopy.

Thus, automatic detection systems are necessary, and several attempts have been made to develop them. For example, capsule endoscopy systems tend to have their own proprietary software for decreasing images (e.g., the QuickView mode on PillCam® system, the omni mode on the EndoCapsule® system, and the express view mode on the MiroCam® system). In addition, a number of systems currently under investigation is based on machine learning methods, such as support vector machines, binary classifiers, or neural networks [87]. However, these software and systems are not yet used in clinical practice. Recently, state-of-the-art deep learning-based methods have shown significantly improved recognition performance in various medical fields [88,89] and are expected to overcome problems in capsule endoscopy reading.

5.2. Developing deep learning-based systems for capsule endoscopy

Recent reports have adopted CNN architectures, which are deep learning-based systems [90]. In general, the diagnostic ability of a CNN-based system depends on the training data quality and amount. However, capsule endoscopy image quality is limited by light limitations, poor focus, low resolution, unpredictable orientation due to free motion of the capsule, and obscuration by bubbles, debris, and bile. In addition, it is difficult to gather images of each abnormality because small bowel diseases are rare. Therefore, such systems are difficult to develop, especially for detecting various abnormalities in the small bowel.

As shown in Table 3, most previous studies have trained a CNN using images of one type of abnormality (polyp, celiac disease, hookworm, bleeding, erosion/ulcer, or angioectasia) and tested the ability of the CNN using selected still images. Most reports have focused on small bowel capsule endoscopy, although capsule endoscopy systems for other elements of the GI tract have been commercially available. The major capsule system studied in this field was the PillCam® system (Medtronic, Minneapolis, MN, USA).

Generally, these CNNs showed high accuracy. Almost all systems achieved more than 90% accuracy, sensitivity, and specificity, with an AUC of more than 0.9 in each image analysis. In particular, systems for detecting bleeding or angioectasia demonstrated extremely high performance (accuracy > 98% and AUC > 0.99) [91], probably because the color cue of these findings was obvious.

CNN-based systems, which can automatically derive features from images, have shown better performance than previous handcrafted image-based analyzes. For example, a CNN-based method by Zou et al. classified digestive organs (the stomach, small bowel, or colon) in capsule endoscopy with an accuracy of 95.5% – higher than conventional scale-invariant feature transform- and SVM-based approaches (90.3%) [90]. Similarly, a CNN-based method by Segui et al. classified motility events (turbid, bubbles, clear blob, wrinkle, or wall) in capsule endoscopy with an accuracy of 96.0% – higher than the conventional approach of combining handcrafted features (82.8%) [92].

Among several CNN-based methods, methodological differences may affect the results. He et al. [93] and Li et al. [94] compared multiple CNN methods in their studies. He et al. reported that their own proprietary deep hookworm detection framework based on CNN showed a higher AUC (0.895) than previous CNN methods such as AlexNet (0.769) and GoogleNet (0.883). Le et al. demonstrated that LeNet was inferior to other types of CNN methods such as AlexNet, GoogLeNet, and VGG-Net regarding F-measure and sensitivity in bleeding detection.

Most recently, CNNs for detecting various lesions have been reported [95–97]. An outstanding report from Ding et al. showed high sensitivity (95.4%) and specificity (97.0%) of a CNN per image analysis. This system was developed and tested using copious amounts of data, including images of various abnormalities and normal images [96].

5.3. Clinical usefulness of deep learning-based systems

It is important that researchers investigate the clinical usefulness of CNNs using full videos. However, only a few such studies have been conducted. Aoki et al. reported the clinical usefulness of their proprietary CNN for detecting erosion/ulcers using 20 full videos of small bowel capsule endoscopy that included a total of 37 erosions/ulcers [99]. A comparison was made between two reading methods (endoscopist alone vs. endoscopist after first screening by a CNN) based on the reading time and detection rate of erosion/ulcers by endoscopists. The mean reading time of small bowel sections by expert endoscopists was significantly shorter after the first screening by the CNN (3.1 min) than by the endoscopist alone (12.2 min). For 37 erosion/ulcers, the detection rate by experts did not significantly decrease in endoscopist readings after the first screening by the CNN (87%) compared to endoscopist-alone readings (84%). Similar trends were observed when the CNN was used by trainee endoscopists, whereby the CNN could reduce the reading time of physicians without decreasing the detection rate of erosion/ulcers. This study suggested that the CNN could be used as a first screening tool.

Ding et al. reported the clinical usefulness of their original CNN for detecting various types of abnormalities, using 5000 full videos of small bowel capsule endoscopy that included a total of 4206 various lesions [96]. In addition to the method reported by Aoki et al. [99], two reading methods (endoscopist-alone readings vs. endoscopist readings after the first screening by the CNN) were compared in terms of reading time and detection rate of abnormalities by endoscopists. The mean reading time per patient was 96.6 min for endoscopist-alone readings and 5.9 min for CNN-based auxiliary readings (p < 0.001). Detection rates of abnormalities were significantly higher for CNN-based auxiliary readings than for endoscopist-alone readings (per-lesion analysis: 99.90% vs. 76.89%; per-patient analysis: 99.88% vs. 74.57%). The same report indicated that the auxiliary use of a CNN system could reduce oversight and decrease reading time. This system was developed and tested based on the Ankon capsule endoscopy system (Ankon Medical Technologies, Hubei, China).

5.4. Perspectives

Several promising CNN-based systems have been reported. The biggest advantage of those systems is reducing the reading time of physicians, which has been shown to play an important role in previous reports that investigated the clinical usefulness of CNNs. However, none of them are commercially available. To allow clinical use, CNN systems to diagnose various abnormalities should be tested using a large number of full videos. Many previous studies using full videos have concluded that the existing QuickView mode of the PillCam® system was unsuitable as a first screening tool because it had an unacceptable miss rate for noteworthy abnormalities, although the mode could reduce the reading time of physicians [86]. In addition, the generalizability of CNNs to multiple capsule endoscopy systems can be evaluated by tests using full videos from several types of capsule endoscopy systems.

Although the aforementioned studies have reported that CNN systems show high performance, the results cannot be directly compared because different test datasets and different evaluation methods were used. Recently, Koulaouzidis et al. proposed the KID database using the MiroCam® system (IntroMedic Co., Seoul, Korea) [98]. It is a publicly available database of annotated capsule endoscopy images and videos that can be used as a reference dataset. Such datasets would accelerate investigations into the generalizability of CNNs and would allow direct comparison between different CNN systems.

The output method of CNN findings may affect the clinical usefulness. For example, Single Shot MultiBox Detector can depict bounding boxes in images to localize the region of interest. Bounding boxes would be useful landmarks for reducing oversight and reading time if the CNN system is highly accurate. However, these boxes can be confusing for physicians if they emphasize areas other than abnormalities, which occurs with low-quality CNNs.

Furthermore, advances in capsule endoscopy are expected in the field, although we focused on CNN systems for small bowel capsule endoscopy in this review. Capsule endoscopy systems for the esophagus and colon are now available, and magnetically guided capsule endoscopies for gastric examination are performed in some institutions. Magnetically guided capsule endoscopy could be a promising alternative screening tool in an era of COVID-19, as this tool is less invasive and would have a lower risk of infection than gastroscopy. Developing CNN systems for different capsule endoscopic applications would be beneficial.

Capsule endoscopy applications can be summarized as follows: (1) capsule endoscopy has the unique characteristic of automatic capturing; (2) previous software and machine learning-based systems have failed to resolve the drawbacks of video reading; (3) promising CNN systems have recently been developed; and (4) these CNN systems require tests using a large number of full videos to investigate their clinical usefulness and generalizability to multiple types of capsule endoscopy systems.

6. Expert opinion

　In recent years, CNN-based AI has been applied in image recognition. Evidently, the ability of AI in general image recognition exceeds that of humans. Although AI was initially applied only to radiographic images (chest X-ray, computed tomography (CT), magnetic resonance imaging (MRI), etc.), research and development (R&D) later expanded its application to GIT endoscopy.

Contemporary AI is only specialized in analyzing image information. A multipurpose AI methodology that can conduct a comprehensive diagnosis by combining medical history and clinical laboratory data does not exist. Developing such an AI method is impossible at present. Currently, AI assists doctors with diagnosis, but is unable to make independent diagnostic decisions.

AI can provide a probable diagnosis, but not a definitive diagnosis. Moreover, AI cannot perceive patients’ emotional status. Doctors can not only convey the diagnosis result in an easy-to-understand manner but also guide or give advice in consideration of the patient’s state of mind. Therefore, developing an AI method that can eliminate the need for the endoscopists is unlikely.

Developing AI to support endoscopic diagnoses benefits medical practitioners. By applying AI in endoscopic examinations, endoscopists can ensure high-quality endoscopy, reduce the burden of additional investigations, and reduce the number of errors. This can be achieved without changing the current technique because endoscopic AI only provides diagnostic assistance during endoscopic examinations. This may also lead to reduced medical errors and increase accuracy of test results in medical organizations. Improving examination accuracy will reduce medical errors and justify the cost of introducing AI for endoscopies.

The introduction of AI to endoscopic clinical practice will also decrease endoscopists’ training time, reduce the burden on training instructors, and compensate for age-related vision problems in experienced doctors. Currently, AI is being used as a tool to assist doctors in diagnosis. Doctors who employ AI-based diagnosis will provide a more accurate diagnosis and treatment.

As outlined in this review, it has become clear at the research level that endoscopic AI can be used with high accuracy in each organ of the gastrointestinal tract. In addition, accuracy validation of AI has moved past simply verifying the AI system capabilities against previous images or movies and is now being verified against the work of an endoscopist. AI has achieved results comparable or superior to those of endoscopists.

On the other hand, if each AI method is validated with different validation materials, it is not possible to determine which AI is superior.

If standardized validation criteria for each disease are established, it may be possible to differentiate with an accuracy that is easy to understand. At the same time, the accumulation of prospective research evidence will prove the effectiveness of AI in clinical practice, and the future of endoscopic AI as a standard medical device in medical practice guidelines will be possible.

There are a number of technical challenges with AI, the current CNN-based AI requires a large amount of teacher data (usually more than 1000 images), and thus AI development for rare diseases with a small amount of case data is difficult. In addition, there is currently a limit to the number of pixels that can be captured and analyzed by AI, and AI analyzes images of lower image quality than those taken with actual endoscopy equipment. A technical challenge remains as to whether AI accuracy can be improved, as the image quality of endoscopic devices progresses. However, there is a possibility that these challenges can be overcome by establishing a system to collect data not only in each country but globally and by developing a new AI algorithm.

It will be difficult for AI to perform endoscopy operations on behalf of humans for the time being because the shape of the digestive tract varies from person to person. Even if AI assists in diagnosis during the endoscopic examination, the endoscope will continue to be operated by a person. Therefore, efforts to improve the technique for operating endoscopes will continue to be essential for endoscopists.

Capsule endoscopy that can observe the entire digestive tract may be developed in the near future. In that case, there will be an era in which the screening examination is first performed with a capsule endoscopy, which is less invasive to the patient, and additional detailed examination is performed with the upper and lower endoscopies depending on the result. Considering the importance of image interpretation with a capsule endoscopy, highly accurate AI will play an important role in reducing this interpretation load on doctors.

In the near future, endoscopic AI will inevitably become commonplace if sufficient evidence is accumulated for endoscopic AI to be recommended in guidelines or covered by insurance. It has not been examined so far, but by that time, the evaluation of the medical economic impact of AI will be advanced.

If it is possible to develop AI that not only assists diagnosis with images but also comprehensively judges clinical progress and clinical test results, AI capabilities will expand to a range of activities to not only support diagnosis but also support treatment policy decisions.

In addition, endoscopic equipment may not only improve image quality but also become three-dimensional, and along with other sensing technologies, it may become possible to perform 3D mapping of organs with endoscopy. What future advances could there be if AI technology evolves along the same lines as the above-mentioned endoscope? There is no end to the possible development of this field.

7. Conclusion

This review outlines the current research and development status and the prospects of AI application in GI tract endoscopy. Employing AI-based endoscopes would enable early cancer detection and consequent improved prognosis. AI technology use for stomach conditions may be beneficial in terms of better diagnostic ability as early gastric cancer diagnostic ability and accuracy varies greatly among endoscopists. AI-based diagnosis of esophageal cancer has improved over the past few years. Without technical difficulty of surveillance, AI would eliminate the misdetection of esophageal cancer in the near future. The use of AI for assessing conditions of the large intestine may help lower cancer incidence by identifying polyps and performing polypectomies at an early stage, and differential diagnosis support may reduce unnecessary biopsy and pathological examination. Furthermore, implementing AI systems with capsule endoscopies would reduce the image interpretation time with unaltered accuracy, which could lead to further applications in capsule endoscopies.

AI will contribute to the development of endoscopic medicine by assisting doctors, but not by lowering their significance. Endoscopists will require the skills to understand and utilize AI.

Article highlights

• AI is effective in image analysis, where in it outperforms the manual interpretations.

• The introduction of the AI in the image analysis will support medical diagnosis.

• AI-based gastric investigations were focused on the detection and diagnosis of the gastric cancers.

• AI used for anatomical classification of the stomach, will monitor unnoticeable regions and reduce the number of undetected cancers.

• AI-based investigations in the esophagus include esophageal squamous cell carcinoma and adenocarcinoma.

• AI was employed in the detection and classification of the colon polyps in the GIT endoscopy.

• AI-based capsule endoscopy would accurately detect the lesions and reduce the interpretation time.

Declaration of interest

Tada T is a shareholder of AI Medical Service Inc. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose

Acknowledgments

We would like to thank Mitsue Takahashi and Motoi Miura for providing assistance in editing manuscript.

We would like to thank Editage (www.editage.com) for English language editing.

Additional information

Funding

This paper was not funded.