Localization strategies for robotic endoscopic capsules: a review

ABSTRACT

Introduction: Nowadays, mass screening campaigns for colorectal cancer diagnosis in the early and curable stage is essential yet limited due to many reasons, for example, invasiveness, fear of pain, and embarrassment for patients. Indeed, mass screening programs, allowing precancerous lesion detection, are the most cost-effective way to reduce mortality and potentially eradicate colorectal cancer threat. Accurate localization represents a key element in capsule endoscopy, that is, estimation of position and orientation of endoscopic capsule devices enables enhancement of technological and medical features, such as reliable closed-loop control of active-locomotion capsules, accurate lesions localization, retargeting of pathologies, and follow-up.

Areas covered: This contribution provides an exhaustive and critical review of localization strategies, both internal and external, implemented so far for endoscopic capsules. Starting from basic theoretical principles, it describes the most promising methodologies, for example, magnetic and electromagnetic-based ones, but also other significant techniques, such as, ultrasound-based localization strategies.

Expert commentary: Authors believe that the integration of external and internal localization methodologies with a multimodality approach can increase the overall accuracy and reliability of the endoscopic device pose estimation both for establishing an optimal control during the endoluminal procedure within a deformable environment and for autonomously identifying position of internal pathologies for retargeting and follow-up.

KEYWORDS:

• Electromagnetic-waves
• robotic capsule endoscopy
• localization strategies
• magnetic-based localization
• radio-waves localization
• visible-waves localization

1. Introduction

Nowadays, gastrointestinal (GI) diseases, i.e. inflammatory bowel disease, peptic ulcer disease, and other sources of bleeding, as well as infections and cancers, remain threats for human health causing significant morbidity and mortality. In particular, cancers of the colorectum and the stomach represent the third and fifth most common cancer worldwide, respectively. In terms of number of deaths, gastric and colorectal cancer (CRC) are ranked third and fourth, respectively, with a mortality rate of around 76% for stomach and 51% for CRC. Esophageal cancer, despite lower number of cases and deaths, has a mortality rate of about 88%. In this regard, diagnosis at an early stage represents a key factor to reduce mortality. For CRC and gastric cancers, the 5-year survival rate in case of early stage pathology is >90%, falling to <20% in case of late diagnosis. The reported interval required for the progression of a precancerous lesion to an advanced neoplasm is around 5–10 years [1–3].

Nowadays, conventional endoscopes represent the gold standard for GI tract examination. Direct visualization of the GI tract is required to provide accurate, timely and reliable diagnosis [4]. However, due to (i) the rigidity of the endoscopes and their shaft dimensions (approximately up to 160 cm in length and up to 14 mm in diameter); and (ii) the possibility of both cross-contamination [5] and intestinal perforation (0.016% among all diagnostic procedures and up to 5% of therapeutic colonoscopies [6], for the latter), patients’ willingness to undergo endoscopy remains low due to fear of pain or discomfort, especially when the endoscopic procedure is performed by nonexperts. Furthermore, procedures performed with conventional endoscopes do not allow examination of the whole GI tract, as conventional scopes and deep enteroscopy are not able to cover the entire small bowel. In order to overcome these limitations, a disruptive endoscopic method, Wireless Capsule Endoscopy (WCE), was introduced by Given Imaging Ltd. (Yokneam Illit, Israel) and approved by the Food and Drug Administration (FDA) in 2001 for small intestine imaging, as its intended use. WCE is now a well-disseminated and promising solution for noninvasive inspection of the GI tract. WCE represents the sole solution so far for the diagnosis of the entire small bowel. In its original configuration, WCE consisted of a capsule-like device, comprising (i) a LED-based illuminating system; (ii) a compact CMOS or CCD camera for image acquisition; (iii) a coin-battery power supply module; and (iv) a radio frequency (RF) module with antenna used to send data out of the body to an external receiving module. Since patients have only to swallow the small pill and its propulsion is performed by natural peristaltic contractions of the intestine, this approach is painless and, consequently, widely accepted by patients and clinicians alike.

Nowadays, commercially available endoscopic technologies are not able to completely cope with the lack of effectiveness of mass screening campaigns for eradicating the CRC in a symptomless stage. On the other hand, the revolutionary approach of WCE technology, and its potential benefits for patients, makes it an important field of research and development for engineers and physicians. Research teams have proposed innovative methods, studies and additional modules to improve screening, diagnostic and therapeutic capabilities, such as: (i) high-efficiency wireless powering; (ii) controllable active locomotion; (iii) accurate localization, etc. [7]. Capsule localization plays a key role in WCE endoscopy, although the best capsule performance can be obtained through an optimum trade-off between all modules and functionalities. Indeed, accurate knowledge of the position and orientation of the capsule – defined as its pose – when it moves along the GI tract and captures images represents an invaluable information that can be exploited by physicians to better: (i) localize internal pathologies; (ii) perform follow-up diagnosis and intervention (such as for drug delivery); (iii) assist navigation of active locomotion WCEs; and (iv) perform power transmission in case of wireless battery charging.

Therefore, due to its importance, several solutions, ranging from the use of magnetic fields to ultrasounds and computer-vision technologies, have been investigated. A critical review on the state-of-the-art of recent localization strategies may represent a useful tool for researchers who are considering entering or further progressing in this challenging field.

To the best of our knowledge, at the date of the submission of this manuscript, two review papers, by Than et al. in [8] in 2012 and Mateen et al. in [9] in 2017, are the most relevant and complete contributions regarding localization strategies for endoscopic robotic capsules. Taking the previous reviews into consideration, we propose this review paper in order (i) to provide an updated view of the state-of-art of localization strategies, especially of the magnetic field-based and electromagnetic wave-based ones, since [8] has been published in 2012 and it is based on reviewing localization methodologies dating until 2011; and, in addition to [9], (ii) to highlight the improvements reached also by alternative localization strategies, such as computer vision-based localization methodologies and to report, for each localization technique, details such as: (a) methodology; (b) model simplifications and key-parameters; (c) hardware and/or software modules used for testing; and, finally, (d) position and/or orientation accuracy in order to provide an exhaustive and integrated analysis.

The review paper is organized into three main sections focused on: (i) magnetic fields-based localization strategies (Section 2); (ii) electromagnetic wave-based localization strategies (Section 3); and (iii) other types of localization strategies, such as ultrasound (Section 4).

2. Magnetic field-based localization strategies

Over the past decade, the use of magnetic fields for medical purposes has significantly captured the attention of numerous academic and industrial groups interested to localize, anchor, or navigate medical devices inside the human body [10,11]. These groups were motivated by the intrinsic advantages of magnetic fields, such as low attenuation through the human body [12] and capability of the magnetic-based sensor technologies to detect targets without the limitation of a line-of-sight, contrarily to visual-based sensor technologies (NDI Polaris Vicra and Spectra optical trackers). However, one of the most challenging problems is the possible interferences between the localization system and contiguous ferromagnetic modules, such as surgical tools, but also the locomotion module itself, in case the capsule’s active propulsion is obtained by high-intensity permanent or electromagnetic sources. For this reason, this section has been organized to detail magnetic localization methodologies in the condition with and without the use of high-intensity, magnetically driven actuation modules. Table A1, summarizing magnetic-based localization strategies, is provided in Appendix A.

2.1. Magnetic localization methodologies without magnetic-based actuation

The typical components of a magnetic tracking system are one or more magnetic sources (transmitters), and one or more sensor modules (receivers). Therefore, based on the relative position between transmitters and receivers, two main approaches are defined for robotic capsule localization. The first approach consists in positioning the magnetic sources inside the capsule and the sensing modules outside the patient’s body (Section 2.1.1), while the second approach consists in positioning the sensing module inside the capsule and the magnetic sources outside of the patient’s body (Section 2.1.2).

2.1.1. Magnetic source inside, sensing module outside the capsule

Initial investigations into localization methodology for capsule endoscopes considered the integration of an internal permanent magnet (IPM) inside the capsule, used as transmitter, and an array of sensors (e.g. Hall-effect, magneto-resistive, or multiturn coils sensors) outside, used as receivers, to detect the capsule pose. In 2005, Hu et al. [13] proposed a localization method based on the previously mentioned methodology (i.e. IPM as transmitter and an array of Hall-effect sensors as receiver), using the M2A capsule developed by Given Imaging Ltd. (Yokneam Illit, Israel). Starting by a magnetic dipole mathematical model [14], the authors implemented an algorithm to localize the capsule with five degrees-of-freedom (DoFs), that is, 3D position and 2D pitch/yaw orientations. Due to the nonlinear properties of the algorithm used to model the magnetic field, the authors first investigated the performance of different optimization techniques, identifying the Levenberg-Marquardt algorithm [15] as the most appropriate solution in terms of computational time. Then, the authors investigated the localization algorithm performance, varying the number of sensing devices outside the patient, in different configurations, in in-silico condition. The most accurate localization results were obtained by the authors with 16 external Hall-effect sensors, providing an average estimation error of 4.6 mm in position and 4.5% in orientation (orientation error is provided as the percentage relative to the unit orientation vector). Finally, the system was validated in in-vitro conditions and results showed an average estimation error of 5.6 mm in position and 4.2% in orientation (maximum error of 12.6 mm in position and 12% in orientation). On the other hand, the overall searching process required a computational time in the range of 0.1 ~ 1.2 s (average execution time of each sample is 0.137 s), which might represent often a limit for continuous fast tracking. In order to reduce the computational time, Hu et al. proposed in [16] a faster algorithm, obtained through a linearization of the algorithm previously presented in [13]. The localization method guaranteed the same performances in pose estimation, but with a lower computational time of 10.6 ms. It is worth mentioning that, as reported by the authors, exploiting this methodology it would be possible – keeping the overall processing time compatible with a real-time approach – to increase the number of sensors to improve the accuracy of the localization system. A similar study was presented by Wang et al. in [17], where a tri-axial magneto resistive-based sensors array was used instead of the Hall-effect based sensor technology. In real-time experiments, authors obtained an average estimated error of 3.3 mm (maximum error of 10 mm) and 3° (maximum error of 5.7°) for position and orientation, respectively.

Due to the nature of the localization algorithms, two kinds of optimizers are often used in literature to detect the capsule pose: a linear and a nonlinear optimizer. Linear optimizers are faster than nonlinear ones, but the localization algorithm accuracy is often lower because of the higher probability of facing with local minima. On the other hand, nonlinear optimizers increase the localization accuracy by reaching the global minima with a higher probability, but they are sensitive to the initial guess and, sometimes, might be slower than linear ones, especially when are based on gradient approaches.

In 2008, Hu et al. proposed in [18] an improvement of their localization methods, previously presented in [13] and [16], merging the benefits of linear and nonlinear optimizers. In particular, the algorithm was divided in two steps: (i) first, a coarse estimation of the capsule position and orientation was calculated using a linear optimizer; then, (ii) a refinement of the capsule position and orientation was computed based on the Levenberg-Marquardt nonlinear optimizer using, as initial guess, the output of the first implemented step. As result of the experimental tests, performed in in-vitro conditions, the accuracy and stability were significantly improved figuring out an average estimation error of 2 mm and 1.6° for position and orientation, respectively.

In 2009, Yang et al. proposed a method able to estimate the six DoFs of the capsule, that is, the 3D position and the 3D orientation, by using a rectangular IPM and a Particle Swarm Optimizer (PSO) [19]. Using a dipole-dipole magnetic model, the authors derived the mathematical model of the magnetic flux generated by a rectangular magnet. Then, by using an array of 16 magneto-resistive sensors, the capsule pose was obtained by solving the nonlinear relative objective function. Also, in this case, the Levenberg-Marquardt optimizer was confirmed to be the most proper algorithm in terms of computational cost and error tolerance. Simulation results showed an average execution time of 0.17 s in case of an average estimation error of 3.9 mm and 5.06°, while an average execution time of 0.63 s in case of an average estimation error of 0.59 mm and 0.66° in position and orientation, respectively.

As mentioned in [13] and [16], the maximum operative distance between the internal magnet and the external array of sensors cannot be higher than 120 mm. This common limitation in the operative range is not always compatible with the human anatomy, where in particular cases eventual application of this technology would require to cover distances as high as 280 mm (side-to-side) [20]. In order to increase the localization operating range, that is, the maximum distance from receivers at which the capsule can be detected, Hu et al. in [21] proposed a real-time tracking system in which the magnetic field of the IPM is measured by using a cube-shape magnetic sensors array. In particular, as shown in Figure 1(a), the system is composed of 64 tri-axial magnetic sensors (HMC1043 sensor by Honeywell – Morris Plains, New Jersey, USA) distributed over four surfaces, that is, 16 sensors per surface, with a cubic inner space of 0.5 × 0.5 × 0.5 m³. Furthermore, exploiting the idea of using a rectangular-shape magnet, as proposed in [21], authors demonstrated to obtain an average estimation error of 1.82 mm and 1.62° in position and orientation, respectively, up to a distance between capsule and sensor of 250 mm; exploiting this approach, six DoFs can be detected in an average computational time of 0.1 s.

Figure 1. Examples of magnetic-based localization systems. (a) Scheme and cubic magnetic array of sensors presented by Hu et al. in [21]. (b) Wearable magnetic localization array of sensors presented by Hu et al. in [23]. (c) Multicoils electromagnetic tracking system presented by Plotkin et al. in [25]. (d) Electromagnetic locomotion and sensors array localization system proposed by Turan et al. in [37]. (e) Application scenario of active magnetic manipulation of a capsule endoscope using a permanent magnet mounted at the end-effector of a robot manipulator presented by Taddese et al. in [38]. (f) Modelling principle used by Popek et al. in [45] that use a magnetic field magnitude generated by a rotating permanent magnetic dipole.

In 2014, Song et al. proposed in [22] a variation of the aforementioned contribution by implementing the same localization methodology but, instead of using a cylindrical or a rectangular magnet incorporated in the capsule, with axial magnetization direction, authors used an annular magnet with a radial magnetization direction. Authors derived a mathematical model, based on the Biot-Savart law, for calculating the magnetic flux and exploited the superposition principle, that is, considered the magnetic flux of an annular magnet as the difference of two cylindrical magnets with different radius and opposite magnetization directions. The results of the simulation showed an average position error of 0.003 mm and an average orientation error of 0.036° using a PSO-based algorithm (computational time of 830 s).

Finally, Hu et al. proposed in 2016 a wearable magnetic sensors array [23], starting by the study proposed in 2005 [13] and refined during the succeeding years. A ring-shaped permanent magnet, embedded onto the capsule, represented the target, while other two permanent magnets, attached to the human body surface, were used as external reference systems (Figure 1(b)). This configuration, composed of different distributed magnetic sources, was implemented to reduce the interferences caused by the motion of the human body during the tracking procedure. The magnetic fields of the three permanent magnets were acquired by a sensors array composed of 32 tri-axial magnetic sensors. By using a PSO-based algorithm, the position and orientation of each magnet was obtained, and the capsule pose was estimated and tracked with respect to these two external reference sources. The authors compared the results obtained using two different in-vitro configurations, that is, with and without motion compensation (i.e. motion of the body surface with respect to the wearable magnetic sensors array), showing the reduction in the average position and orientation errors from 30.1 mm and 17.7°, without motion compensation, to 3.82 mm and 2.2°, with motion compensation.

2.1.2. Magnetic source outside, sensing module inside the capsule

With respect to the second localization approach, that is, sensing module inside the capsule and magnetic sources outside of the patient’s body, Plotkin et al. proposed in 2003 [24,25] a localization method using a single sensing element inside the capsule to measure the magnetic fields generated by different external coils. As illustrated in Figure 1(c), the tracking system was composed of a single sub-miniature circular induction coil sensor (0.9 mm in diameter, 3 mm in height, and composed of 1248 turns of 28 micron copper wire) and a 8 × 8 array of circular coplanar transmitting coils (26 mm outer diameter, 18 mm inner diameter, 20 mm in height, and 500 turns of 0.4 mm Litz wire). An alternating current of 1 A amplitude and 50 kHz frequency was used to power the transmitting coils. Experimental results have showed an accuracy of 0.75 mm in position and 0.6° in orientation, within a maximum operative range of 200 mm in height above the transmitting array. A similar design approach is used by a commercially available electromagnetic tracking system (Aurora, NDI – Northern Digital Inc., Ontario, Canada).

In 2004, Nagaoka and Uchiyama proposed in [26] a localization method for WCEs embedding a single-axis coil, used as magnetic sensor (6.5 mm in diameter, 2.3 mm in height, and 160 turns copper wire), and five alternating external magnetic field sources. Each magnetic field source was powered at different frequencies and was fixed externally to the patient’s body. The authors demonstrated the capability of the system to localize the capsule with an error of 2.8 ± 2.2 mm and 13.4 ± 20.9°, within distances up to 400 mm from the transmitters. However, according to the relationship between induced voltage magnetic flux and number of turns of a sensing coil (Faraday’s law), cylinder-shape coil could result in configurations with considerable bulky volumes.

In order to obtain an optimal trade-off between induced voltage, required for localization purposes, and capsule dimension, Islam and Fleming proposed in 2014 [27] an innovative configuration of the sensing coils, composed of three orthogonal rectangular coils, made of a 0.08 mm diameter copper wire, enclosed into a wireless capsule of 26 mm in length and 11 mm in diameter (i.e. same dimension of the GI PillCam SB capsule). Two coils have a surface of 20 × 7 mm² and the third one has a surface of 7 × 7 mm²; in total, the sensing coils occupied a volume of 48.88 mm³, representing the 1.92% of the whole capsule volume. Three external perpendicular circular coils, sequentially powered, were used as magnetic field source generators. Compared to [24], this configuration permits to obtain a six DoFs localization of the endoscopic capsule with an estimated error of 6 ± 0.28 mm in position and 1.11 ± 0.04° in orientations.

In summary, the most notable solution of magnetic localization strategy suitable with not magnetically driven WCE with sensing module outside the capsule has been developed and improved since 2005 by the Chinese University of Hong Kong. The latest configuration of the system integrates an internal permanent magnet into the WCE, and a wearable array of Hall-effector sensors placed outside the patient. The 6-DoFs localization strategy demonstrated an average error under 5 mm and 3° for position and orientation, respectively. It should be noted that in all the analyzed works, it has been shown that the accuracy of the localization system increases with the number of external sensors [23].

On the other hand, in case the receiver module is placed inside the capsule, the main solution for magnetic-based localization without magnetic-driven actuation is to use one or more coils inside the capsule and several alternating-current electromagnets placed outside the patient body. This solution, developed by several authors, generally provides a position and orientation average error under 1 mm and 1°, respectively.

2.2. Magnetic localization methodologies with magnetic-based actuation

As previously mentioned, several studies were performed for ensuring both magnetic-based locomotion [28] and localization of a CE, generally by exploiting the magnetic interaction between two or more magnetic sources placed outside the patient’s body and one or more placed inside the endoscopic capsule. Obviously, the use of magnetic sources for both locomotion and localization purposes may result in undesired interferences with the localization system, since it could be unable to distinguish the magnetic source used for locomotion purposes or being affected by the strong generated magnetic field, leading to sensors saturation.

2.2.1. Static magnetic field-based actuation

In 2009, Ciuti et al. proposed the first approach to perform minimally invasive diagnosis of the GI tract using a magnetically driven capsule robot [29]. The system consisted of a capsule device provided with a set of IPMs and a MEMS tri-axial accelerometer. Capsule locomotion was performed by exploiting the magnetic force interaction between the IPMs and an external permanent magnet (EPM) moved by a six DoFs, anthropomorphic, teleoperated robotic arm. Through a preprogrammed scanning procedure, the robotic arm moves the magnetic end-effector above the patient’s body until the capsule was magnetically attracted. The establishment of the capsule IPMs-EPM magnetic link is detected through recording the accelerometer module impulse signal. These inertial signals were also used for providing a rough 2D position estimation, that is, x and y coordinates with respect to the robotic reference system, with an average spatial resolution error of approximately 30 mm. Furthermore, the same accelerometer was used for providing additional orientation information, that is, pitch and roll angles of the capsule, with an accuracy of 6°. This robotic system was tested in ex-vivo conditions using a porcine colon segment. Even if the system did not directly employ a magnetic-based localization approach, it was one of the first studies describing a magnetically driven capsule robot. The same authors, in an additional study [30], integrated in a magnetically driven tethered capsule three mono-axial Hall-effect magnetic sensors with an orthogonal configuration (CY-P15A, Chen Yang Technologies GmbH & Co.KG, Finsing, Germany). These were used for assisting locomotion through a simple analysis and processing of the magnetic field module.

In 2011, Salerno et al. in [31] proposed an innovative methodology using the same magnetic-based sensor technology, that is, three mono-axial magnetic sensors embedded onto the capsule in orthogonal configuration. With the aim of using the EPM magnetic flux for capsule 3D position estimation, the authors proposed an IPMs configuration such as to avoid the Hall-effect sensors saturation, and to guarantee a good magnetic force link with the EPM. The localization method was based on a triangulation principle using a single source (EPM) moved above the patient abdomen on six 3D predefined positions; in this case, during the localization procedure, capsule locomotion is stopped and the EPM is moved at a low EPM-IPM interaction distance. System performance was tested in in-vitro condition with a colon simulator, by performing measurements on 75 points distributed over three different parallel planes (25 point per plane – 150×150×200 mm³) in order to highlight the performance’s dependency on the distance. The mean and standard deviation of the error between estimated and real positions were −3.2 ± 18 mm along X-axis, 5.4 ± 15 mm along Y-axis and −13 ± 19 mm along Z-axis. However, this strategy was not designed for providing information about the capsule orientations. An on-line five DoFs localization system (20 Hz data rate) was also developed by Salerno et al. in [32] using a 3D Hall-effect sensor and a 3D MEMS accelerometer embedded onto the capsule and deriving the capsule position, pitch and roll through pre-calculated magnetic field maps describing the external-source magnetic field. A position error lower than 10 mm was obtained when the capsule was at 120 mm from the external permanent magnet. However, it is recognized from literature that the distance between the abdominal external surface and the colonic upper wall ranges between 36.0 ± 18.0 mm (transverse segment) and 154.1 ± 17.6 mm (sigmoid segment) [20].

In 2013, Di Natali et al., proposed a novel approach to detect the real-time six DoFs pose of a magnetic endoscopic capsule [33]. Authors investigated a similar approach as the one proposed by Ciuti et al. in [29] for the magnetically driven locomotion, but integrating 6 Hall-effect sensors and a tri-axial MEMS accelerometer, embedded onto the capsule, for pose estimation. The six Hall-effect mono-axial sensors were placed around a cylinder-shape axially magnetized IPM, one perpendicular to each other along the three main axes. Pitch and yaw angles were estimated through the accelerometer processed data, whereas the capsule 3D positions and roll angle were calculated thanks to an iterative procedure. First, the localization system was calibrated by positioning the capsule in a known orientation in order to obtain the initial roll angle as baseline. Then, the iterative procedure followed four steps: (i) acquisition of the magnetic field and pitch/yaw angles from the Hall-effect sensors and accelerometer, respectively; (ii) comparison of the measured magnetic field with the value of a pre-calculated numerical map obtained recording, with the endoscopic capsule, the external magnetic field moving the EPM in the 3D space; (iii) selection of the closest value deriving the capsule pose vector; and (iv) estimation of the roll angle as the angle between X-axis of the external magnet and the projection of capsule distance vector in the horizontal plane. System performance was evaluated through in-vitro experimental tests. The setup consisted of (i) a capsule prototype, placed on a gimbal and equipped with an IPM, six Hall-effect sensors and a tri-axial MEMS accelerometer; (ii) an EPM moved at different distances from the capsule through a robotic arm; and (iii) an optical tracker for visual feedback. By placing the capsule within 150 mm from the EPM, the mean error and the standard deviation were −3.4 ± 3.2 mm along axis X, −3.8 ± 6.2 mm along axis Y, −3.4 ± 7.3 mm along axis Z, and −19 ± 50° for roll angle, while 6 ± 18° and 3 ± 20° for pitch and yaw angles, respectively. As regards computational time, a single instance of the algorithm takes 14 ms, while the search requires an average of 5 ms. In 2016, authors improved this iterative localization method by adopting a Jacobian-based approach [34]. In particular, by defining the closed-form expression for the Jacobian of the EPM magnetic field, relative to the changes in the position of the capsule, the authors were able to obtain an iterative localization at a faster computational time than the previous one presented in [33]. In particular, a searching cycles of 7 ms was obtained, which is compatible with the closed-loop navigation of an endoscopic medical capsule (travel speed around 5mm/s) [35]. The average localization error, expressed in cylindrical coordinates, was 6.2 ± 4.4 mm and 6.9 ± 3.9 mm in the radial and axial components, respectively, and 5 ± 7.9° in the azimuthal component, while orientation error is 0.27 ± 0.17° for pitch, 0.34 ± 0.18° for yaw and 1.8 ± 1.1° for roll. However, the localization algorithms presented in [33] and [34] were based on two main assumptions: (i) a unique correlation between positions in the workspace and magnetic field vectors for each EPM pose, and vice versa; and that (ii) changes in acquired magnetic fields always occur for changes in capsule pose.

In 2017, Son et al. [36] developed a five DoFs localization system for untethered magnetic robots manipulated by an electromagnetic system. The localization module, placed above and below the workspace, is composed by a 2D array (8×8) of mono-axial Hall-effect sensors (placed above) and an omnidirectional electromagnet made of three box-shaped orthogonal coils and a soft iron core (placed below). First, the magnetic field contribution of the electromagnet, modelled according to the dipole assumption, was subtracted by the measured data. Then, the obtained magnetic field value was decomposed along the perpendicular direction of the array in order to minimize the electromagnetic field contribution. Finally, the position and the orientation of the magnetic robot were calculated minimizing the error between the measured magnetic field and the modelled magnetic field. The resulting position error was 2.1 ± 0.8 mm and orientation error was 6.7 ± 4.3°, within the applicable range of 70×70×50 mm³ at 200 Hz. Similar technology was applied by Turan et al. in 2017 [37] to drive a robotic endoscopic capsule, as shown in Figure 1(d). Authors integrated the information deriving from the localization system described in [36] with the information deriving from the camera of the endoscopic capsule.

In 2018, Taddese et al. [38] highlighted that workspace regions exist where the previously mentioned assumptions fail due to magnetic field singularities. Since magnetically driven endoscopic capsule applications often require capsule to be located in such regions of singularity, that is, the plane that passes through the EPM center and normal to its dipole magnetic moment, authors proposed and validated a solution to solve this issue. The approach consisted in attaching an electromagnetic source to the EPM, which generates a weak time-varying magnetic field, with a dipole magnetic moment orthogonal to the EPM magnetic moment (Figure 1(e)). The experimental validation has been performed through predefined in-vitro trajectories and designed to imitate the general form of the human colon anatomy. The results showed the strength of this localization approach even in the presence of singular regions. Capsule pose estimation error from static tests along a spiral trajectory was less than 5 mm and 6° for position and orientation, respectively.

2.2.2. Alternating magnetic field-based actuation

Olympus Group (Olympus Corporation Shinjuku, Tokyo, Japan) in [39] and Siemens AG (Siemens AG – Berlin and Munich, Germany) in [40] investigated for the first time the use of alternating magnetic fields for both locomotion and localization purposes. Olympus Group proposed a spiral-structured wireless endoscopic capsule with an IPM embedded. The actuation system consisted of three pairs of coils, placed around the patient’s body along three axial directions and orthogonal each other. Coils were powered in order to generate a rotating external magnetic field that, through the interaction with the capsule IPM, and thanks to the spiral structure itself, allowed the propulsion of the capsule both in forward and backward directions. To avoid fast movements of the capsule inside the GI tracts, the rotation frequency of the external magnetic field should not exceed 10 Hz. In addition, alternating magnetic fields in the range of 1 ÷ 10³ kHz, were employed for capsule localization purpose. An array of exciting coils, which generates such alternating magnetic fields, were placed around the patient body. Thanks to the mutual induction phenomena, a resonant circuit, embedded into the capsule, generates a correspondent magnetic field. The total magnetic field, which in this case was the combination of the excitation and resonant field, was measured through an auxiliary array of detecting coils, also placed around the patient body. The localization of the capsule was performed by exploiting the dependence of the resonant magnetic fields on the capsule position and orientation.

The approach proposed by Siemens AG was like the one proposed by the Olympus Group. The main difference was related to the capsule pose estimation methodology. In this case, instead of using a resonant circuit, the authors proposed to measure the external alternating magnetic fields through a detection/receiver coil embedded onto the capsule. Then, the acquired data was sent out from the patient’s body through an RF module and processed by a local workstation in order to estimate the position and orientation of the capsule itself.

A similar study, performed by Hashi et al. [41,42], consisted in a system composed by (i) an exciting coil; (ii) an LC marker enclosed into the capsule; and (iii) a pick-up coil array, consisting of 25 pick-up coils placed into a 5 × 5 matrix. The marker consists of a Ni-Zn ferrite core with a wound coil and a chip capacitor, representing an LC circuit designed to operate at a predefined frequency. A distance of 200 mm between exciting coil and pick-up coil array exists. Induced voltage detected at the pick-up coils includes both the induction of the exciting field and the marker field. The marker contribution is obtained by subtracting the induced voltage with marker from the induced voltage without marker, previously obtained. The six DoFs of the marker is calculated by solving an inverse problem, starting from the values of the flux density measured on each pick-up coil. Experimental tests demonstrated that with this methodology, it is possible to achieve a detection accuracy of the submillimeter order with a maximum operative range distance of 120 mm from the pick-up coil array.

An alternative way to generate a variable magnetic field consists in applying an angular rotation to an EPM (rotating permanent magnet – RPM) with the aim of propelling a spiral-shape capsule through its rotation, as shown in Figure 1(f). Intensive studies on how exploiting this approach for endoscopic capsule actuation and pose estimation were conducted by J.J. Abbott and colleagues between 2013 and 2018 [43–47]. Authors designed a WCE with a helical-shape structure and internally embedding: (i) an IPM for locomotion; (ii) six Hall-effect sensors for localization; and (iii) an RF module for data communication. The teleoperated control of the capsule was guaranteed through a six DoFs robotic arm (Yaskawa, Motoman MH5, Japan) with an appropriately designed end-effector (seven DoFs) equipped with a Maxon 24V DC-motor and a NdFeB grade-N42 cylindrical diametrically magnetized EPM (25.4 mm in diameter and 25.4 mm in length). Authors imposed an angular rotation of the EPM to generate a variable rotating magnetic field, exploited for the propulsion of the WCE. First, the authors derived in [43,44] a mathematical model, based on the magnetic dipole assumption, to describe how the rotating magnetic field is distributed over the operative workspace and how this magnetic distribution can be properly exploited to obtain the endoscopic capsule propulsion. As reported by the authors, it is fundamental to know the capsule pose in order to obtain desired and safe capsule propulsion. To solve this issue, authors then proposed and validated in [45,47] a novel approach to estimate position and orientation of a capsule actuated through an RPM. For this scope, the previous developed mathematical model was improved in order to derive the magnetic field magnitude perceived by the capsule as function of the relative angle θ. between the RPM and the capsule rotation axis. Once this dependence was derived, authors elaborated an algorithm to estimate both capsule position and orientation. Inputs for the algorithm were the magnetic flux densities measured by the three pairs of tri-axial Hall-effect sensors. Experimental results showed an average error of 4.9 ± 2.7 mm and 3.3 ± 1.7° for position and orientation, respectively. However, the main disadvantage of this approach was the fact that propulsion and localization could not be carried out at the same time. In 2017, an improvement of their previous studies was presented in [46], proposing a real-time propulsion and localization strategy in order to localize the capsule without stopping the propulsive locomotion procedure with an average speed of 2.2 mm/s. The improvement was provided through the introduction of an extended Kalman filter and by the assumption that the capsule movement is restricted to translation and rotation along its principle longitudinal axis. Average position and orientation errors were 8.5 mm and 7.1°, respectively.

It is worth mentioning that all the localization strategies presented in this section (i.e. Section 2.2) embed the sensing module inside the endoscopic capsule (i.e. second localization approach of Section 2.1).

In summary, the most notable localization solution for active-locomotion capsules in case of static magnetic field-based actuation was developed by the University of Leeds. The latest configuration of the system presents a hybrid solution that integrated an EPM and an electromagnet, held by the end effector of an anthropomorphic robot, to guide and localize in real time an active soft-tethered capsule. It was able to perform a 6-DoFs localization with an average error in position and orientation under 5 mm and 6°, respectively [38].

On the other hand, in case of alternating magnetic field-based actuation, the most remarkable solution was developed since 2013 by University of Utah. The main difference with Taddese et al. [38] is that the spiral-shape capsule is propelled applying an angular rotation to an RPM. Similar results were reported both for position and orientation. It should be noted that in both cases Hall-effector and/or inertial sensors were embedded inside the capsule as sensing units.

3. Electromagnetic wave-based localization strategies

Over the last century, electromagnetic (EM) waves represented a powerful tool for a wide set of applications, ranging from telecommunication to medical engineering. An important application that still receives a lot of interest from researchers and engineers is the use of EM waves for localizing a target both in outdoor and indoor environments. One of the most popular technologies is the global positioning system (GPS) [48]. GPS technology was designed to detect critical targets in outdoor military applications and subsequently was used for assisting navigation systems both in civil and military fields with different levels of precision, reaching today accuracies in the order to a few centimeters [49]. In addition, several studies were performed in the medical field on how using the electromagnetic spectrum to localize medical targets, such as needles and endoscopic capsules, introduced into the human body.

According to the conventional representation of the whole EM spectrum, we can identify: (i) radio waves (up to 3 GHz); (ii) microwaves (3 GHz–300 GHz; (iii) infrared (300 GHz–478 THz); (iv) visible waves (430–478 THz); and (v) X-rays and gamma rays (over 478 THz). Each of them is used for different purposes due to their propagation and attenuation range inside a specific environment. In particular, due to the propagation and attenuation properties of the human body, only radio waves, visible waves, X-rays, and gamma rays can be exploited for localization in medical procedures.

However, to the best of our knowledge, current state of the art on endoscopic tracking systems are mainly focused on using RF and visible wave spectrum. This can be explained by two fundamental factors: (i) RF and visible waves are safer than X-rays or gamma rays (i.e. ionizing radiation); and for (ii) complexity/cost factors for the implementation of X-rays and gamma rays transmission systems.

The following subsections provide an analysis of these two localization approaches, that is, radio (Section 3.1) and visible (Section 3.2) wave-based localization methodologies. It is worth mentioning that if not explicitly indicated, radio waves and radio frequencies represents interchangeably terms. A summary table, describing EM wave-based localization methodologies is provided in Appendix B (Table A2).

3.1. Radio wave-based localization methodologies

During the last decade, substantial improvements were observed regarding miniaturization processes and costs reduction of semiconductor devices. These improvements made possible the design and development of small and low-cost wireless communication devices capable of supporting the development of wireless sensor networks [50]. In the medical domain, this allowed the development of WCEs used both for localized physiological parameters analysis inside the human body and for visual-based diagnosis of the GI tract, primarily the small bowel. Several WCEs were developed and commercialized, such as: (i) PillCam®, originally called M2A and produced by Given Image Ltd. (Yokneam Illit, Israel), now released by Medtronic Inc. (Minneapolis, MN, USA); (ii) EndoCapsule® from Olympus Group; (iii) OMOM® capsule, developed by Jinshan Science & Technology Group Co., Ltd (Chongqing, China); and (iv) MiroCam® (IntroMedic Co Ltd., Guro-Gu, Seoul, South Korea). All these capsules are equipped with camera modules in order to capture images of the GI tract and, usually, they embed an RF module to wirelessly transfer images and sensors data outside the human body. RF modules are also commonly used to localize the position of the capsule; due to the intrinsic features of the RF signal, the precision is in the order of centimeters. As an example, not exhaustive but significant in the scenario of commercially available WCEs, Given Imaging Inc. integrates a RF-based triangulation method for localization of PillCam® systems. Eight sensors, placed on the human body abdominal surface, receive the strength of the RF signals directly emitted by the capsule. Average position error is 37.7 mm and maximum position error is higher than 100 mm [51].

A possible way to improve localization accuracy would be to exploit higher frequencies (e.g. in microwave domain) in order to use wavelengths comparable to the capsule target dimension; however, in that case, the signals attenuation through the human body would be too much high to guarantee a good signal-to-noise ratio (SNR). For these reasons, an inevitable trade-off between precision and attenuation factor should be taken in consideration. Therefore, several studies were performed to investigate how exploiting RF signals, transmitted by endoscopic capsules, in order to perform accurate localization.

3.1.1. Modelling propagation of RF through human body

One of the most relevant aspects that strongly influences the performances of a capsule RF-based localization method is the signal propagation loss model used, since tissue absorption varies from person to person. The human body can exhibit high power absorption, central frequency shift, and radiation pattern disruption, depending on the operation frequency. The International Telecommunication Union (ITU) has reserved the 402–405 MHz bands for medical implant communication. However, since operation in this band requires a license, another frequency band, near the 400 MHz and unlicensed for conducting studies and performance analyses of RF-based tracking systems, is the 433 MHz Industrial, that is, the Medical and Scientific (IMS) band.

Initially, researchers proposed to overcome the dependency on operating frequency by using a precalculated look-up table obtained by simulation analysis. For example, Shah et al. in [52] proposed an RF-based tracking system based on using look-up tables. These tables were populated with data obtained through off-line simulations performed using different phantoms made to emulate the conductivity of different organs into the IMS band. Using these precalculated tables, authors were able to estimate the capsule transition through different parts of GI tract with an accuracy comparable to that of the look-up tables.

Wang et al. in [53], starting from the empirical propagation loss model developed by the National Institute of Standard Technology (NIST) [54], proposed a novel Finite-Difference Time-Domain (FDTD)-based propagation model, which included signal attenuation dependencies due to the relative position and orientation between the transmitting capsule module and the receiving array of antennas. Simulations of the novel FDTD model were performed by SEMCAD (Schmid & Partner Engineering AG, Zurich, Switzerland) [55]. Comparing both NIST model and FDTD model with real measurements, authors showed how the Root-Mean-Square-Error (RMSE) decreases from 40 dB to 20 dB, respectively.

Since the FDTD analysis resulted to be very useful to simulate the RF signal propagation inside the human body, Makarov et al. in 2011 [56] implemented in MATLAB (MathWorks, Natick, Massachusetts, USA) an FDTD algorithm, which was faster and equally accurate than the commercially available solvers, for example, 100 times faster than Ansoft HFSS (ANSYS, Canonsburg, Pennsylvania, USA).

3.1.2. Radio frequency-based methodologies

A completely different approach for the design of a propagation model, was presented by Chandra et al. in [57]. The authors proposed to obtain approximated electrical proprieties of the tissues by first imaging the human body at 403.5 MHz frequency. Then, these a priori information were used as input for the position estimator implemented through a nonlinear least square algorithm. Through this localization method, validated in in-vitro conditions using phantoms with different shapes and dielectric proprieties, authors obtained a mean estimation error in the order of 9 mm using a realistic heterogeneous phantom (mimicking anatomical and electrical properties of the human body) to 3 mm for circular simple heterogeneous phantom (including two circular geometries with constant electrical properties along the main axis of the phantom). However, these performances cannot be guaranteed for distances higher than 50 mm between transmitter and receiver.

Li et al. presented in [58] a three-dimensional maximum likelihood algorithm as localization method, based on the received RF signal strength indication (RSSI), that is, the measurement of the power present in a received radio signal. Authors considered a scenario in which a WCE, equipped with a transmitting RF module, was moved through the small intestine in a volume of 400×200×350mm³ in length, width, and height, respectively. Authors also considered four receiving antennas placed above the patient’s abdomen and dedicated to the RF signal detection. Signal attenuation was estimated and considered in the localization methodology by using the NIST model. According to the Cramer Rao Lower Bound (CRLB) [59] of this estimator, capsule position error ranges between 80 mm and 110 mm in 95% of cases. However, using the signal attenuation model, the authors obtained a capsule position estimation error which ranges between 25 mm and 140 mm in 95% of cases, with an average error of 80 mm. Authors concluded that large distribution of the capsule position accuracy, obtained through three-dimensional maximum likelihood algorithm, is due to the high dependence of the WCE to the receiving antennas distance.

Ye et al. presented in [60] significant improvements on RF-based localization methods. In particular, the authors investigated the accuracy of the RSSI-based methods using a human-body 3D model, which includes frequency-dependent dielectric proprieties of more than 300 parts of the human body. As regards the signal propagation model, starting from the model developed by NIST, authors proposed a propagation model which considers these dielectric proprieties. Starting from these assumptions, the authors derived the CRLB variance limits for the 3D RSSI-based capsule localization method, through several simulations performed by varying the number of sensors and their positions [61]. From these studies, authors found that at least 32 detecting sensors arranged above the patient body surface with a three-dimensional range of 266×323×312 mm³ are necessary for obtaining an estimation RMSE of 45–50 mm.

Another type of localization method, based on electromagnetic field strength measurements, is the Radio Frequency Identification (RFID) technique. The most common RFID localization setup is illustrated in Figure 2(a). In this configuration, there are three main modules: (i) an RFID reader with 3D array of antennas surrounding the human body; (ii) an RFID seed attached to a medical instrument, for example, needles, endoscopic capsule, and catheter; and (iii) a data-processing unit to estimate the target position. Each antenna, during predefined temporal slices, sends an electromagnetic wave, which acts as a polling request for the tag. If the RFID tag is activated by this signal, it generates a feedback, which could be received from all the nearest antennas.

Figure 2. Examples of some electromagnetic waves-based localization systems. (a) Design of an RFID localization system composed by reader, tag and computer proposed by Zhang et al. in [63]. (b) Design of a ToA localization system composed by the capsule and the array of sensors, mounted on body surface presented by Pourhomayoun et al. in [68]. (c) Circular arrays and inertial measurement unit for DoA/ToA/TDoA-based endoscopy capsule localization presented by Nafchi et al. in [69]. (d) Overview of the cyber physical system for localization and distance travelled inside the small intestine, presented by Pahlavan et al. in [71]. (e) Illustration of a FCN-VGG for polyp detection presented by Brandao et al. in [75]. (f) Visual geometric odometry of wireless capsule endoscopes aided by artificial neural networks presented by Dimas et al. in [78].

In 2009 Hou et al. proposed in [62] a novel localization method and system architecture for in-vivo capsule endoscopy using Ultra-High Frequency (UHF) band (915 MHz). Through this method, based on the theory that the tag feedback signal is detected by the closest antenna, authors obtained a localization accuracy of 20 mm, without needing a signal attenuation model for the human body. Zhang et al. presented a similar analysis in [63] showing that the performance of the same system presented in [62] did not change varying the antenna radiation pattern. Another important study was conducted by Hekimian et al. [64], in which authors shown how to exploit phase differences between two or more receiving antennas to compute an accurate localization (i.e. up to 1.8 mm). However, this work was focused for indoor localization and not for medical purposes. Starting from these results, Wille et al. in [65] presented an experimental medical navigation system based on RFID technologies. By applying the Support Vector Regression (SVR) algorithm to phase difference data gathered from multiple RFID receivers, the authors observed a strong difference between the result on a 5 mm and a 10 mm calibration grid, obtaining a mean accuracy of 1.6 ± 1.2mm and 7.8 ± 5.2mm, respectively. Results demonstrate the potentiality to apply this method in capsule position estimation for medical applications, such as localized drug delivery and distance travelled calculation.

Time of Arrival (ToA) and Direction of Arrival (DoA)-based localization methods were also investigated. The first one is based on measured the time which a radio signal employs in travelling from the transmitter to the receiver, as distance estimation. The second, employs a set of receiving antenna array for estimating the direction from which the propagation waves arrive, respect a given direction. Although these methods might be more accurate than RSSI and RFID-based methods, however, they required additional, costly and unwieldy hardware, for example, synchronization system between transmitter and receiver and particular array of antennas patterns. In addition, it is also required an accurate knowledge of the propagation proprieties of the EM waves in passing through human tissues. Khan et al. presented a comparison between a RSSI-based and a ToA-based method [66]. They used the FDTD algorithm previously developed in [56] for simulating the RF signal propagation loss model through human body and the CRLB variance of RMSE of each estimator as performance evaluation parameters. Results obtained through simulations shown that ToA-based estimator outperforms better than RSSI-based estimator. Indeed, estimation errors obtained from the ToA-based method were in the millimeter range, while the ones obtained from RSSI-based were in the range of centimeters. Liu et al. [67] waveform propagation behavior in homogeneous and nonhomogeneous tissues at 6 GHz was simulated through the SEMCAD X platform; a comparison with empirical measurements shown the good reliability of this simulation tool. Similar studies at 400 MHz were conducted in [60,61]. In these works, the authors have also shown how it was possible to obtain estimation errors lower than 15 mm through a ToA-based approach. In [68] localization in a 3D space, based on spatial diversity, was proposed by Pourhomayoun et al. By exploiting the spatial diversity of the emitter, that is, the biomedical implant, authors estimated the weighted average of the signal propagation velocity, as well as the path loss model parameters for each transmitter-receiver pair (Figure 2(b)). Through a validation of this scenario, conducted through simulations, an estimation error lower than 8.8 mm was obtained. In 2014, Nafchi et al. [69] proposed a localization method based on the usage of an extended Kalman filter and a circular array of antennas (Figure 2(c)). First, they showed how this system could be exploited for implementing both a ToA or a DoA-based localization. Then, through an in-vitro experimental validation, they verified the strength of this method by obtaining estimation errors lower than 20 mm. In addition, they showed how the usage of an Inertial Measurement Unit (IMU) can further improve this method by obtaining position errors lower than 10 mm and 0.5 cm/s velocity error using a 16 antennas array. A similar study on DoA-based localization was conducted by Goh et al. in [70] by using an unscented Kalman filter. The DoA measurement was performed via array of antennas placed within a medical ward and an IMU was embedded into the WCE capsule. Additional beacons were attached to the patient in order to further improve the system performance. As in [69], through this method, an estimation error less than 10 mm is obtained. Although it has been proved that DoA and ToA-based algorithms provide higher accuracy than the RSSI and RFID-based localization systems, these techniques still require improvements in order to result less sensitive to the channel variations.

In summary, the RF-based localization methodology represents a very interest solution for WCE because it uses hardware technology already embedded into the WCE (i.e. the image transmission module). Acquisition and processing of the transmitted signal are managed outside the patient by applying solutions with different complexities. Two general approaches can be generally identified: the first is based on the strength of the signal, that is, the RSSI and the RFID, and the second one takes into consideration more complex signal features, that is, the ToA and the DoA. The first approach is simpler and less expensive than the second one because of lower and less unwieldy equipment requirements. However, the second approach can guarantee average error position in a range of 10–15 mm, while the first one in the range of 20–50 mm.

3.2. Visible wave-based localization methodologies

Visible waves are in the range of the electromagnetic spectrum containing wavelengths from about 390 up to 790 nm. As stated before, since wavelength and resolution are inversely proportional, that is, higher is the wavelength worst is the resolution, localization methods which exploit optical spectrum, could reach better performances than RF-based methodologies. However, unlike the RF-based localization methods, due to their poor penetration through the human tissues, these wavelengths are used for the aim of capsule localization through computer vision techniques. As illustrated in Figure 2(d) by Pahlavan et al. in [71], generally a WCE is equipped with: (i) light emitting diodes, which act as illumination source; and (ii) a miniaturized camera for capturing endoscopic images during its travel along the GI tract. Unlike the previously mentioned localization methodologies, in which the capsule pose was identified with respect to an external reference frame (i.e. external localization), for example, antennas and magnetic sources, images can be used to localize the capsule (and also lesions and pathologies) with respect to the surrounding deformable anatomical environment and targets (i.e. internal localization), for example, landmarks and luminal 3D geometry.

Initially, image analysis for GI pathologies identification and capsule pose detection was entirely performed by physicians. In order to: (i) automatize the analysis; (ii) reduce the processing time; and (iii) perform a pathology detection not dependent by physician’s skills (currently of interest for commercial applications), several image classifiers based on Artificial Neural Network (ANN), Vector Quantization (VQ), Support Vector Machines (SVM) and so on, were investigated. In [72] Duda et al. conducted a performance analysis, comparing ANN, VQ and VQ in addiction to Principal Component Analysis (PCA) classification algorithms. In this study, the authors used the homogeneous texture of the MPEG-7 coded multimedia as content descriptor [73]. By analyzing images obtained from a WCE diagnosis of the GI tract (from esophagus to duodenal cap), authors showed how ANN-based classifier better outperforms than VQ-based, reaching an 85% of success in recognition. Furthermore, authors also showed how the introduction of PCA considerably reduces the computational speed but maintains the same performances.

A Region-Based Kernel Support Machine Vector (K-SVM) classifier was proposed and presented in [74] by Bao et al., and used to estimate the motion of an endoscopic capsule. In this work, images were segmented into several subregions and then a statistical region growing algorithm was used. Subsequently, images were divided into two basic label sets called: (i) Facing the Tunnel (FT), which is the case when the focal axis of the camera is parallel to the center of the intestinal tube; and (ii) Facing the Lumen wall (FL), which is the case when the capsule does not move. Through a comparison with region-based ANN and region-based linear SVM, it was shown how the K-SVM better outperforms, thus increasing the recognition success percentage from 80% to 90%. In [75], Brandao et al. proposed a Fully Convolutional Neural Network (FCN) to identify and segment polyps in colonoscopy images (Figure 2(e)). This framework, validated during the 2017 MICCAI polyp detection challenge dataset, showed a high accuracy, that is, 93.3%, in polyp detection.

Other studies were conducted in [76–78]. In [76], Aghanouri et al. proposed a method for estimating the rotation angle and the distance from the stomach wall of an active WCE. Through the usage of a Speeded Up Robust Features (SURF) method [79], the authors first extracted the images features; then, the Fast Library for Approximate Nearest Neighbours (FLANN) [80] and the M-estimator SAmple and Consensus (MSAC) method [81] were used in order to estimates the distance of the capsule from the wall of the stomach and its orientation from two consecutive frames. Estimation errors lower than 0.3° in orientation were obtained.

A similar approach was proposed by Iakovidis et al. in [77]. In this case, instead of using WCE video sequences from public repositories, authors validated the feasibility of their approach with an in-vitro experiment. Authors used: (i) a PillCam SB3 for data acquisition; (ii) a 300 mm lifelike double layer bowel phantom (Lifelike Biotissue Inc, Ontario, Canada) with a set of 0.95 mm circular pins attached in the interior synthetic wall and used as landmarks; and (iii) a high precision robotic system (RV3SB robot, Mitsubishi, Tokyo, Japan) which, through programmable parameters, that is, acceleration and speed, moved steadily the WCE along the bowel phantom. Due to the lack of information about the technical details of the commercial WCEs, camera calibration was performed according to the Zhang’s method [82] and the Brandt’s method [83]. Subsequently, the Scale Invariant Feature Transformation (SIFT) [84] was used to estimate the travelled distance through the scaling factor of consecutive frames. An extension of the work in [77] was presented by Dimas et al. in [78] (Figure 2(f)) and Iakovidis et al. in [85] In particular, authors proposed to improve the system performances in estimating the travelled distance, by using a multilayer Feedforward Neural Network (MFNN) [86] previously trained.

All the aforementioned works were focused: (i) on localization of GI abnormalities, such as polyps and lesions; and (ii) on giving localization information based on anatomic landmarks, for example, general landmarks, pylorus, stomach, and small/large intestine. In order to enable a more accurate 3D and 2D localization in the abdominal space, Bao et al. [87] proposed a hybrid localization technique of a WCE inside the small intestine obtained by data fusion of vision and RF sensors. As testbed, they considered a virtual 3D space where the small intestine was emulated as a cylindrical tube with size, shape, and colors extracted from real clinical data. Capsule transition inside the intestine was emulated by changing viewpoints along the cylindrical tube. Bunch of calibrated antennas attached to the external abdominal wall, which detected the commercial capsule RF transmitted signals (using a RSSI method), were also supposed. A Kalman filter, which combined vision and RF data, was used to estimate capsule position. Experimental results showed that such hybrid localization system better outperform than using separately either RF-based or image-based localization. An average error lower than 50 mm was obtained.

Further elaborations about this hybrid technique were conducted by Geng et al. in [88]. Large intestine and small intestine were considered for this study. In particular, by varying: (i) the number (from 8 to 72 receivers) and the topology (different arrangements of the receivers on two opposite planes of a cube) of the on-body RF receivers; and (ii) the accuracy of the image processing, the authors derived the 3D dimensional Posterior Cramer Rao Lower Bound (PCRLB) for such hybrid WCE localization. Through this simulation analysis, the authors showed that these methods could not achieve an accuracy lower than 30 mm.

With advancement of both data availability and of the algorithms using deep learning to detect and characterize WCE images, it may become possible to use this information within navigation system. The network detection outputs could act as reference guides to external navigation sensors and the fusion system may achieve added robustness.

Summarizing, several computer-vision algorithms have been implemented benefiting from artificial intelligence methodologies for computer-assisted diagnosis with very promising results and high accuracy for internal localization of pathologies, such as visual odometry to measure the distance travel [85] and automatic polyps detection using deep artificial neural networks [88]. In this direction, the localization strategy presented by Bao et al. [87]. can be identified as a very interesting and promising solution in this field. The mentioned localization technique, obtained by the fusion of multidimensional data (i.e. vision and RF signals), is a first significant example of a hybrid multimodality approach, even though it has not yet achieved high accuracy (lower than 50 mm).

4. Other types of localization strategies

In addition to the previously described consolidated methodologies, there are other and less investigated localization methods, which exploit traditional medical imaging principles, such as, X-Ray, Magnetic Resonance Imaging (MRI) and Ultrasounds (US), for pose tracking of general surgical devices, for example, needles and catheters. Some of them could be described in the previous sections, being, for example, classifiable as compatible or not compatible with locomotion capabilities. However, authors decided to describe them in a dedicated section for the benefit of a clearer and organic comprehension.

A first approach for calculating position and orientation of a catheter by two-dimensional X-Ray images was proposed in [89] by Boese et al. in 2005. The system consisted of an X-Ray beam source and a radiation detector plane. Shadows of the radiated object, collected on the radiation plane, were used for obtaining the corresponding 2D images. Since to a change of the radiated target pose corresponds a change in its projection on the detector plane, authors proposed an iterative algorithm comparing the shadow of the acquired image with the pre-processed 2D image shadow. Based on this approach, Kurth et al. [90] designed a system to estimate the position and orientation of an endoscopic capsule guided by external closed-loop controlled magnetic fields generated by an array of coils placed around the patient. As in [89], the tracking parameters, that is, position and orientation, were obtained from the shadow of the radiation images compared with a priori obtained 2D projection of the target. In 2011, Carpi et al. [91] validated this approach inside the different districts of the GI tract (i.e. esophagus, stomach, small bowel, and colon) by performing in-vivo tests in a domestic pig. Although this method was promising to deliver high accuracy localization, it could represent a potential hazard for the patient for the need of X-rays. This was anyway limited as the internal components of the capsule were metallic or radiation-opaque, so that it was possible to obtain X-ray images with a good quality using an extremely low radiation dose. The authors propose a mixed localization strategy based on RF triangulation approach to perform a rough 2D localization and then a finer 2D and 3D localization based on X-ray images, suitable for high diagnostic accuracy.

As regards the use of MRI for real-time localization of interventional devices, a possible approach was proposed by Dumoulin et al. [92]. This method consisted in incorporating one or more miniature RF coils into the medical device, which were used to sense a custom pulse series exploited for localization purpose. In 2005, this method was validated by Krieger et al. [93] for determining six DoFs position and orientation of a biopsy needle used for prostate biopsy. Although they obtained in in-vitro conditions a position and orientation error of 0.19 ± 0.25 mm and 0.33°±0.42°, respectively, the average in-plane displacement error for 20 biopsies in in-vivo condition was 1.8 mm (due to the thickness of the slice it is not possible to evaluate 3D position and 3D orientation). These methods were based on custom-programmed pulse sequences, which are different from the standard pulse sequences of commercial MRI scanners. Therefore, it is worth mentioning that the requirements of the generation of custom pulses could increase the entire system costs and complexity.

MRI technology has been also employed in designing a novel magnetic technology to steer a capsule during a gastric examination. In that case, as for the computer-vision based technology, capsule localization is performed with respect to the surrounding deformable anatomical environment and targets (i.e. internal localization), for example, landmarks and 3D geometry. That system [94,95], developed cooperatively by Olympus Inc. and Siemens Healthcare (Erlangen, Germany), includes an Olympus Inc. capsule endoscope (31 mm in length and 11 mm in diameter, provided with two 4 frames/s image sensors) and Siemens magnetic guidance equipment, composed of magnetic resonance imaging and computer tomography (CT). Two joysticks were used by physicians in order to perform a five DoFs steering control of the capsule, that is, 3D translation and tilting and rotation, inside the stomach. This framework, validated over a set of 55 volunteers and consentient patients, shows a promising recognition of the gastric pylorus, antrum, body, fundus, and cardia in 96%, 98%, 96%, 73%, and 75% of participants, respectively.

Finally, also the use of ultrasounds was investigated for localization purposes, which resulted to be very promising in providing high speed, safe, and low-cost localization outcomes. The position information could be obtained with two different approaches: (i) a passive approach [96], which is based on measuring the time of flight (ToF) between ultrasonic pulses transmitted from an external source and the echoes reflected by the capsule; and (ii) an active approach [97], consisting in embedding an ultrasound transducer inside the capsule, and external receivers located around the patient’s abdomen to detect the emitted ultrasonic signals. The second approach presents several advantages compared to the echo pulse detecting one. In particular, clear advantages are: (i) an improved signal-to-noise ratio since the pulse must only propagate once through the media; and (ii) better identification of minimally invasive medical devices like endoscopes and catheters, which are difficult to be identified with pulse-echo ultrasounds. However, there is still the drawback on how to integrate an ultrasound emitter inside such medical devices – mainly in case of wireless medical systems – since available ultrasonic emitters consist of piezoelectric crystals excited with a harmonic voltage of up to several hundred Volts, or microelectromechanical (MEMS) devices that still require up to 25 V driving voltage.

In 2009, Nagy et al. [98] proposed and implemented a wireless resonant magnetic micro actuator for ultrasound generation, which consisted of two parallel soft magnetic square dies of nickel (facing area of 1 mm², and thickness of 50 μm), separated by a 125-μm-thick plastic shim. Thanks to an external alternating magnetic field generated by an electromagnet, the soft magnetic dies becomes magnetized and then the attractive/repulsive force between them results in ultrasounds generation.

It is worth mentioning – even if not directly related to localization so far – the significant research-oriented effort on designing new endoscopic capsules with embedded ultrasound modules for diagnostic and therapeutic applications. Endoscopic devices were developed within the framework of the UK project, named Sonopill [99]. A summary descriptive table about the aforementioned localization strategies is provided in Appendix C (Table A3).

5. Expert commentary

Conventional flexible endoscopy is the gold standard for both diagnostic and therapeutic GI procedures. Traditional endoscopes are made of a semiflexible continuum tube – manually pushed into the lumen from the outside – and the localization of the tip is deduced by the length of the inserted instrument and by identification of landmarks and characteristic shapes of the lumen. However, the flexibility of the large intestine and its capacity of stretching make the measurement of the distance travelled into the lumen not accurate enough if not in the retrieving condition. For this reason, conventional flexible endoscopy is not able to offer a precise localization system especially for lower GI. Endoscopists are often obliged to perform internal localization, mainly in case of lesions/pathologies retargeting and for follow-up, injecting ink tattoos in the submucosal layer creating artificial targets. Modern endoscopes offer the possibility of transillumination to perform a not-continuous external localization allowing the transparietal visualization of the light coming from the tip of the scope.

With the introduction of new methods for GI tract investigation, such as WCEs, clinicians have lost the direct manual control of the scope. For this reason, a reliable and accurate localization system to calculate the endoscope position is needed, in order to: (i) measure the travelled distance and identify the position of pathologies during the endoscopic procedure for follow-up interventions and other therapeutic operations, such as drug delivery; and (ii) externally guide the capsule through an active closed-loop navigation method, for example, magnetically driven capsule locomotion, by also guaranteeing at the same time patient safety.

Indeed, penetration of the endoscopic market for these new smart technologies is mainly hampered by numerous important and still unresolved problems, among which the realization of an accurate and reliable localization system is of fundamental importance. One of the key challenges concerns the relationship between precision and robustness of the data and size of the localization system. In general, more precise systems require larger dimensions, which however are not always acceptable. Furthermore, most of the systems presented in this article are still under research (with a low Technology Readiness Level – TRL – usually below 4) and have been tested under controlled in-vitro conditions and/or in limited volumes. For example, most of the magnetic tracking systems guarantee good operational functions in a small workspace. Others, however, are affected by numerous interferences that strongly limit their use in clinical practice. On the other hand, there is great potential in systems that exploit electromagnetic waves in the visible field, as they can guarantee sufficiently accurate results. However, this type of systems alone does not allow to locate a target with respect to an external reference system, key-aspect when facing with a soft and deformable environment, such as the GI tract, for achieving closed-loop locomotion.

For this reason, the authors believe that the paramount challenge is the integration of multiple systems (currently, the most-promising approaches are magnetic and visible waves-based approaches) to increase the accuracy, but also reliability, of the localization module. In particular, thanks to a magnetic approach, it is possible to estimate the position and orientation of the capsule with respect to an external reference system, while with the visual module, already embedded into the capsule as cameras, the position and orientation of the capsule can be identified with respect to an internal reference system, that is deformable and unstructured (i.e. combination of external and internal localization approaches). In this way, it is possible to define the correct pose and then the optimal trajectory of the capsule movement during the entire procedure, within a deformable environment, together with pathologies location.

However, in order to succeed with this challenge, it is still necessary to reliably solve some open questions, such as: (i) correct modelling of the magnetic distribution in space (i.e. in the near and far field); (ii) management of electromagnetic interferences deriving from the external environment; (iii) modelling of the environmental deformability of the GI tract, which is highly unstructured during the procedure, etc.

6. Five-year view

There are numerous significant improvements that are due to be accessible within the next five years. In particular, at the beginning of a progressive replacement of the current endoscopic techniques with automatic or semiautomatic robotic systems, the authors envisage the clinician will only supervise the medical procedure and will intervene only at critical times, for example, in the endosurgical phase for excision and in the decisional ones after, for instance, a pathology detection alert. In particular, the robotic system could autonomously define the optimal trajectory in order to cover the entire GI tract and to keep the tip-located camera in the correct position to be able to view and semi-autonomously diagnose 100% of the internal GI walls.

However, this perspective will only attainable with improvements to: (i) performing an accurate pose estimation; and to (ii) gaining the capability of a highly reliable and precise closed-loop control. As already mentioned above, currently no approach, considered individually, is enough to guarantee these improvements; however, the authors envisage that it will be achievable soon using the fusion of information, for example, deriving from different approaches (i.e. multimodal localization module, internal and external).

Vision will also be one of the fundamental components of an automatic system because: (i) it is present by definition in any endoscopic probe; (ii) it does not introduce complexity to the system; and (iii) at the same time, it guarantees to constantly have an internal reference of the deformable lumen. In addition, autonomous or semiautonomous diagnose of GI pathologies is drawing more and more the attention of several research and industrial groups that are focusing on machine and deep-learning techniques to outline the medical report of GI pathologies for assisting physicians in the on-line (during conventional endoscopy) or in the off-line (during postprocedure WCEs analysis) diagnosis. It is worth mentioning that, the latter, together with an automatic or semi-automatic robotic navigation, significantly contribute to the overall reduction of the hospital cost for GI endoscopic procedures, that is, expert clinicians supervise several examinations in parallel, intervening only and if needed.

Finally, it is worth to mention that another aspect, which is beginning and will continue to become increasingly important during all surgical procedures, is the usage of additional patient’s anatomical information coming from different multimodal analysis, for example, computed tomography, X-Ray, Gamma-Ray etc. In addition to increasing the accuracy of the localization systems, this type of data fusion will be helpful in reducing the invasiveness of numerous surgical interventions.

Key issues

• GI diseases represent a critical threat to quality of the life and often lead to death. The reported number of deaths for colorectal and gastric cancers are ranked third and fifth in all cancer mortality. Diagnosis at an early stage represents a key-factor to reduce mortality. For CRC and gastric cancers, the 5-year survival rate in case of early stage pathology is >90%, falling to <20% in case of late diagnosis.

• Since the advent of the fiber optics, conventional endoscopy (i.e. gastroscopy and colonoscopy) represents the gold standard medical procedure for diagnosis of the GI tract. Commonly, endoscopes are manually inserted and pushed in the gastrointestinal tract through mouth or anus, depending if the upper or lower part of the GI tract is explored.

• However, the willingness of patients to undergo endoscopy (especially for the lower GI tract) is consistently low due to pain and discomfort, particularly in case of not sufficiently skilled physicians, for several reasons, for example: (i) invasiveness, rigidity and large dimensions of the scopes, (ii) fear of pain during due to locomotion and insufflation, and (iii) the possibility of both cross-contamination and intestine perforation.

• WCE, such as: (i) PillCam®, originally called M2A and produced by Given Image Ltd. (Yokneam Illit, Israel), now released by Medtronic Inc. (Minneapolis, MN, USA); (ii) EndoCapsule from Olympus Group; (iii) OMOM, developed by Jinshan Science & Technology Group Co., Ltd (Chongqing, China); and (iv) MiroCam® (IntroMedic Co Ltd., Guro-Gu, Seoul, South Korea), were developed and commercialized. Since patients have only to swallow it and propulsion is performed through natural peristaltic contractions of the intestine, this screening approach results to be painless and more accepted by patients.

• Accurate knowledge of the position and orientation of the capsule, when it moves along the GI tract and captures images, represents an essential information that can be exploited by physicians to: (i) localize pathologies; (ii) follow-up diagnosis and intervention; and (iii) to assist navigation of active locomotion WCE. Due to its importance, several strategies for WCE pose estimation, that range from the use of magnetic/electromagnetic fields to ultrasounds and computer-vision technologies, have been investigated in the last years.

• Magnetic field-based localization strategies have significantly captured the attention of several academic and industrial groups. Interest is essentially motivated by the intrinsic advantages of magnetic fields such as: (i) the low attenuation factor passing through the human body; (ii) the capability of the magnetic-based sensor technologies to detect the target without the limitation of the line-of-sight between target and detection system; and (iii) the possibility to use magnetic fields for capsule steering and locomotion. One of the most challenging problems for such strategies is related to the possible interference between localization and locomotion modules.

• Starting from the well-consolidated Radio Frequency (RF) based localization strategies – used for localizing a target both in external and internal environments with a precision in the order several centimeters – a few solutions have been proposed to adapt these strategies for WCE pose estimation, obtaining a millimeter-based accuracy. These strategies are based on exploiting the RF signal transmitted from the WCE out of patient’s body. However, these techniques still require significant improvements in order to limit the dependence on (i) receiver/detector position, (ii) number of detectors, and (iii) noise from external sources.

• Visible waves are used for the aim of capsule localization through computer vision methodologies. Unlike the magnetic field-based and RF-based localization strategies, in which the capsule’s pose is identified with respect to an external reference frame (i.e. external localization), in computer vision strategies, images that come from capsule are used to localize the capsule itself (and also lesions and pathologies) with respect to the surrounding deformable anatomical environment and targets (i.e. internal localization). One of the most challenging problems for vision strategies is related to the computational cost and complexity, required in visual data analysis.

• Other types of localization strategies have been proposed for WCE and medical device pose calculation. Although these alternative methods should reach significantly high accuracy in pose detection, they may represent a potential hazard for the patient due to the use of ionizing radiation and may result in consistently high cost for the equipment and for each use.

• In the vision of a progressive replacement of the current endoscopic manually driven techniques with automatic or semi-automatic robotic systems, in which the clinician will supervise the medical procedure and will intervene only at critical points, in the next five years, authors envisage improvements in performing capsule accurate pose estimations and then in the capability of obtaining a highly reliable and precise closed-loop control, through the fusion of information deriving from different internal and external localization approaches. Moreover, cameras will be also used to feed machine and deep-learning-based algorithms for autonomous or semiautonomous diagnosis in order to assist medical doctors in the on-line (during conventional endoscopy) or in the off-line (during postprocedure WCEs analysis) screening.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgments

The authors thank all the collaborators of the EU Endoo Project and the colleagues of the Beijing Advanced Innovation Center for Intelligent Robots and Systems (Beijing Institute of Technology, Beijing, China) involved in the robotic capsule research initiative. In addition, authors thank Dr. Calogero Maria Oddo for the valuable support on this research topic.

Additional information

Funding

The work described in this paper was supported by the European Commission within the framework of the ‘Endoscopic versatile robotic guidance, diagnosis and therapy of magnetic-driven soft-tethered endoluminal robots’ Project, H2020-ICT-24-2015 (Endoo EU Project, G.A. number: 688592).

Appendix A

Table A1. Summary table of magnetic field-based localization strategies.

Magnetic field-based localization
Reference(s)	Compatibility with magnetic locomotion [YES/NO]	Update rate	Main components of the localization system	Passive/active components embedded into the device	Test conditions	DoFs	Position Accuracy (PA) Orientation Accuracy (OA)	Notes
Hu et al., 2005 [13]	NO	~137 ms	Circular IPM; #16 3-axis Hall-effect array of sensors	Passive, sensing magnetic field of IPM	Simulations and free-space validation	3D position; 2D orientation (pitch/yaw)	PA: 5.6 mm; and OA: 4.26% unit orientation vector	Linear optimizer
Hu et al., 2005 [16]	NO	<10.6 ms	Circular IPM; #16 3-axis Hall-effect array of sensors	Passive, sensing magnetic field of IPM	Simulations and free-space validation	3D position; 2D orientation (pitch/yaw)	PA: 5.6 mm; and OA: 4.26% unit orientation vector	Linear optimizer
Wang et al. [17]	NO	100–200 ms	Circular IPM; #16 3-axis magneto-resistive array of sensors	Passive, sensing magnetic field of IPM	Simulations and free-space validation (100 mm above the array of sensors)	3D position; 2D orientation (pitch/yaw)	PA: 3.3 mm (max. 10 mm); and OA: 4.5° (max. 5.7°)	Linear optimizer
Hu et al., 2008 [18]	NO	N/A	Circular IPM; #16 3-axis magnetic array of sensors	Passive, sensing magnetic field of IPM	Simulations and free-space validation (240x240 mm^2)	3D position; 2D orientation (pitch/yaw)	PA: 2 mm; OA: 1.6°	2-step localization: rough estimation with linear algorithm and fine estimation with nonlinear optimizer
Yang et al. [19]	NO	t1*: 170 ms; t2**: 630 ms	Rectangular IPM; #16 3-axis magneto-resistive array of sensors	Passive, sensing magnetic field of IPM	Simulations	3D position; 3D orientation	PA: 3.9 mm* and 0.59 mm**; OA: 5.06°* and 0.66°**	Particle swarm optimizer (PSO)
Hu et al. [21]	NO	~100 ms	Rectangular IPM; cubic magnetic array of sensors composed by #64 HMC1043 3-axis magnetic sensors	Passive, sensing magnetic field of IPM	Simulations and free-space validation (up to 250 mm between the sensor and the capsule)	3D position; 3D orientation	PA: 1.82 mm; OA: 1.62°	Particle swarm optimizer (PSO); improved performance compared to Yang et al. [18]
Song et al. [22]	NO	830 s	Annular IPM; cubic magnetic array of sensors composed by HMC1043 3-axis magnetic sensors	Passive, sensing magnetic field of IPM	Simulations (360 × 370 × 300 mm^3)	3D position; 3D orientation	PA: 0.003 mm; OA: 0.036°	Particle swarm optimizer (PSO) and derived mathematical model (analytical magnetic model)
Hu et al. [23]	NO	N/A	Ring-shape IPM; wearable magnetic array of sensors of #32 3-axis magnetic sensors	Passive, sensing magnetic field of IPM	Simulation and free-space validation	3D position; 3D orientation	PA: 3.82 mm; OA: 2.2°	Another 2 permanent magnets attached to the surface of the human body used as reference sources for body motion compensation
Plotkin et al. [24,25]	NO	20 ms	Sub-miniature induction coil embedded in the capsule as sensor; 8 × 8 of circular coplanar transmitting coils	Active, magnetic field generated by external sources detected by an embedded coil	Free-space validation (200 mm distance between the coaxial receiving and transmitting coils)	3D position; 3D orientation	PA: ~0.75 mm; OA: 0.6°	Maximum operative range of 200 mm; Levenberg-Marquardt algorithm
Nagaoka and Uchiyama [26]	NO	66 ms	Internal coil (secondary) used for FM signal generation; 5 alternating magnetic fields generated outside from solenoid coils (primary)	Active, magnetic field generated by external sources detected by an internal coil	Free-space validation (up to 500 mm to the magnetic field generator)	3D position	PA: 2.8 ± 2.2 mm; OA: 13.4°± 20.9°	Maximum operative range of 500 mm; flat band between 50 and 500kHz obtained through characterization of living tissue
Islam and Fleming [27]	NO	N/A	3-orthogonal rectangular coils inside a capsule and used as tri-axial sensor	Active, magnetic field generated by external sources detected by an internal coil	Free-space validation (up to 257 mm to the magnetic field generator)	3D position; 3D orientation	PA: 6 ± 0.28 mm; OA: 1.11° ± 0.04°	Optimization of coil-shaped sensor volume inside the WCE
Ciuti et al. [29]	YES	50 ms	6-DoFs robotic arm with an EPM; a capsule device with IPMs; IEEE 802.15.4 module; 3-axis accelerometer	Active, localization based on impulse signals obtained from the accelerometer	Ex-vivo experiments	2D position; 2D orientation (pitch/roll)	PA: 30 mm; OA: 6°	For orientation accuracy see STM LIS331DL sensor datasheet
Salerno et al. [31]	YES	27 s	6-DoFs robotic arm with an EPM; capsule device with IPMs and embedded Hall-effect sensors	Active, sensing magnetic field of EPM	Free-space validation (150x150x200 mm^3)	3D position	PA: −3.2 ± 18 mm along X; 5.4 ± 15 mm along Y: and −13 ± 19 mm along Z	Analytical charge-model of external and internal magnetic field
Di Natali et al. [33,34]	YES	7 ms	6-DoFs robotic arm with an EPM; capsule device with IPMs; embedded Hall-effect sensors; 3-axis accelerometer	Active, sensing magnetic field of EPM	Free-space validation (200×200×60-120 mm^3)	3D position; 3D orientation	PA: 6.2 ± 4.4 mm radial; 6.9 ± 3.9 mm axial; 5.4° ± 7.9° azimuth; OA: 0.27° ± 0.17° pitch; 0.34° ± 0.18° yaw; 1.8° ± 1.1° roll	Maximum operating range of 150 mm; pitch and roll angle obtained from accelerometer
Son et al. [36]	YES	5 ms	2D array (8x8) of monoaxial Hall-effect sensors and an omnidirectional electromagnet made of three box-shaped orthogonal coils and a soft iron core	Passive, sensing magnetic field of IPM	Free-space validation (70×70×50 mm^3)	3D position; 2D orientation	PA: 2.1 ± 0.8 mm; OA: 6.7° ± 4.3°	2-step algorithm for reducing the influence of the magnetic locomotion module interference acting with electromagnetic-based fields
Taddese et al. [38]	YES	10 ms	6-DoFs robotic arm with an EPM and an electromagnet fixed around the EPM; a capsule device with IPMs; Hall-effect sensors; 3-axis accelerometer	Active, sensing magnetic field of EPM	Free-space validation (radii of the hemisphere: 150–200 mm, 10 mm step)	3D position; 3D orientation	PA: <5 mm; OA: <6° (static tests along a spiral trajectory)	Hybrid permanent magnet/electromagnet-based model (point-dipole model) with particle filtering method
Abbott et al. [43–47]	YES	3400 ms	6-DoFs robotic arm with a rotational permanent magnet; Maxon motor for controlling the magnetic end-effector; capsule device with an IPM and six mono-axial Hall-effect sensors	Active, sensing magnetic field of EPM	Simulations and free-space validation (spherical shell from 3ρ to 8ρ – ρ: 25.4 mm, i.e., radius of the external spherical magnet)	3D position; 3D orientation	PA: 4.9 ± 2.7 mm; OA: 3.3° ± 1.7°	Dipole-dipole analytical magnetic model with Levenberg-Marquardt method

Table A2. Summary table of electromagnetic field-based localization strategies.

Electromagnetic field-based localization
Reference(s)	Compatibility with magnetic locomotion [YES/NO]	Update rate	Main components of the localization system	Passive/active components embedded into the device	Test conditions	DoFs	Position Accuracy (PA) Orientation Accuracy (OA)	Notes
[53–56]	N/A	N/A	Capsule device with RF transmitting antenna	Active strategy, localization based on transmitted RF signals	Simulations	N/A	N/A	Studies focused on modelling propagation of RF waves through human body
Chandra et al. [57]	N/A	From 3 s to 17 s (cell size)	Capsule device with RF transmitting antenna; 4 receiving antennas on the human body surface	Active strategy, localization based on transmitted RF signals	Simulations and test on different phantoms (realistic phantom* and circular phantom**)	3D position	PA: 9mm*; 3 mm**	The authors proposed to acquire an image of the human body before the procedure in order to obtain electrical properties
Li et al. [58]	N/A	N/A	Capsule device with RF transmitting antenna; 4 receiving antennas on human body surface	Active strategy, localization based on transmitted RF signals	Simulations (400 ×200×350 mm^3)	3D position	PA: range 80–110 mm	Localization based on Cramer Rao Lower Bound (CRLB); used signal propagation model developed by NIST
Ye et al. [60]	N/A	N/A	Capsule device with RF transmitting antenna; array of 32 receiving antennas on human body surface	Active strategy, localization based on transmitted RF signals	Simulations (268×323×312 mm^3)	3D position	PA: range 45–50 mm	Localization based on Cramer Rao Lower Bound (CRLB); frequency depends by dielectric properties of more than 300 parts of a male human body; modification of NIST propagation loss model
Hou et al. [62]	N/A	N/A	Capsule device with RFID tag; 3D array of antennas surrounding human body	Hybrid	Simulations and ex-vivo tests	3D position	PA: 20 mm	No need of signal attenuation model of the human body
Hekimian-Williams et al. [64]	N/A	N/A	3D array of antennas surrounding the human body	Hybrid, based on phase difference of receiving antennas (RFID)	Simulation	3D position	PA: 1.8 mm	Localization based on maximum likelihood estimation algorithm
Wille et al. [65]	N/A	N/A	Capsule device with RF transmitting antenna; 3D array of antennas surrounding the human body	Active, based on phase difference of receiving antennas (RFID)	Simulation (500×900×200 mm^3)	3D position	PA: 1.6 ± 1.2 mm @grid 5 mm, 7.8 ± 5.2 mm @grid 10 mm	Localization based on support Vector Regression Algorithm (VRA)
Liu et al. [67]	N/A	N/A	Capsule device with RF transmitting antenna; 3D array of antennas surrounding the human body	Active, based on Time of Arrival (ToA)	Simulations (597x21.6x203 mm^3)	3D position	N/A	Mathematical model derived by experimental tests and compared with SEMCAD X software simulations
Pourhomayoun et al [68]	N/A	10 s for 16 sensors and 3 s for 4 sensors	Capsule device with RF transmitting antenna; 3D array of antennas surrounding the human body	Active, based on spatial diversity of transmitting antenna	Simulations	3D position	PA: <8.8 mm	Mathematical model derived by extensive Monte Carlo simulations for radiofrequency emission signals
Nafchi et al. [69]	N/A	1 s (sampling time); 1 ms processing time	Capsule device with RF transmitting antenna; 3D array of antennas surrounding the human body; Inertial Measurement Unit (IMU)	Active, based on both DoA and ToA	Simulations	3D position	PA: 10 mm with 16 sensors	Extended Kalman filter for position estimation
Duda et al. [72]	N/A	N/A	Endoscopic device with an integrated camera	Active, based on computer vision	Analysis of acquired images	3D position/classification	N/A	Comparative analysis of ANN, VQ and VQ + PCA classification algorithms
Bao et al. [74]	N/A	N/A	Capsule device with an integrated camera	Active, based on computer vision	Analysis of acquired images	3D position/classification	N/A	Region-Based Kernel Support Machine Vector (K-SVM) classifier proposed to estimate endoscopic capsule motion
Brandao et al. [75]	N/A	N/A	Endoscopic device with an integrated camera	Active, based on computer vision	Analysis of acquired images	3D position/classification	N/A	Polyps detection through Fully Convolutional Neural Network algorithm
Aghanouri et al. [76]	N/A	N/A	Capsule device with an integrated camera	Active, based on computer vision	Analysis of acquired images	3D position 3D orientation	OA: 0.3°	Evaluation of travelled distance and orientation change from consecutive image frames
Iakovidis et al. [77,78,85]	N/A	N/A	Capsule device with an integrated camera; 6-DoFs Mitsubishi RV3SB robotic arm as benchmark	Active, based on computer vision	Analysis of acquired data from in-vitro experiments	3D position	PA: 1 mm	Evaluation of travelled distance (visual odometry)
Geng et al. [88]	N/A	N/A	Capsule device with an integrated camera and a transmitting antenna	Active, based on computer vision and RF data fusion	Simulations	3D position	PA: <30 mm	Localization based on Cramer Rao Lower Bound (CRLB)

Table A3. Summary table of other types of localization strategies.

Other types of localization strategies
Reference(s)	Compatibility with magnetic locomotion [YES/NO]	Update Rate	Main components of the localization system	Passive/active components embedded into the device	Test conditions	DoFs	Position Accuracy (PA) Orientation Accuracy (OA)	Notes
Boese et al. [89]	YES (not mentioned)	N/A	X-Ray beam; detector plane	Passive, based on the shadow of radiation pattern	N/A	3D position; 3D orientation	N/A	Test conditions not provided (patent)
Kurth et al. [90]	YES	N/A	Capsule device with IPM; X-Ray beam; detector plane; electromagnetic coils for steering	Passive, based on the shadow of radiation pattern	N/A	3D position; 3D orientation	N/A	Test conditions not provided (patent)
Carpi et al. [91]	YES	N/A	Magnetic driven capsule device (Stereotaxis system); RF signal triangulation complemented with X-ray imaging.	Hybrid, based on the shadow of radiation pattern and RF signal triangulation	In-vivo tests on domestic pig	3D position;	PA: 1 mm; OA: N/A	Hybrid localization strategy: signal triangulation (rough 2D localization) and X-ray imaging (finer 2D and 3D localization)
Krieger et al. [93]	YES (not mentioned)	N/A	Biopsy needle; Magnetic Resonance Pulse generator; RF coils for pulse detection	Passive, based on Magnetic Resonance Pulse	In-vitro, in-vivo canine model and in-vivo clinical trials	3D position; 3D orientation	PA: 0.19 ± 0.25 mm; OA: 0.33°±0.42° (in-vitro)	Method applied for needle prostate biopsy but applicable to WCE
Fluckiger et al. [97]	YES (not mentioned)	N/A	Capsule device with ultrasound emitter; ultrasounds receiver on the human body surface	Active, based on ultrasound transmitted by the capsule	In-vitro tests (circular area of 140 mm in diameter)	3D position	PA: 748 ± 310 µm	Algorithm based on FEM simulations with heterogeneous media