Influence of sampling accuracy on augmented reality for laparoscopic image-guided surgery

Abstract

Purpose

This study aims to evaluate the accuracy of point-based registration (PBR) when used for augmented reality (AR) in laparoscopic liver resection surgery.

Material and methods

The study was conducted in three different scenarios in which the accuracy of sampling targets for PBR decreases: using an assessment phantom with machined divot holes, a patient-specific liver phantom with markers visible in computed tomography (CT) scans and in vivo, relying on the surgeon’s anatomical understanding to perform annotations. Target registration error (TRE) and fiducial registration error (FRE) were computed using five randomly selected positions for image-to-patient registration.

Results

AR with intra-operative CT scanning showed a mean TRE of 6.9 mm for the machined phantom, 7.9 mm for the patient-specific phantom and 13.4 mm in the in vivo study.

Conclusions

AR showed an increase in both TRE and FRE throughout the experimental studies, proving that AR is not robust to the sampling accuracy of the targets used to compute image-to-patient registration. Moreover, an influence of the size of the volume to be register was observed. Hence, it is advisable to reduce both errors due to annotations and the size of registration volumes, which can cause large errors in AR systems.

Keywords:

• Image-guided surgery
• augmented reality
• target registration error
• laparoscopic surgery
• hand-eye calibration

Introduction

Image-guided surgery (IGS) systems aim to provide navigation to surgeons in order to improve the accuracy and safety of the procedures. IGS utilizes computer-based systems to provide virtual image overlays and target the surgical sites [1]. In the past 30 years, with the technological advances in computer science and medical imaging, IGS has greatly expanded [1–3]. IGS combines medical images, such as magnetic resonance imaging (MRI) or computed tomography (CT), with the intra-operative images shown during the surgery to the surgeon through a laparoscope camera, endoscope camera or ultrasound (US) image. This information is displayed to surgeons either through 3D models on separate monitors or overlaid as augmented reality (AR). Instrument tracking technologies are used in IGS to provide reliable information regarding position and orientation of surgical instruments. Moreover, tracking technologies are also used for registration tasks [4].

This study’s field of application of IGS is Liver Laparoscopic Resection Surgery (LLRS). Conventionally, before LLRS, the patient undergoes volumetric scans, such as CT or MRI, to diagnose and to plan optimal treatment. These scans, known as pre-operative scans, are used by the surgeons to plan the removal of tumours from the liver [5]. However, when the patient is on the surgical table, his/her position and orientation are different from the scanning position. For this reason, surgeons use their understanding of the anatomy of the liver and intra-operative imaging modalities (such as laparoscopic video and US) to spatially correlate the anatomy of the liver to the diagnostic CT or MRI scans. This approach may not only introduce inaccuracies, but also makes the surgery more dependent on the experience of the surgeon. Moreover, in laparoscopic surgery, pneumoperitoneum (inflation of the patient’s abdomen) is always performed to have sufficient space to introduce laparoscopic instruments and the laparoscopic camera into the abdomen. This is problematic because pneumoperitoneum also deforms the shape of the organs [6], making the anatomical correlation with the CT scan even more complicated for the surgeon.

IGS can help the surgeon avoid an intra-operative unfavourable incidence by providing surgical navigation. This is becoming more important especially because parenchyma-sparing (PS) liver resection has become a standard surgical treatment for colorectal liver metastases (CRLM) [7] because it facilitates repeated liver resections, which can increase survival [8]. During these procedures, lesions need to be located intra-operatively, resection performed with planned margin, and vessels cut at optimal locations. IGS can provide surgeons with an overview of the vascular structures, position of the lesion, etc. directly in the operating field. For example, for resection in posterosuperior liver segments, which in some cases are more complicated than formal resections due to accessibility and visibility [9], IGS could be used to focus attention in the correct direction and on the structure where the lesion is located.

Within IGS, AR is a computer vision technique in which computer-generated images are superimposed onto video frames to enhance the visualization and improve the spatial understanding of the scene. In IGS, AR is achieved via superposition of 3D reconstructions of segmented medical volumes, such as the CT scan, of structures of interest (e.g. tumours or blood vessels as shown in Figure 1), on the laparoscope camera [10].

Figure 1. Use of segmentation models and CT scans as a navigation map for laparoscopic liver resection surgery.

This study aims to understand the influence of human-induced error for AR on the laparoscope camera during LLRS. This experimental work studies AR accuracy in three experiments: an accuracy verification phantom, a patient-specific liver phantom and an in vivo porcine model with intra-operative CT scan.

Material and methods

Background methodology

In LLRS, the surgeon conventionally performs surgery visualizing the organs through the video of a laparoscopic camera, together with assistance of medical images and, when available, 3D reconstructed surface models through segmentation. All this information can be displayed on a separate screen within the OR (as shown in Figure 1) or can be combined to the laparoscope perspective into AR (as shown in Figure 2). As aforementioned, this study focuses on AR. The following chapter describes the main algorithms used to achieve AR in this study: hand-eye camera calibration, point-based registration (PBR) and AR re-projection.

Figure 2. Frames of AR showing re-projected blood vessel structures in the phantom (left) and the in vivo study (right).

Hand-eye camera calibration

‘Hand-eye’ camera calibration was coined in the robotics field [11]. In IGS for LLRS, hand-eye camera calibration aims to compute the transformation between a stereo laparoscope camera (the ‘eye’), ENDOEYE flex3D (Olympus, Tokyo, Japan), and the instrument markers rigidly attached to the camera (the ‘hand’), as shown in Figure 3. Instrument tracking was achieved in this study using the optical tracking system Polaris Spectra (NDI, Waterloo, Canada). Optical markers were rigidly attached to the laparoscope camera using an NDI Polaris Rigid Body (Part Number 8700449). The six degrees of freedom of the laparoscope camera were then tracked at a sampling rate of 60 Hz. To perform camera calibration and hand-eye camera calibration, a calibration plate with a 96-dot pattern, with four optical markers at accurately machined locations, was manufactured by Cascination (Cascination, Bern, Switzerland). In this study, the left channel of the stereo laparoscope camera was calibrated and used for AR evaluation. The calibration plate is white with laser-printed black circles, OpenCV algorithms were used for detection of the ellipsoidal centroids and for camera calibration [12]. To compute hand-eye calibration, Equation (1)(1) $$T_C^M=\mathrm{ }{T}_O^M·\mathrm{ }{T}_S^O·\mathrm{ }{T}_{SC}^S·\mathrm{ }{T}_C^{SC}$$(1) could have been used in order to compute the hand-eye calibration transform from a single pose (i.e.$\mathrm{ }{T}_C^M,$ according to Figure 3) (1) $$T_C^M=\mathrm{ }{T}_O^M·\mathrm{ }{T}_S^O·\mathrm{ }{T}_{SC}^S·\mathrm{ }{T}_C^{SC}$$(1)

Figure 3. Hand-eye camera calibration transformation diagram, from the laparoscope camera (C) to the optical markers attached to it (M).

where O is the coordinate system for the optical tracking system, M for the optical markers attached to the laparoscope camera, S for the optical markers on the calibration plate, SC for the calibration plate and C is the coordinate system origin for the camera pose. The notation used in this paper indicates as superscript the coordinate system with respect to which the transformation is applied, and subscripted is the towards which coordinate system. Moreover, all transformations described in this study are 4 × 4 matrices in homogeneous coordinates. $T_O^M$ and $T_O^S$ are provided by the optical tracking system, whereas $T_{SC}^S\mathrm{ }$ is obtained though pose estimation of the calibration plate according to Zhang [12]. The calibration plate was manufactured so that the axes and origins of S and SC coincide, therefore, for this study, $T_{SC}^S$ is a 4 × 4 identity matrix.

In order to improve both accuracy and reliability of the hand-eye camera calibration, instead of using Equation (1)(1) $$T_C^M=\mathrm{ }{T}_O^M·\mathrm{ }{T}_S^O·\mathrm{ }{T}_{SC}^S·\mathrm{ }{T}_C^{SC}$$(1) , the authors of this study implemented a multiple posed (N-posed) hand-eye camera calibration, based on Lee et al. [13]. In order to extend the set of equation for multiple (N) poses, we can rewrite Equation (1)(1) $$T_C^M=\mathrm{ }{T}_O^M·\mathrm{ }{T}_S^O·\mathrm{ }{T}_{SC}^S·\mathrm{ }{T}_C^{SC}$$(1) as follows: (2) $$\left[\begin{array}{c}{\left[\begin{array}{cc}I\otimes {\left(R_O^M\right)}^{-1}& Z_{9,3}\\ Z_{3,9}& {\left(R_O^M\right)}^{-1}\end{array}\right]}_1\\ ⋮\\ {\left[\begin{array}{cc}I\otimes {\left(R_O^M\right)}^{-1}& Z_{9,3}\\ Z_{3,9}& {\left(R_O^M\right)}^{-1}\end{array}\right]}_N\end{array}\right]\times \left[\begin{array}{c}vec\left(R_C^M\right)\\ t_C^M\end{array}\right]=\mathrm{ }\left[\begin{array}{c}{\left[\begin{array}{c}vec\left(R_C^O\right)\\ t_C^O+ {\left(R_O^M\right)}^{-1}·t_O^M \end{array}\right]}_1\\ ⋮\\ {\left[\begin{array}{c}vec\left(R_C^O\right)\\ t_C^O+ {\left(R_O^M\right)}^{-1}·t_O^M \end{array}\right]}_N\end{array}\right]$$(2)

Minimization of this linear system of matrices through least squares estimation results in matrix $T_C^M.$ More information regarding the accuracy and theory of the algorithms is available from studies by Lee et al. [13] and Lai and Shan [14].

Point-based registration

In the medical field, image registration aims to establish spatial correspondences between volumetric datasets [15]. The alignment between volumetric CT/MRI images (and segmented models) to the liver configuration when the patient is on the operating table is a field of image registration known as image-to-patient or image-to-physical registration. Two solutions are most commonly in use in the literature to perform image-to-patient registration: PBR [16] and the single landmark registration method [17]. Both algorithms work by sampling and matching a set of correspondent positions between coordinate systems. Within this study, PBR was implemented and tested to connect the image space I (the CT/MRI coordinates) and the patient’s position on the surgical table P. Consequently, image-to-patient registration is represented by transform $T_I^P.$ To achieve AR on the camera perspective, we use image-to-patient registration to transform the image coordinates I into camera coordinates C, as shown in Figure 4 following the equation: (3) $$T_I^C=\mathrm{ }{\left(T_C^M\right)}^{-1}\mathrm{ }·T_O^M·\mathrm{ }{T}_P^O·\mathrm{ }{T}_I^P$$(3)

Figure 4. AR transformation diagram which combines hand-eye calibration and image-to-patient registration.

However, if the points used for registration are already with respect to the optical tracking system, transformation $T_P^O\mathrm{ }$ will be an identity matrix. This can happen if we are registering to coordinate system O by sampling positions using an optically tracked instrument.

Re-projection of volumes in augmented reality

To complete the AR after Equation (3)(3) $$T_I^C=\mathrm{ }{\left(T_C^M\right)}^{-1}\mathrm{ }·T_O^M·\mathrm{ }{T}_P^O·\mathrm{ }{T}_I^P$$(3) , re-projection of 3D volumes to 2D images was performed using an additional transformation, commonly referred to as the perspective projection matrix or camera intrinsic matrix [12]. Re-projection of models in the I (Image) coordinate system as AR on the camera view can be performed through Equation (4)(4) $$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathrm{ }\underset{K}{\underbrace{\left[\begin{array}{ccc}{f}_x& \gamma & u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\mathrm{ }}}·{\left(T_C^M\right)}^{-1}\mathrm{ }·T_O^M·\mathrm{ }{T}_P^O·\mathrm{ }{T}_I^P·\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$(4) : (4) $$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathrm{ }\underset{K}{\underbrace{\left[\begin{array}{ccc}{f}_x& \gamma & u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\mathrm{ }}}·{\left(T_C^M\right)}^{-1}\mathrm{ }·T_O^M·\mathrm{ }{T}_P^O·\mathrm{ }{T}_I^P·\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$(4) where u and v are the 2D positions on the image plane in pixels, x, y, z are the 3D positions in the P coordinate system and matrix K is the intrinsic parameters matrix, computed through camera calibration. t is a nonhomogeneous transformed vector 3D positions in C coordinates (3 × 1 vector of the resulting 4 × 1 transformed points). Rectification parameters are also used during projection of the volumes.

Experimental protocol

The experiment protocol of this study aims to examine the influence of human errors (registration-related errors) on AR re-projection errors. The algorithms to generate the AR were kept consistent throughout each experiment. Three experiments were conducted with decreasing accuracy of sampling positions during image-to-patient registration. The first experiment makes use of a precisely machined, custom built optical validation phantom which follows ASTM standard F2554-10 for optical tracking accuracy measurement, described by Teatini et al. [4]. The second experiment evaluates AR accuracy on a patient-specific liver phantom with markers visible in the CT. Finally, in order to test the AR in a fully clinical scenario, the algorithms described above were tested through a porcine experiment (in vivo model).

For each experiment, image-to-patient registration was performed multiple times, always using five registration landmarks and the rest of the landmarks to compute the inaccuracy of the AR. Re-projection error in AR was computed as distance between manually annotated positions (ground truth positions) and their correspondent re-projected positions on the AR frames, as done also by Teatini et al. [18]. An example of the re-projected points, for all three experiments, is visible in Figure 5. Two parameters were evaluated in this study: fiducial registration error (FRE) and target registration error (TRE). FRE represents the accuracy of re-projected markers used to compute image-to-patient registration, and TRE the accuracy for all other points across the rest of the volume [16]. Both errors are computed as re-projection errors. These TRE and FRE fully represent, in our opinion, the accuracy of AR in terms of how well re-projected volumes are in comparison to ground truth positions. Moreover, to provide the reader with FRE and TRE of the AR in millimetres, and not only in pixels, the authors make use of the inverse of matrix K in Equation (4)(4) $$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathrm{ }\underset{K}{\underbrace{\left[\begin{array}{ccc}{f}_x& \gamma & u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\mathrm{ }}}·{\left(T_C^M\right)}^{-1}\mathrm{ }·T_O^M·\mathrm{ }{T}_P^O·\mathrm{ }{T}_I^P·\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$(4) , as described by Thompson et al. [19].

Figure 5. AR frames for each experiment procedure, validation phantom (left top), liver phantom (right top) and in vivo study (bottom). The registration targets were manually annotated from the frames, TRE and FRE were computed as the distances to the re-projected corresponding dots.

Validation phantom

The optical validation phantom was custom produced with 28 titanium target divot pins designed for TRE calculation on various planes and orientations, more details are available in [4]. Based on the algorithms previously described, the divot pins were registered and then re-projected onto the laparoscope frame as AR (shown in Figure 5). In this scenario, the accuracy of sampling registration targets is very good because they are precisely machined targets at measured locations.

Liver phantom

The patient-specific liver phantom is a phantom designed based on the CT scans of a patient. The liver phantom used throughout the experiments was produced by the ARTORG research centre (ARTORG, Bern, Switzerland) according to Pacioni et al. [20]. Fourteen stainless steel metallic M6 washers were glued around the whole surface of the liver phantom and served as landmarks. An intra-operative SOMATON Definition Edge CT scan (SIEMENS, Munich, Germany) was used to obtain a CT scan in the OR with the liver phantom positioned on the surgical table. The washers were segmented from the CT scan through intensity-based thresholding and clustered into positions through fuzzy means classification.

In comparison to the previous experiment on the validation phantom, the liver phantom introduces more inaccuracy in determining the correct positions to sample. This is due to the deformability of the phantom as well as the fact that the surgeon has to aim to the centre of the washers with a tracked pointer, instead of using precisely machined divot holes. Moreover, the 3D spatial locations of the markers are not precisely measured (clustered from the CT scan, as aforementioned), which further increases the sampling error.

In vivo model

The pre-clinical trial was necessary to calculate the AR accuracy in a more realistic clinical scenario: the positions for PBR are points sampled directly on the liver surface. For this reason, they are neither visible in the CT scan (like the metallic washers in the liver phantom), nor precisely machined divot positions (as in the validation phantom). Hence, the correspondence between the positions in the CT scan and the laparoscopic camera perspective is based on the surgeon’s anatomical understanding of the liver when required to annotate the locations sampled laparoscopically on the CT scan. An in vivo model of 59.5 kg was positioned on the surgical table in an OR equipped with the intra-operative CT scanner. After establishing the pneumoperitoneum to 13 mmHg through a Veress needle, an intra-operative CT scan was performed. Intra-operative imaging was performed rather than pre-operative imaging to minimize the inaccuracy due to soft-tissue deformation due to pneumoperitoneum. Successively, through an optically tracked monopolar cauterizer (Aesculap, Tuttlingen, Germany) 15 cauterization marks (ablation marks) were performed by the surgeon on the liver surface across the whole visible surface (similarly to [15]).

Reproducing the previous experiments, five of the targets were used to perform image-to-patient registration, whereas the other 10 were used to compute the accuracy. The cauterization marks performed on the liver surface were matched with the annotations made by the surgeon on a segmentation model of the liver parenchyma from the intra-operative CT scan.

This experiment relied on the surgeon’s anatomical understanding to calculate the positions of the landmarks on the CT scan. These annotation errors are computed through fiducial localization error (FLE) [21]. FLE was evaluated by laparoscopic insertion of needles at the centre location of the ablation marks, as to provide approximate ground truth positions for the annotations (approximate because insertion of the fiducials can cause some deformation to the liver tissue; moreover, segmentation and reconstruction errors may also present).

Results

For each experiment, a total of 100 AR frames were manually annotated to evaluate TRE and FRE. Five markers were used for registration and the rest for accuracy evaluations. Table 1 summarizes the average TRE for all three experiments for each registration procedures in millimetres.

Table 1. Results of TRE and FRE in (mm) for each experiment, separated into registration procedure, with standard deviation.

Registration	Validation phantom TRE (mm)	Validation phantom FRE (mm)	Liver phantom TRE (mm)	Liver phantom FRE (mm)	In vivo study TRE (mm)	In vivo study FRE (mm)
1	8.57 ± 2.83	9.02 ± 1.2	7.32 ± 5.53	7.72 ± 8.03	13.07 ± 5.44	12 ± 5.37
2	8.44 ± 1.84	7.39 ± 0.96	7.51 ± 6.06	6.69 ± 6.05	11.72 ± 6.3	10.6 ± 6.24
3	4.95 ± 1.26	5.46 ± 0.93	8.02 ± 6.85	6.68 ± 5.66	11.21 ± 5.53	10.63 ± 5.86
4	6.08 ± 1.41	6.94 ± 0.77	9.15 ± 7.17	7.25 ± 5.49	12.85 ± 4.89	11.89 ± 6.62
5	8.72 ± 6.08	7.92 ± 4.73	7.57 ± 5.92	6.45 ± 5.77	14.08 ± 6.86	12.71 ± 6.1
6	5.7 ± 1.15	7.37 ± 1.87	8.25 ± 6.14	8.56 ± 5.01	12.16 ± 6.17	12.21 ± 5.87
7	5.9 ± 1.57	4.93 ± 0.54	7.12 ± 5.76	7.88 ± 6.47	14.25 ± 6.97	12.67 ± 8.07
8	8.41 ± 1.52	7.53 ± 0.86	8.09 ± 6.4	5.18 ± 3.94	14.99 ± 5.52	12.58 ± 6.11
9	4.82 ± 0.48	5.5 ± 0.24	6.99 ± 5.69	9.16 ± 7.06	13.89 ± 6.69	12.99 ± 6.83
10	7.08 ± 1.35	7.3 ± 0.85	8.44 ± 6.42	5.7 ± 3.3	15.48 ± 8.07	10.09 ± 7.33
Mean	6.87 ± 1.95	6.93 ± 1.29	7.85 ± 6.19	7.13 ± 5.68	13.37 ± 6.25	11.84 ± 6.44

Note that the registration targets are not related between experiments.

Validation phantom

A total of 748 re-projected divot positions were used to evaluate TRE and 164 for FRE. The average TRE across registrations was found to be µ = 6.87 mm, with standard deviation of σ = 1.95 mm. FRE resulted in µ = 6.93 mm, with standard deviation of σ = 1.29 mm.

Liver phantom

A total of 1403 re-projected metallic marker centroid positions were used to evaluate TRE and 450 for FRE. The average TRE across registrations was found to be µ = 7.85 mm, with standard deviation of σ = 6.19 mm. FRE resulted in µ=7.13 mm, with standard deviation of σ = 5.68 mm.

In vivo model

A total of 3074 re-projected cauterization points were used in order to compute TRE and 1137 for FRE. The average TRE across registrations was found to be µ = 13.37 mm, with standard deviation of σ = 6.25 mm. FRE resulted in µ = 11.84 mm, with standard deviation of σ = 6.44 mm. FLE, computed as the rms between the inserted fiducials and the cauterization points annotated was, on average, 16.40 mm.

Because of non-normality of parts of the data, six Kruskal-Wallis tests were conducted in SPSS (IBM, Armonk, NY) to compare the TRE and FRE across registration procedures for each experiment. Significant differences (p < .05) between registration procedure accuracies were found for the validation phantom, for both TRE and FRE, χ²(9) = 290.06, p = 3.34E⁻⁵⁷ and χ²(9) = 78.69, p = 2.93E⁻¹³, respectively. No significant differences were found for the liver phantom TRE but for the FRE χ²(9) = 21.28, p = .011. Significant differences between registration procedures were found for TRE and FRE the in vivo experiment, χ²(9) = 152.04, p = 3.33E⁻²⁸ and χ²(9) = 29.96, p = .00045, respectively.

Successively, an additional Kruskal-Wallis test (because of non-normality of the data) showed that there was a statistically significant difference between the different experiments in terms of TRE, χ²(2) = 1223.61, p = 1.976E⁻²⁶⁶, with a mean rank score of 1666.21 for the validation phantom, 1781.41 for the liver phantom and 3222.93 for the in vivo study. FRE also revealed significant differences, χ²(2) = 254.72, p = 4.88E⁻⁵⁶, with a mean rank score of 668.46 for the validation phantom, 595.34 for the liver phantom and 1017.01 for the in vivo study.

Discussion

Based on the results obtained from the three experiments, the accuracy with which a position is annotated on a CT scan volume affects both the TRE and FRE in AR. This is inferred from the comparison between the results in the validation phantom and liver phantom, where points are measured or automatically clustered, and the results on the in vivo model, which depends on human interaction through annotation.

It is noteworthy to mention that, although the average TREs and FREs of the validation and the liver phantoms are very similar, TRE varies significantly between registration procedure in the validation phantom but not for the liver phantom. This can be explained by the differences in sizes between the volumes to be registered. The validation phantom presents a volume of 4320 cm³, whereas the patient-specific phantom is 1882 cm³ (the in vivo liver was 2393 cm³). Moreover, the validation phantom targets are partially symmetrical, and some positions are almost collinear (as can be seen in Figure 5). Sampling of five positions across the volume was performed randomly in each of the experiments. Therefore, depending on the positions of the targets, a larger volume will be more affected than a smaller volume. Based on these results, for registration of large volumes, it is preferable to use a larger number of targets to compute the image-to-patient registration and possibly better spatial disposition of these targets with respect to the volume (such as ensuring non-collinearity between registration targets or closeness to areas of interest such as tumours).

The statistical differences between TRE and FRE in the in vivo study may depend on the FLE for the cauterization markers used to compute image-to-patient registration. If, e.g. the accuracy of annotating a position were to be inexact, this could greatly affect the image-to-patient registration matrix, causing an increase in both TRE and FRE (meaning, a decrease in AR accuracy). In order to mitigate the effect of sampling error in PBR, the authors propose using intra-operative fiducials on the liver surface, which could be removed post-surgery (or made of biocompatible/biodegradable material). These fiducials could be detected in the intra-operative CT/MRI scan and used to perform PBR. This would probably reduce the sampling error to the accuracy evaluated in the liver phantom testing (it would greatly reduce FLE) and may greatly improve the accuracy of the AR re-projection. However, this may prolong and complicate the surgical procedure.

Within this study, targets located on the centre of the liver surface were the most complicated to be annotated correctly and should therefore be avoided. Alternatively, using targets on the edges of the liver could also reduce FLE (if the parenchyma were rigid enough during sampling). Positions that are also stable in the liver include intersections of the parenchyma with major blood vessels (such as the portal vein) or bifurcations of blood vessel structures. These structures can serve as solid registration targets; however, they can only be sampled from within the liver after resection.

Some limitations to this study include the fact that the scan used to perform AR was intra-operative, which is currently not within the surgical workflow for most hospitals. Pre-operative imaging does not account for intra-operative deformations, such as pneumoperitoneum (static deformations); hence, inaccuracies due to the non-rigidity of the liver would be present in the current surgical workflow, as mentioned by Thompson et al. [19,22]. However, intra-operative CT and MRI scanners are under production for Hybrid OR suites, and show very large increases in AR accuracy and alternative methods of acquiring intra-operative data (such as stereo surface reconstruction [23] or laparoscopic US [24]) could be used. Alternatives to intra-operative imaging, such as the use of biomechanical modelling are a valid solution, as described in [25–28], but may be affected by the need of estimating of viscoelastic properties of soft tissue and boundary conditions. If elastography were to be used to characterize the viscoelasticity of the tissue, and the boundary conditions were known for each patient, biomechanical modelling could be used to account for deformations.

It would be interesting to study the effect of spatial disposition of registration targets across volumes of variable size, to validate the assumption that the volume changes caused the differences in accuracy between the validation phantom and the other experiments. Moreover, another limitation to the study is that the cauterization marks were performed on the liver surface and might not represent fully the error in the depth axis. However, the marks were performed across the liver as deep as possible towards the diaphragm, which allowed us to calculate TRE and FRE for positions at various depths, though not as much as a blood vessel or a tumour deep within the tissue. Another limitation is the fact that both camera calibrations and hand-eye calibrations are performed as they would be in a surgical environment, without thorough refinement of the calibration procedures. Furthermore, the annotations for PBR could have been performed by multiple surgeons to further validate the hypothesis that annotations (FLE) could cause significant differences in terms of registration accuracy for AR.

The investigated inaccuracy in terms of TRE and FRE is larger than that accepted by surgeons. However, the use of AR, complemented with intra-operative US, could still be useful for visualization of the structures in the resection field. Thus, it might help surgeons better understand spatial distribution of anatomical structures and lead to safer surgery. Even with current quality, where resection lines cannot be followed blindly due to the system’s inaccuracy, there is still clinical value in the use of this AR system, especially for spatial understanding.

Conclusions

This study aims to show that accuracy in sampling registration targets can contribute to decreases/increases in the accuracy of the AR through PBR. The laparoscope camera, CT scanner, optical markers and algorithms used were consistent throughout all experiments, the only difference was the volumes to be registered. Results show that the accuracy through PBR can change based on the accuracy in sampling the positions to compute image-to-patient registration, possibly also the size of the volume to be registered and spatial disposition of registration targets used to perform image-to-patient registration.

The overall accuracy for the AR in terms of TRE for the in vivo model was around 13 mm, and 11 mm in terms of FRE. However, the results also say that the TRE AR accuracy can worsen greatly based on the registration procedure, if the targets are based on the surgeon’s annotations (TRE can result to be larger than a centimeter, as shown in Table 1 and validated in other studies [22,18]).

The main indication for liver resection is CRLM in the western world. If the proposed solution were to be used for clinical use in CRLM, the error for the AR should not exceed 6 mm of inaccuracy according to the authors. The reason is that 6 mm are acceptable for surgeons because the safety margin for CRLM (resection margin) is 1–3 mm [29], and normally, surgery is planned with a 1 cm margin. Therefore, a planned resection line with 1 cm of margin using an AR with 6 mm of error, will allow 3 mm of space in addition to the 1 mm safety margin.

Based on the results of this study, it is necessary to improve the image-to-patient registration, possibly with the use of user-independent fiducials for registration and a smaller volume to be registered. Overall, improvements of this AR system are necessary; however, we have proven that better sampling accuracy can lead to much better accuracies, which will allow AR to be of use for surgery.

Ethical approvals

All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted. This article does not contain any studies with human participants performed by any of the authors.

Acknowledgements

The authors would like to express their gratitude to all the OR team at ‘The Intervention Centre’, Oslo University Hospital.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by H2020-MSCA-ITN Marie Skodowska-Curie Actions, Innovative Training Networks (ITN) 2016 GA EU project number 722068 High Performance Soft Tissue Navigation (HiPerNav).