Abstract
The aim of this paper is to analyze the variations in registration accuracy for computer-assisted surgical navigation using three different modes of registration, in order to explore the behavior of random error, and to highlight the precision of neuronavigation as a concept distinct from accuracy.
The operational accuracy of three different registration modes (bone fiducials, scalp adhesive fiducials and an auto-registration mask) was evaluated in a total of 20 fresh cadaveric heads. The precision of the neuronavigation system was then assessed by evaluating the variation in the accuracy measurements associated with each registration mode. The coefficient of variation was employed to quantify the degree of variation in the attained accuracy using the following formula: Coefficient of variation = standard deviation/mean * 100.
For external targets, the precision of the neuronavigation system was greatest with mask registration (43.75 and 51.41 for anterior and posterior external targets, respectively) and lowest with bone registration (65.30 and 67.17 for anterior and posterior external targets, respectively). For internal targets, the precision of the neuronavigation system was greatest with bone registration (47.69 and 42.6 for anterior and posterior internal targets, respectively) and lowest with mask registration (62.9 and 58.67 for anterior and posterior internal targets, respectively).
The precision (reproducibility) of the neuronavigation system is another important quantity besides accuracy that characterizes the performance of the system. Understanding both of these quantities for a given registration mode enhances the use of a neuronavigation system in neurosurgery.
Introduction
Like all devices that measure physical quantities, the performance of any neuronavigation system should be evaluated in terms of both its accuracy and its precision. The reliability or precision of a neuronavigation system refers to the ability of the system to produce the same values repeatedly Citation[1], Citation[2]. The accuracy of a neuronavigation system refers to the agreement between the observed value and the true value or, in other words, the extent to which the location of a target as detected by the system matches the real location of that target Citation[1], Citation[2]. Nevertheless, almost all the published studies of neuronavigation performance have focused on accuracy Citation[3–8]. Surprisingly, the precision of neuronavigation systems has been neglected or treated as if it were synonymous with accuracy; indeed, only a few papers distinguish between precision and accuracy Citation[8–11].
The aim of this study was to evaluate the precision of a specific neuronavigation system (iNtellect Cranial, Stryker Instruments, Kalamazoo, MI) by using three different registration modes, namely bone fiducials, scalp fiducials, and an auto-registration mask ().
Figure 1. The three different registration modes used in the study: (a) the auto-registration mask; (b) scalp fiducials; and (c) bone fiducials. In (d) the pointer is seen touching one of the external targets.
We have previously reported the overall application accuracy of this neuronavigation system using different methods of registration; in this paper, we highlight the difference between the precision and the accuracy, using the same data bank and methodological approach reported previously Citation[12].
Materials and methods
Specimen preparation
Twenty fresh cadaver heads were used in the study. Each specimen was fitted with both implanted bone fiducials and adhesive fiducials. A CT scan (Siemens, Erlangen, Germany) was then performed using the following protocol: slice thickness 1 mm; contiguous non-overlapping slices; gantry setting 0°; scan window diameter 225 mm; pixel size >0.44 × 0.44).
Fiducials
Each specimen was implanted with ten 1.0 mm × 4.0 mm titanium micro-screws (Synthes Low Profile Neuro System, Synthes USA, West Chester, PA). Six of these micro-screws were implanted in the skull, one on each side of the frontal, parietal and occipital bones.
For bone registration, three skull micro-screws (two parietal and one either frontal or occipital) were selected as bone fiducials for registration purposes, and the other three screws were used as external targets. None of these screws was used both for registration and as an external target. When the mask or scalp fiducial registration was used, four skull micro-screws were used as external targets.
In all the registrations (whether using bone fiducials, the auto-registration mask or scalp fiducials) the frontally implanted micro-screws were designated as anterior external targets and the occipital micro-screws as posterior external targets.
The other four micro-screws were implanted on the skull base endonasally using an endoscope to function as internal targets (). Two of these skull-base micro-screws were implanted in the planum (as anterior internal targets) and two in the clivus (as posterior internal targets).
Figure 2. (a) Endoscopic view showing one of the internal targets being placed using endoscope guidance. (b) Endoscopic view showing the final location of the implanted internal targets (the planum and clivus).
In addition to the bone fiducials, each specimen was also fitted with six adhesive fiducials to serve as scalp fiducials. Four to six of these scalp fiducials were randomly selected for registration.
Image guidance system
A Stryker Navigation System was used (Stryker Instruments, Kalamazoo, MI). The system includes iNtellect Cranial planning software, a tracking camera on which three optical sensors are mounted, a patient tracker, and a pointer.
Mask registration device
The auto-registration mask (Stryker Leibinger GmbH & Co. KG, Freiburg, Germany) () is an adhesive, marker-free mask fitted with light-emitting diodes (LEDs) which is applied to the patient's face at the time of the surgical procedure.
The mask can be used as a point-based registration system if the subject is scanned with the mask in place, or as a surface-based registration system if the mask is applied after scanning. In this study we applied the mask after the scanning process and immediately prior to the procedure, so it was used as a surface-based registration system.
Registration
Two registration metrics were evaluated: the fiducial (or surface) registration error and the target registration error.
Fiducial (or surface) registration error (FRE)
This metric represents the error between the alignment achieved by the neuronavigation image dataset and the real anatomical space of the patient Citation[3]. The FRE is given in mm ± RMS (root mean square) Citation[3]. Its value is generated by the navigation software system after the fiducial (surface) registration process is completed. An effort was made to attain an FRE value of <1 mm.
All registrations were performed on specimens held in position by a Mayfield pin fixation device. Each specimen was registered using three different registration modes: scalp fiducial registration, bone-implanted fiducial registration, and mask auto-registration. Each registration mode was completed five times by a team of five operators for a total of 15 registrations for each specimen. Each bone registration was accomplished by using three skull micro-screws; scalp registration was accomplished using a minimum of four scalp fiducials; and mask auto-registration was accomplished by allowing the infrared camera to visualize a minimum of 28 LEDs.
Target registration error (operational accuracy) (TRE)
In this phase of the study, the accuracy of the neuronavigation system was measured indirectly by calculating the difference between the virtual and the true location of a target. This was done by using three or four skull micro-screws as external targets (three external targets when using bone registration and four external targets when using the scalp/mask registration algorithms) and four skull base micro-screws as internal targets Citation[12]. Two sets of coordinates were defined by using the neuronavigation system. The first set of coordinates consisted of the target coordinates as set by the neuronavigation system, and the value of this set of coordinates was read by placing the cross-hair on the target shown in the image space (virtual target). The second set of coordinates consisted of the true target coordinates as determined in the real anatomical space of the specimen. This second set of coordinates was read when the navigation probe touched the target shown in the specimen space (real target) (). Finally, the Euclidean distance between the real and virtual coordinates was calculated using the formula
Precision measurement
The precision (also referred to as the reliability or reproducibility) of the neuronavigation system was assessed by evaluating the variation in the accuracy measurements associated with each registration mode. The coefficient of variation was employed to measure the degree of variation in relation to the mean by using the following formula:
Statistical analysis
The commercially available SPSS software (Version 16.0, SPSS Inc., Chicago, IL) was used for statistical analysis. The target registration errors (operational accuracy) were compared using a paired t-test. In contrast, the fiducials (or surface) registration errors were compared using an unpaired two-tail Student's t-test. A P-value of <0.05 was considered significant.
Results
Fiducial (or surface) registration error and precision
Accuracy
The mean neuronavigation error reported by the neuronavigational software as an RMS value was 0.62 mm (SD 0.14) for scalp fiducials and 0.64 mm (SD 0.17) for bone fiducials. For the auto-registration mask, the mean surface registration error (SRE) was 0.69 mm (SD 0.11) ().
Table I. Comparison of fiducial (or surface) registration error as calculated by the neuronavigation system using different registration modalities. The associated coefficient of variation is also shown.
No statistically significant difference was found between the RMS values for scalp fiducials and bone fiducials (P > 0.05), while the RMS value for bone fiducials or scalp fiducials was significantly lower than the SRE of the auto-registration mask (P < 0.05 and P < 0.001, respectively, for bone and scalp fiducials).
Precision
The coefficient of variation of the errors recorded by the neuronavigation system during this phase of registration was 16.32 for the mask, 22.46 for scalp fiducials, and 27.296 for bone fiducials (). In other words, the precision (reproducibility) of the neuronavigation system for this phase of the registration is greatest when the auto-registration mask is used.
Target registration accuracy and precision
Accuracy
The mean target registration errors in millimeters (mm) for external anterior targets were 1.96, 3.12 and 3.20 for bone fiducials, scalp fiducials, and the auto-registration mask, respectively. For external posterior targets, the mean TREs were 1.95, 3.24 and 3.19 mm for bone fiducials, scalp fiducials, and the auto-registration mask, respectively. In the case of internal anterior targets, the mean TREs were 2.60, 3.65 and 2.16 mm for bone fiducials, scalp fiducials, and the auto-registration mask, respectively, while the mean TREs for internal posterior targets were 2.91, 3.83 and 2.41 mm for bone fiducials, scalp fiducials, and the auto-registration mask, respectively ().
Table II. Operational accuracy of the navigation system in reaching targets using different registration modalities, and the associated coefficient of variation.
For external targets (both anterior and posterior), bone fiducial registration was shown to be linked to better operational accuracy than that associated with scalp fiducials and the auto-registration mask (P < 0.001). The slight difference between the accuracy with scalp fiducials and that with the auto-registration mask was not statistically significant (P > 0.05).
For internal targets, auto-registration mask registration was shown to be linked to better operational accuracy when compared to the operational accuracy associated with bone fiducial registration (P < 0.05 for anterior targets and P < 0.01 for posterior targets) or scalp fiducial registration (P < 0.001). Also, for internal targets, the operational accuracy associated with bone fiducial registration was significantly higher than that associated with scalp fiducial registration (P < 0.001) ().
Precision
For external targets, the precision of the neuronavigation system was greatest with mask registration (43.75 and 51.41 for anterior and posterior external targets, respectively) and lowest with bone registration (65.30 and 67.17 for anterior and posterior external targets, respectively).
For internal targets, the precision of the neuronavigation system was greatest with bone registration (47.69 and 42.6 for anterior and posterior internal targets, respectively) and lowest with mask registration (62.9 and 58.67 for anterior and posterior internal targets, respectively) ().
Discussion
Accuracy versus precision
The total error (or total uncertainty) associated with the registration processes of a neuronavigation system incorporates both systemic error and random error.
Systemic error is defined as that part of the overall error that remains constant or varies in a predictable way; it represents the mean of measurements subtracted from the true value. Random error typically arises from unpredictable variations of influence quantities in repeated observations of the measurement Citation[13–19]. The majority of neuronavigation system studies focus on the systemic error, which is connected to the neuronavigation overall accuracy, while the random error, which is related to the precision of the system, has rarely been addressed or highlighted Citation[9], Citation[10], Citation[11], Citation[17].
Precision is the qualitative term for the closeness of agreement between independent test results obtained under stipulated conditions. It depends only on the distribution of random errors, and does not relate to the true value or the accepted reference value. Rather, it describes the strength of the measuring instrument and its degree of freedom from random error. Depending on the experimental context, the precision could be synonymous with the repeatability or reproducibility Citation[3–18].
Trueness is the qualitative term indicating the closeness of agreement between the value of the test results and an accepted reference value. It is usually expressed in terms of bias (the total systematic error). According to the Joint Committee for Guides in Metrology, trueness (bias) is denoted as accuracy Citation[18]. However, the International Organization for Standardization defines accuracy as consisting of trueness and precision Citation[19]. Nevertheless, this same organization emphasizes that accuracy is not equivalent to precision Citation[19].
Significance of the present study
The aim of this study was to assess the effect of registration mode on the precision of the investigated neuronavigation system and to assess the relationship between the accuracy and the precision of this system.
We consider this to be important because the random error (precision) is difficult to control and predict, meaning it could be the real Achilles’ heel of using a neuronavigation system in neurosurgery. Thus, having an idea of the possible behavior of the random error in different situations (i.e., different registration modes as studied in this paper) may make the user more cautious in taking steps that could decrease the impact of this type of error. We also think that understanding both sources of error (uncertainty) that can occur during both registration processes (fiducial and target) is important to achieving a deeper understanding of the neuronavigation system and its potential problems.
In addition, it is important to stress the difference between fiducial and target registration error, because many neuronavigation users equate the number given to them by the neuronavigation software after the registration process is completed (the FRE) with operational accuracy, and the manufacturers certainly do not highlight the difference between FRE and TRE.
Analysis of results
There is a variation in the accuracy that is achieved with a neuronavigation system. In other words, the same amount of accuracy cannot be expected for every operation. This means that it is not always the case, for example, that the neuronavigation system is more accurate for external targets when using bone registration rather than mask or adhesive registration and vice versa.
The neuronavigation operational accuracy that can be achieved by a specific registration method changes according to the target location (for internal targets the accuracy of the mask is higher than that of bone fiducials, whereas bone fiducial registration is associated with better accuracy for external targets). This can be explained on the basis of the distance of the target from the reference frame. Therefore, with regard to the mask, the mask itself is the reference frame, and in the registration process the software calculates the location of each LED position to create a reference “sphere”. Since the external targets are outside this sphere, the accuracy is diminished for these targets but increased for internal targets that are inside the sphere. On the other hand, with bone fiducials, the patient tracker is the reference frame and is closer to the external targets than to the internal targets.
The precision of the neuronavigation system changes according to the target location and the method of registration (for internal targets the precision of the mask is lower than that for bone fiducials, whereas the opposite is true for external targets).
It is puzzling that in our experiment the precision and accuracy diverged. For example, the mask results for internal targets are close to the real values, but are more scattered amongst themselves than the results for the bone fiducials for the same targets, making the mask registration more accurate but less precise. The opposite is true for external targets. In our experiments, when the systemic error decreased, the random error became more obvious (problematic) and weighed more heavily on the overall uncertainty of the neuronavigation system. In this situation, continuous external reference-based verification of the neuronavigation performance may help in reducing the impact of this type of error. In this regard, intraoperative imaging with defined landmarks (even artificial ones, such as micro-screws) included in the imaged volume could provide this type of feedback.
The sources of the random error could be explained by inter-observer and intra-observer variations, as well as by anatomical variations between the specimens or by variations in the technical set-up of the system.
Practical meaning of the results
The main practical application of our findings is straightforward: The neuronavigation system should not be trusted blindly. Irrespective of the registration mode used and the recorded system metric, the operational accuracy should be always verified according to anatomical landmarks and the logical judgment of the neurosurgeon. Again, intraoperative imaging may be of great help in creating an image dataset that can be used to provide this type of feedback.
Also, any neuronavigation system should ideally be evaluated on the basis of both accuracy and precision, and these different quantities should be clearly known to the user. Indeed, some users may privilege one over the other.
The precision of a neuronavigation system is a different quantity that characterizes the performance of the system, and has a different meaning from the accuracy of the system. The ideal system should be one that combines good accuracy and good precision.
Limitations of the present study and future research opportunities
Our study was a cadaver study, and suffers from all the limitations of such studies. However, we feel that the experimental conditions used in the study are similar to those encountered in clinical practice when a CT data set is used for navigation purposes.
The study was performed with CT imaging only, because this avoids the image distortion associated with MRI. However, of course, in real-world situations for cranial cases, MRI imaging is used with or without CT, which could impart an additional source of systematic error Citation[3], Citation[12], Citation[20].
Precision may be further segmented into two separate concepts: the reproducible conditions of precision (the variation resulting when the test is conducted using different instruments and operators), and precision under repeatable conditions (the variation resulting when the test is conducted using the same instruments and the same operator) Citation[14]. The design of our experiment addresses the reproducible conditions of precision; precision under repeatable conditions should be addressed in future studies to look deeper into the concept of precision.
Finally, in this study we evaluated the precision following three different registration methods (mask auto-registration, bone-implanted fiducials, and scalp fiducials). It would be worthwhile to see if our findings are reproducible using other types of registration, such as ultrasound and laser surface registration Citation[21–23].
Conclusions
The precision (reproducibility) of a neuronavigation system is another important quantity that characterizes the performance of the system, and should be included as a separate quantity – in addition to the system accuracy – when evaluating a neuronavigation system.
Our study demonstrates that when the systemic error is decreased, the random error infringes more heavily on the overall uncertainty of the neuronavigation system.
Declaration of interest: This research was partially funded through a research grant from Stryker Leibinger GmbH & Co. KG, Freiburg, Germany.
References
- Taylor JR. An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements, Second. University Science Books, Sausalito, CA 1997
- Morris AS. Measurement and Instrumentation Principles. Butterworth-Heinemann, Boston 2001
- Eggers G, Mühling J, Marmulla R. Image-to-patient registration techniques in head surgery. Int J Oral Maxillofac Surg 2006; 35: 1081–1095
- Woerdeman PA, Willems PW, Noordmans HJ, Tulleken CA, van der Sprenkel JW. Application accuracy in frameless image-guided neurosurgery: A comparison study of three patient-to-image registration methods. J Neurosurg 2007; 106: 1012–1016
- Mascott CR, Sol JC, Bousquet P, Lagarrigue J, Lazorthes Y, Lauwers-Cances V. Quantification of true in vivo (application) accuracy in cranial image-guided surgery: Influence of mode of patient registration. Neurosurgery 2006; 59: ONS146–ONS156
- Wolfsberger S, Rössler K, Regatschnig R, Ungersböck K. Anatomical landmarks for image registration in frameless stereotactic neuronavigation. Neurosurg Rev 2002; 25: 68–72
- Raabe A, Krishnan R, Wolff R, Hermann E, Zimmermann M, Seifert V. Laser surface scanning for patient registration in intracranial image-guided surgery. Neurosurgery 2002; 50: 797–802
- Giese H, Hoffmann KT, Winkelmann A, Stockhammer F, Jallo GI, Thomale UW. Precision of navigated stereotactic probe implantation into the brainstem. J Neurosurg Pediatr 2010; 5: 350–359
- Benardete EA, Leonard MA, Weiner HL. Comparison of frameless stereotactic systems: Accuracy, precision, and applications. Neurosurgery 2001; 49: 1409–1415
- Bjartmarz H, Rehncrona S. Comparison of accuracy and precision between frame-based and frameless stereotactic navigation for deep brain stimulation electrode implantation. Stereotact Funct Neurosurg 2007; 85: 235–242
- Pfisterer WK, Papadopoulos S, Drumm DA, Smith K, Preul MC. Fiducial versus nonfiducial neuronavigation registration assessment and considerations of accuracy. Neurosurgery 2008; 62: 201–208
- Makiese O, Pillai P, Salma A, Sammet S, Ammirati M. Accuracy validation in a cadaver model of cranial neuronavigation using a surface autoregistration mask. Neurosurgery 2010; 67: ONS85–ONS90
- Engelke K, Gluer CC. Quality and performance measures in bone densitometry. Part 1: Errors and diagnosis. Osteoporos Int 2006; 17: 1283–1292
- Hofer M, Strauß G, Koulechov K, Dietz A, Definition of accuracy and precision - evaluating CAS-systems. In: Lemke HU, Inamura K, Doi K, Vannier MW, Farman AG, editors. Computer Assisted Radiology and Surgery: Proceedings of the 19th International Congress and Exhibition (CARS 2005), Berlin, Germany, June 2005. Amsterdam: Elsevier; 2005. pp 548–552
- Kirkup L, Frenkel B. An introduction to uncertainty in measurement using the GUM (guide to the expression of uncertainty in measurement). Cambridge University Press, Cambridge, UK 2006
- Menditto A, Patriarca M, Magnusson B. Understanding the meaning of accuracy, trueness and precision. Accred Qual Assur 2007; 12: 45–47
- Strauss G, Hofer M, Korb W, Trantakis C, Winkler D, Burgert O, Schulz T, Dietz A, Meixenberger J, Koulechov K. [Accuracy and precision in the evaluation of computer assisted surgical systems. A definition] [In German]. HNO 2006; 54: 78–84
- JCGM 100:2008. Evaluation of measurement data – Guide to the expression of uncertainty in measurement. Joint Committee for Guides in Metrology; 2008
- ISO 5725-1. Accuracy (trueness and precision) of measurement methods and results. International Organization for Standardization, Geneva 1994
- Golfinos JG, Fitzpatrick BC, Smith LR, Spetzler RF. Clinical use of a frameless stereotactic arm: Results of 325 cases. J Neurosurg 1995; 83: 197–205
- Schlaier J, Warnat J, Brawanski A. Registration accuracy and practicability of laser-directed surface matching. Comput Aided Surg 2002; 7: 284–290
- Schicho K, Figl M, Seemann R, Donat M, Pretterklieber ML, Birkfellner W, Reichwein A, Wanschitz F, Kainberger F, Bergmann H, et al. Comparison of laser surface scanning and fiducial marker-based registration in frameless stereotaxy. Technical note. J Neurosurg 2007; 106: 704–709
- Li QH, Zamorano L, Pandya A, Perez R, Gong J, Diaz F. The application accuracy of the NeuroMate robot - A quantitative comparison with frameless and frame-based surgical localization systems. Comput Aided Surg 2002; 7: 90–98