A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Figures & data

Table 1. Past vs. present: comparison of MR IGS in or.

Study period	1993–2011	2017–2023
Papers published	467	965
Average publish/year	24.6	137.9
Papers went to OR	15	79
Number of cases studied	70	1592
Percentage of OR applications	3.2%	8.2%
Average OR application/year	0.2%	1.2%

Figure 1. PRISMA chart—inclusion and exclusion criteria.

Table 2. Base exclusion reason categories and counts (2022–2024).

Base exclusion reason	# Of studies
Training and planning	73
Method development	26
Phantom study	12
Cadaver study	3
Patient education	3
VR for diagnosis	1

Figure 2. Distribution of imaging modalities in different studies.

Figure 3. Different types of visualized data and processing. (A) CT slice image visualized in corner of Google Glass [25]. (B) Contours and transparency used in BrainLab’s HUD showing lesion (yellow), the carotid arteries (blue), the chiasm (yellow), and the optic nerves (orange) [26]. (C) Wireframe mesh rendering of uterus with tumor shown in yellow [27]. (D) Surface rendering of spine, vasculature, and tumor anatomy as seen through the HoloLens [28]. (E) Volume rendering of vessels in neurovascular surgery with colors representing venous/arterial flow [29].

Figure 4. Distribution of visualization processing in different studies.

Table 3. Categorization of the papers based on the settings used for data visualization.

Perception location	# Of studies	Percentage
Surgical field (SF)	50	63.3%
SF and monitor	11	13.9%
Monitor	9	11.4%
Field of view (FOV)	4	5.1%
SF + FOV + Monitor	2	2.5%
FOV + Monitor	1	1.3%
SF (with offset)	1	1.3%
SF + FOV	1	1.3%

Data (surgical field includes patient, HMD, microscope).

Figure 5. Examples of display types used for MR IGS. (A) Google Glass allows for the projection of simple images/text in the corner of the glasses [25]. (B) The HoloLens HMD displays holograms of preoperative plans and anatomy in the surgical field of view in an abdominal cancer surgery case [28]. (C) An iPad device used to show an AR view of the patient anatomy in neurosurgery [42]. (D) The monitor of the neuronavigation system is used to show the AR view [29]. (E) The surgical microscope’s HUD can be used to show the AR view in the oculars [17].

Figure 6. Display devices utilized to view visualized data.

Figure 7. Type of surgery: comparison between the 2013 review [2] and the current review for the count of papers for each type of surgery.

Figure 8. Surgical domains: the chart displays surgical domains using generic categories, with ‘spine surgery’ falling under ‘neurosurgery’ as a superclass.

Table 4. The number of cases studied across various types of surgeries (other includes plastic, hepatectomy, gastrointestinal, gynecology, colorectal, lumpectomy).

Type of surgery	# Of studies	# Of cases
Orthopedic	18	440 (27.6%)
Neurosurgery	16	369 (23.2%)
Endoscopic/Laparoscopic	6	340 (21.4%)
Thoracic	3	145 (9.1%)
Craniomaxillofacial	15	77 (4.8%)
Cholecystectomy	1	58 (3.6%)
Urology	3	55 (3.5%)
Vascular	1	20 (1.3%)
Prostatectomy	1	20 (1.3%)
Thyroidectomy	3	19 (1.2%)
Laparotomy	2	18 (1.1%)
Other	10	31 (1.9%)
Total	79	1592

Table 5. Included studies: evaluated or validated components (≥15 patients).

Reference	Clinical context	# Cases	Evaluation
			Quantitative	Qualitative
Butler et al. [23]	Orthopedic	164	Time (pedicle screw placement)	–
Li et al. [41]	Thoracic	142	Time, blood loss, hospital stay	Feasibility of the system in improving surgical visualization and precision
Zeiger et al. [22]	Endoscopic/Laparoscopic	134	–	Minimizing surgical risks and optimizing outcomes
Linxweiler et al. [69]	Endoscopic/Laparoscopic	100	Time	–
Liu et al. [9]	Neurosurgery	77	Rate of resection, hospital stay, surgery time	Surgical outcome, Patient’s quality of life (KPS)
Diana et al. [75]	Cholecystectomy	58	Time, identification of anatomy	Image quality, visualization
Aoyama et al. [76]	Orthopedic	49	Accuracy	–
Bopp et al. [26]	Neurosurgery	48	Time, accuracy	–
Carl et al. [36]	Endoscopic/Laparoscopic	47	Accuracy, radiation dosage	Feasibility of clinical setting
Edström et al. [33]	Orthopedic	44	–	Feasibility and surgical outcome
Carl et al. [16]	Neurosurgery	42	Accuracy, complications, radiation dosage	Workflow, benefits of system use
Schwam et al. [21]	Endoscopic/Laparoscopic	40	–	Ease of use, utility
Kim et al. [19]	Neurosurgery	34	–	Potential positive impact of the system in brain tumor surgery
Bhatt et al. [24]	Orthopedic	32	Accuracy	Post-operative surgical complications
Borgmann et al. [25]	Urology	31	–	Feasibility and safety, distraction, intra- and post-operative complications
Incekara et al. [77]	Neurosurgery	25	Time, tumor localization	Ergonomics
Guo et al. [55]	Orthopedic	21	Time, blood loss, complication, and Hardegger function	Post-operative complaints
Elmi-Terander et al. [34]	Orthopedic	20	Time, accuracy (pedicle screw placement)	–
Burström et al. [31]	Orthopedic	20	Accuracy (pedicle screw placement)	–
Porpiglia et al. [74]	Prostatectomy	20	Accuracy (capsular involvement)	Clinical outcomes
Carl et al. [14]	Neurosurgery (vascular)	20	Accuracy, radiation dose, surgical complications	Usability, workflow, benefits of system use
Zhu et al. [78]	Craniomaxillofacial	20	Accuracy	–
Edström et al. [32]	Orthopedic	20	Time (operation, preparation, navigation)	–
Schneider et al. [59]	Endoscopic/Laparoscopic	18	Time (preparation), precision	Registration process, system usability and utility, post-operative complication
Hu et al. [79]	Orthopedic	18	Time (operation), accuracy	Post-operative complaints and pain scale
Rychen et al. [20]	Neurosurgery	15	–	Usefulness and utility
Goto et al. [44]	Neurosurgery	15	–	Efficacy

Borgmann H, Rodríguez Socarrás M, Salem J, et al. Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World J Urol. 2017;35(6):967–972. doi: 10.1007/s00345-016-1956-6.

 PubMed Web of Science ®Google Scholar

Bopp MH, Saß B, Pojskić M, et al. Use of neuronavigation and augmented reality in transsphenoidal pituitary adenoma surgery. J Clin Med. 2022;11(19):5590. doi: 10.3390/jcm11195590.

 PubMed Web of Science ®Google Scholar

Collins T, Pizarro D, Gasparini S, et al. Augmented reality guided laparoscopic surgery of the uterus. IEEE Trans Med Imaging. 2020;40(1):371–380. doi: 10.1109/TMI.2020.3027442.

 PubMed Web of Science ®Google Scholar

Ivanov VM, Krivtsov AM, Strelkov SV, et al. Practical application of augmented/mixed reality technologies in surgery of abdominal cancer patients. J Imaging. 2022;8(7):183. doi: 10.3390/jimaging8070183.

 PubMed Web of Science ®Google Scholar

Kersten-Oertel M, Gerard I, Drouin S, et al. Augmented reality in neurovascular surgery: feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg. 2015;10(11):1823–1836. doi: 10.1007/s11548-015-1163-8.

 PubMed Web of Science ®Google Scholar

Léger É, Reyes J, Drouin S, et al. Marin: an open-source mobile augmented reality interactive neuronavigation system. Int J Comput Assist Radiol Surg. 2020;15(6):1013–1021. doi: 10.1007/s11548-020-02155-6.

 PubMed Web of Science ®Google Scholar

Carl B, Bopp M, Saß B, et al. Microscope-based augmented reality in degenerative spine surgery: initial experience. World Neurosurg. 2019;128:e541–e551. doi: 10.1016/j.wneu.2019.04.192.

 PubMed Web of Science ®Google Scholar

Kersten-Oertel M, Jannin P, Collins DL. The state of the art of visualization in mixed reality image guided surgery. Comput Med Imaging Graph. 2013;37(2):98–112. doi: 10.1016/j.compmedimag.2013.01.009.

 PubMed Web of Science ®Google Scholar

Butler AJ, Colman MW, Lynch J, et al. Augmented reality in minimally invasive spine surgery: early efficiency and complications of percutaneous pedicle screw instrumentation. Spine J. 2023;23(1):27–33. doi: 10.1016/j.spinee.2022.09.008.

 PubMed Web of Science ®Google Scholar

Li C, Zheng B, Yu Q, et al. Augmented reality and 3-dimensional printing technologies for guiding complex thoracoscopic surgery. Ann Thorac Surg. 2021;112(5):1624–1631. doi: 10.1016/j.athoracsur.2020.10.037.

 PubMed Web of Science ®Google Scholar

Zeiger J, Costa A, Bederson J, et al. Use of mixed reality visualization in endoscopic endonasal skull base surgery. Oper Surg. 2019;19(1):43–52. doi: 10.1093/ons/opz355.

 Google Scholar

Linxweiler M, Pillong L, Kopanja D, et al. Augmented reality-enhanced navigation in endoscopic sinus surgery: a prospective, randomized, controlled clinical trial. Laryngosc Investig Otol. 2020;5(4):621–629. doi: 10.1002/lio2.436.

 PubMed Web of Science ®Google Scholar

Liu J, Li X, Leng X, et al. Effect of 3D slicer preoperative planning and intraoperative guidance with mobile phone virtual reality technology on brain glioma surgery. Contrast Media Mol Imaging. 2022;2022. doi: 10.1155/2022/9627663.

 Web of Science ®Google Scholar

Diana M, Soler LP, Agnus VP, et al. Prospective evaluation of precision multimodal gallbladder surgery navigation: virtual reality, near-infrared fluorescence, and x-ray-based intraoperative cholangiography. Ann Surg. 2017;266(5):890–897. doi: 10.1097/SLA.0000000000002400.

 PubMed Web of Science ®Google Scholar

Aoyama R, Anazawa U, Hotta H, et al. Augmented reality device for preoperative marking of spine surgery can improve the accuracy of level identification. Spine Surg Relat Res. 2022;6(3):303–309. doi: 10.22603/ssrr.2021-0168.

 PubMed Web of Science ®Google Scholar

Carl B, Bopp M, Voellger B, et al. Augmented reality in transsphenoidal surgery. World Neurosurg. 2019;125:e873–e883.

 PubMed Web of Science ®Google Scholar

Edström E, Burström G, Persson O, et al. Does augmented reality navigation increase pedicle screw density compared to free-hand technique in deformity surgery? Single surgeon case series of 44 patients. Spine. 2020;45(17):E1085–E1090. doi: 10.1097/BRS.0000000000003518.

 PubMed Web of Science ®Google Scholar

Carl B, Bopp M, Saß B, et al. Spine surgery supported by augmented reality. Spine J. 2020;23:27–33.

 Google Scholar

Schwam ZG, Kaul VF, Bu DD, et al. The utility of augmented reality in lateral skull base surgery: a preliminary report. Am J Otolaryngol. 2021;42(4):102942. doi: 10.1016/j.amjoto.2021.102942.

 PubMed Web of Science ®Google Scholar

Kim HM, Lee MH, Lee TK. Application of augmented reality using neuro-navigation system for individualized brain tumor surgery. J Korean Skull Base Soc. 2022;17(2): 68–74.

 Google Scholar

Bhatt FR, Orosz LD, Tewari A, et al. Augmented reality-assisted spine surgery: an early experience demonstrating safety and accuracy with 218 screws. Global Spine J. 2023;13(7):2047–2052. doi: 10.1177/21925682211069321.

 PubMed Web of Science ®Google Scholar

Incekara F, Smits M, Dirven C, et al. Clinical feasibility of a wearable mixed-reality device in neurosurgery. World Neurosurg. 2018;118:e422–e427. doi: 10.1016/j.wneu.2018.06.208.

 PubMed Web of Science ®Google Scholar

Guo Q, Li X, Tang Y, et al. Augmented reality and three-dimensional plate library-assisted posterior minimally invasive surgery for scapula fracture. Int Orthop. 2022;46(4):875–882. doi: 10.1007/s00264-022-05303-6.

 PubMed Web of Science ®Google Scholar

Elmi-Terander A, Burström G, Nachabé R, et al. Augmented reality navigation with intraoperative 3D imaging vs fluoroscopy-assisted free-hand surgery for spine fixation surgery: a matched-control study comparing accuracy. Sci Rep. 2020;10(1):707. doi: 10.1038/s41598-020-57693-5.

 PubMed Web of Science ®Google Scholar

Burström GM, Nachabe RP, Homan RB, et al. Frameless patient tracking with adhesive optical skin markers for augmented reality surgical navigation in spine surgery. Spine. 2020;45(22):1598–1604. doi: 10.1097/BRS.0000000000003628.

 PubMed Web of Science ®Google Scholar

Porpiglia F, Checcucci E, Amparore D, et al. Augmented-reality robot-assisted radical prostatectomy using hyper-accuracy three-dimensional reconstruction (HA 3DTM) technology: a radiological and pathological study. BJU Int. 2019;123(5):834–845.

 Google Scholar

Carl B, Bopp M, Benescu A, et al. Indocyanine green angiography visualized by augmented reality in aneurysm surgery. World Neurosurg. 2020;142:e307–e315. doi: 10.1016/j.wneu.2020.06.219.

 PubMed Web of Science ®Google Scholar

Zhu M, Liu F, Chai G, et al. A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep. 2017;7(1):42365. doi: 10.1038/srep42365.

 PubMedGoogle Scholar

Edström E, Burström G, Nachabe R, et al. A novel augmented-reality-based surgical navigation system for spine surgery in a hybrid operating room: design, workflow, and clinical applications. Oper Neurosurg. 2020;18(5):496–502. doi: 10.1093/ons/opz236.

 Web of Science ®Google Scholar

Schneider CS, Thompson SJS, Totz JNR, et al. Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study. Surg Endosc. 2020;34(10):4702–4711. doi: 10.1007/s00464-020-07807-x.

 PubMed Web of Science ®Google Scholar

Hu MH, Chiang CC, Wang ML, et al. Clinical feasibility of the augmented reality computer-assisted spine surgery system for percutaneous vertebroplasty. Eur Spine J. 2020;29(7):1590–1596. doi: 10.1007/s00586-020-06417-4.

 PubMed Web of Science ®Google Scholar

Rychen J, Goldberg J, Raabe A, et al. Augmented reality in superficial temporal artery to middle cerebral artery bypass surgery: technical note. Oper Surg. 2019;18(4):444–450. doi: 10.1093/ons/opz176.

 Google Scholar

Goto Y, Kawaguchi A, Inoue Y, et al. Efficacy of a novel augmented reality navigation system using 3D computer graphic modeling in endoscopic transsphenoidal surgery for sellar and parasellar tumors. Cancers. 2023;15(7):2148. doi: 10.3390/cancers15072148.

 PubMed Web of Science ®Google Scholar