A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

References

• Milgram P, Takemura H, Utsumi A, et al. Augmented reality: a class of displays on the reality-virtuality continuum. In: Telemanipulator and telepresence technologies, vol. 2351. SPIE; 1995. p. 1–25.
   Google Scholar
• Kersten-Oertel M, Jannin P, Collins DL. The state of the art of visualization in mixed reality image guided surgery. Comput Med Imaging Graph. 2013;37(2):98–112. doi: 10.1016/j.compmedimag.2013.01.009.
   PubMed Web of Science ®Google Scholar
• Kersten-Oertel M, Jannin P, Collins DL. DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Graph. 2011;18(2):332–352. doi: 10.1109/TVCG.2011.50.
   Web of Science ®Google Scholar
• Móga K, Hölgyesi Á, Zrubka Z, et al. Augmented or mixed reality enhanced head-mounted display navigation for in vivo spine surgery: a systematic review of clinical outcomes. J Clin Med. 2023;12(11):3788. doi: 10.3390/jcm12113788.
   PubMed Web of Science ®Google Scholar
• Han J, Kang HJ, Kim M, et al. Mapping the intellectual structure of research on surgery with mixed reality: bibliometric network analysis (2000–2019). J Biomed Inform. 2020;109:103516. doi: 10.1016/j.jbi.2020.103516.
   PubMed Web of Science ®Google Scholar
• Gorpas D, Phipps J, Bec J, et al. Autofluorescence lifetime augmented reality as a means for real-time robotic surgery guidance in human patients. 2019.
   Google Scholar
• Alfonso-Garcia A, Bec J, Sridharan Weaver S, et al. Real-time augmented reality for delineation of surgical margins during neurosurgery using autofluorescence lifetime contrast. J Biophotonics. 2020;13(1):e201900108. doi: 10.1002/jbio.201900108.
   PubMed Web of Science ®Google Scholar
• Cabrilo I, Bijlenga P, Schaller K. Augmented reality in the surgery of cerebral aneurysms: a technical report. Oper Neurosurg. 2014;10 Suppl 2(2):252–261. doi: 10.1227/NEU.0000000000000328.
   PubMedGoogle Scholar
• Liu J, Li X, Leng X, et al. Effect of 3D slicer preoperative planning and intraoperative guidance with mobile phone virtual reality technology on brain glioma surgery. Contrast Media Mol Imaging. 2022;2022. doi: 10.1155/2022/9627663.
   Web of Science ®Google Scholar
• Simone M, Barile G, Luca RD, et al. Experimental setup employed in the operating room based on virtual and mixed reality: analysis of pros and cons in open abdomen surgery. J Healthc Eng. 2020;2020:8851964. doi: 10.1155/2020/8851964.
   PubMed Web of Science ®Google Scholar
• Lu L, Wang H, Liu P, et al. Applications of mixed reality technology in orthopedics surgery: a pilot study. Front Bioeng Biotechnol. 2022;10:740507. doi: 10.3389/fbioe.2022.740507.
   PubMed Web of Science ®Google Scholar
• Sauer IMM, Queisner M, Tang P, et al. Mixed reality in visceral surgery: development of a suitable workflow and evaluation of intraoperative use-cases. 2017.
   Google Scholar
• Bernard F, Haemmerli J, Zegarek G, et al. Augmented reality–assisted roadmaps during periventricular brain surgery. Neurosurg Focus. 2021;51(2):E4. doi: 10.3171/2021.5.FOCUS21220.
   PubMed Web of Science ®Google Scholar
• Carl B, Bopp M, Benescu A, et al. Indocyanine green angiography visualized by augmented reality in aneurysm surgery. World Neurosurg. 2020;142:e307–e315. doi: 10.1016/j.wneu.2020.06.219.
   PubMed Web of Science ®Google Scholar
• Carl B, Bopp M, Saß B, et al. Implementation of augmented reality support in spine surgery. Eur Spine J. 2019;28(7):1697–1711. doi: 10.1007/s00586-019-05969-4.
   PubMed Web of Science ®Google Scholar
• Carl B, Bopp M, Saß B, et al. Spine surgery supported by augmented reality. Spine J. 2020;23:27–33.
   Google Scholar
• Carl B, Bopp M, Saß B, et al. Microscope-based augmented reality in degenerative spine surgery: initial experience. World Neurosurg. 2019;128:e541–e551. doi: 10.1016/j.wneu.2019.04.192.
   PubMed Web of Science ®Google Scholar
• Carl B, Bopp M, Saß B, et al. Augmented reality in intradural spinal tumor surgery. Acta Neurochir. 2019;161(10):2181–2193. doi: 10.1007/s00701-019-04005-0.
   PubMed Web of Science ®Google Scholar
• Kim HM, Lee MH, Lee TK. Application of augmented reality using neuro-navigation system for individualized brain tumor surgery. J Korean Skull Base Soc. 2022;17(2): 68–74.
   Google Scholar
• Rychen J, Goldberg J, Raabe A, et al. Augmented reality in superficial temporal artery to middle cerebral artery bypass surgery: technical note. Oper Surg. 2019;18(4):444–450. doi: 10.1093/ons/opz176.
   Google Scholar
• Schwam ZG, Kaul VF, Bu DD, et al. The utility of augmented reality in lateral skull base surgery: a preliminary report. Am J Otolaryngol. 2021;42(4):102942. doi: 10.1016/j.amjoto.2021.102942.
   PubMed Web of Science ®Google Scholar
• Zeiger J, Costa A, Bederson J, et al. Use of mixed reality visualization in endoscopic endonasal skull base surgery. Oper Surg. 2019;19(1):43–52. doi: 10.1093/ons/opz355.
   Google Scholar
• Butler AJ, Colman MW, Lynch J, et al. Augmented reality in minimally invasive spine surgery: early efficiency and complications of percutaneous pedicle screw instrumentation. Spine J. 2023;23(1):27–33. doi: 10.1016/j.spinee.2022.09.008.
   PubMed Web of Science ®Google Scholar
• Bhatt FR, Orosz LD, Tewari A, et al. Augmented reality-assisted spine surgery: an early experience demonstrating safety and accuracy with 218 screws. Global Spine J. 2023;13(7):2047–2052. doi: 10.1177/21925682211069321.
   PubMed Web of Science ®Google Scholar
• Borgmann H, Rodríguez Socarrás M, Salem J, et al. Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World J Urol. 2017;35(6):967–972. doi: 10.1007/s00345-016-1956-6.
   PubMed Web of Science ®Google Scholar
• Bopp MH, Saß B, Pojskić M, et al. Use of neuronavigation and augmented reality in transsphenoidal pituitary adenoma surgery. J Clin Med. 2022;11(19):5590. doi: 10.3390/jcm11195590.
   PubMed Web of Science ®Google Scholar
• Collins T, Pizarro D, Gasparini S, et al. Augmented reality guided laparoscopic surgery of the uterus. IEEE Trans Med Imaging. 2020;40(1):371–380. doi: 10.1109/TMI.2020.3027442.
   PubMed Web of Science ®Google Scholar
• Ivanov VM, Krivtsov AM, Strelkov SV, et al. Practical application of augmented/mixed reality technologies in surgery of abdominal cancer patients. J Imaging. 2022;8(7):183. doi: 10.3390/jimaging8070183.
   PubMed Web of Science ®Google Scholar
• Kersten-Oertel M, Gerard I, Drouin S, et al. Augmented reality in neurovascular surgery: feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg. 2015;10(11):1823–1836. doi: 10.1007/s11548-015-1163-8.
   PubMed Web of Science ®Google Scholar
• Pratt P, Ives M, Lawton G, et al. Through the Hololens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. Eur Radiol Exp. 2018;2(1):2. doi: 10.1186/s41747-017-0033-2.
   PubMedGoogle Scholar
• Burström GM, Nachabe RP, Homan RB, et al. Frameless patient tracking with adhesive optical skin markers for augmented reality surgical navigation in spine surgery. Spine. 2020;45(22):1598–1604. doi: 10.1097/BRS.0000000000003628.
   PubMed Web of Science ®Google Scholar
• Edström E, Burström G, Nachabe R, et al. A novel augmented-reality-based surgical navigation system for spine surgery in a hybrid operating room: design, workflow, and clinical applications. Oper Neurosurg. 2020;18(5):496–502. doi: 10.1093/ons/opz236.
   Web of Science ®Google Scholar
• Edström E, Burström G, Persson O, et al. Does augmented reality navigation increase pedicle screw density compared to free-hand technique in deformity surgery? Single surgeon case series of 44 patients. Spine. 2020;45(17):E1085–E1090. doi: 10.1097/BRS.0000000000003518.
   PubMed Web of Science ®Google Scholar
• Elmi-Terander A, Burström G, Nachabé R, et al. Augmented reality navigation with intraoperative 3D imaging vs fluoroscopy-assisted free-hand surgery for spine fixation surgery: a matched-control study comparing accuracy. Sci Rep. 2020;10(1):707. doi: 10.1038/s41598-020-57693-5.
   PubMed Web of Science ®Google Scholar
• Golse N, Petit A, Lewin M, et al. Augmented reality during open liver surgery using a markerless non-rigid registration system. J Gastrointest Surg. 2021;25(3):662–671. doi: 10.1007/s11605-020-04519-4.
   PubMed Web of Science ®Google Scholar
• Carl B, Bopp M, Voellger B, et al. Augmented reality in transsphenoidal surgery. World Neurosurg. 2019;125:e873–e883.
   PubMed Web of Science ®Google Scholar
• Bourdel N, Chauvet P, Calvet L, et al. Use of augmented reality in gynecologic surgery to visualize adenomyomas. J Minim Invasive Gynecol. 2019;26(6):1177–1180. doi: 10.1016/j.jmig.2019.04.003.
   PubMed Web of Science ®Google Scholar
• Wierzbicki R, Pawłowicz M, Job J, et al. 3D mixed-reality visualization of medical imaging data as a supporting tool for innovative, minimally invasive surgery for gastrointestinal tumors and systemic treatment as a new path in personalized treatment of advanced cancer diseases. J Cancer Res Clin Oncol. 2022;148(1):237–243. doi: 10.1007/s00432-021-03680-w.
   PubMed Web of Science ®Google Scholar
• Léger É, Drouin S, Collins DL, et al. Quantifying attention shifts in augmented reality image-guided neurosurgery. Healthc Technol Lett. 2017;4(5):188–192. doi: 10.1049/htl.2017.0062.
   PubMed Web of Science ®Google Scholar
• Lim AK, Ryu J, Yoon HM, et al. Ergonomic effects of medical augmented reality glasses in video-assisted surgery. Surg Endosc. 2022;36:988–998.
   PubMed Web of Science ®Google Scholar
• Li C, Zheng B, Yu Q, et al. Augmented reality and 3-dimensional printing technologies for guiding complex thoracoscopic surgery. Ann Thorac Surg. 2021;112(5):1624–1631. doi: 10.1016/j.athoracsur.2020.10.037.
   PubMed Web of Science ®Google Scholar
• Léger É, Reyes J, Drouin S, et al. Marin: an open-source mobile augmented reality interactive neuronavigation system. Int J Comput Assist Radiol Surg. 2020;15(6):1013–1021. doi: 10.1007/s11548-020-02155-6.
   PubMed Web of Science ®Google Scholar
• Hussain R, Guigou C, Lalande A, et al. Vision-based augmented reality system for middle ear surgery: evaluation in operating room environment. Otol Neurotol. 2022;43(3):385–394. doi: 10.1097/MAO.0000000000003441.
   PubMed Web of Science ®Google Scholar
• Goto Y, Kawaguchi A, Inoue Y, et al. Efficacy of a novel augmented reality navigation system using 3D computer graphic modeling in endoscopic transsphenoidal surgery for sellar and parasellar tumors. Cancers. 2023;15(7):2148. doi: 10.3390/cancers15072148.
   PubMed Web of Science ®Google Scholar
• Mozaffari K, Foster C, Rosner M. Practical use of augmented reality modeling to guide revision spine surgery: an illustrative case of hardware failure and overriding spondyloptosis. Oper Neurosurg. 2022;23(3):212–216. doi: 10.1227/ons.0000000000000307.
   Web of Science ®Google Scholar
• Phan TN, Prakash KJ, Elliott RJS, et al. Virtual reality–based 3-dimensional localization of stereotactic EEG (sEEG) depth electrodes and related brain anatomy in pediatric epilepsy surgery. Childs Nerv Sys. 2022;38:537–546.
   PubMed Web of Science ®Google Scholar
• Satoh M, Nakajima T, Yamaguchi T, et al. Evaluation of augmented-reality based navigation for brain tumor surgery. J Clin Neurosci. 2021;94:305–314. doi: 10.1016/j.jocn.2021.10.033.
   PubMed Web of Science ®Google Scholar
• Alshomer F, Alazzam A, Alturki A, et al. Smartphone-assisted augmented reality in craniofacial surgery. Plast Reconstr Surg Glob Open. 2021;9(8):e3743.
   PubMedGoogle Scholar
• García-Mato D, Moreta-Martínez R, García-Sevilla M, et al. Augmented reality visualization for craniosynostosis surgery. Comput Methods Biomech Biomed Eng Imaging Vis. 2021;9(4):392–399. doi: 10.1080/21681163.2020.1834876.
   Web of Science ®Google Scholar
• Guerriero L, Quero G, Diana M, et al. Virtual reality exploration and planning for precision colorectal surgery. Dis Colon Rectum. 2018;61(6):719–723. doi: 10.1097/DCR.0000000000001077.
   PubMed Web of Science ®Google Scholar
• Yodrabum N, Rudeejaroonrung K, Chaikangwan I, et al. Precision of low-cost augmented reality in prefabricated cutting guide for fibular free flap surgery. J Craniofac Surg. 2022;33(3):916–919. doi: 10.1097/SCS.0000000000008074.
   PubMed Web of Science ®Google Scholar
• Tang R, Ma L, Xiang C, et al. Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: a case report. Medicine. 2017;96(37):e8083. doi: 10.1097/MD.0000000000008083.
   PubMed Web of Science ®Google Scholar
• Lee D, Yu HW, Kim S, et al. Vision-based tracking system for augmented reality to localize recurrent laryngeal nerve during robotic thyroid surgery. Sci Rep. 2020;10(1):8437. doi: 10.1038/s41598-020-65439-6.
   PubMed Web of Science ®Google Scholar
• Arensmeyer J, Bedetti B, Schnorr P, et al. Novel application of real-time editable 3D-reconstructed imaging in a mixed-reality holographic overlay using a video pass-through head-mounted display – a pathway to future navigation in thoracic surgery? 2022.
   Google Scholar
• Guo Q, Li X, Tang Y, et al. Augmented reality and three-dimensional plate library-assisted posterior minimally invasive surgery for scapula fracture. Int Orthop. 2022;46(4):875–882. doi: 10.1007/s00264-022-05303-6.
   PubMed Web of Science ®Google Scholar
• Han W, Yang X, Wu S, et al. A new method for cranial vault reconstruction: augmented reality in synostotic plagiocephaly surgery. J Craniomaxillofac Surg. 2019;47(8):1280–1284. doi: 10.1016/j.jcms.2019.04.008.
   PubMed Web of Science ®Google Scholar
• Liu J,  Li X, Leng X, et  al. Effect  of  3D slicer  preoperative planning and intraoperative guidance  with  mobile phone virtual reality  technology  on brain glioma surgery. Contrast Media  Mol  Imaging. 2022;2022:9627663. doi: 10.1155/2022/9627663.
   PubMed Web of Science ®Google Scholar
• Ceccariglia F, Cercenelli L, Badiali G, et al. Application of augmented reality to maxillary resections: a three-dimensional approach to maxillofacial oncologic surgery. J Pers Med. 2022;12(12):2047. doi: 10.3390/jpm12122047.
   PubMed Web of Science ®Google Scholar
• Schneider CS, Thompson SJS, Totz JNR, et al. Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study. Surg Endosc. 2020;34(10):4702–4711. doi: 10.1007/s00464-020-07807-x.
   PubMed Web of Science ®Google Scholar
• Scherl C, Stratemeier J, Karle C, et al. Augmented reality with Hololens in parotid surgery: how to assess and to improve accuracy. Eur Arch Otorhinolaryngol. 2021;278(7):2473–2483. doi: 10.1007/s00405-020-06351-7.
   PubMed Web of Science ®Google Scholar
• Scherl C, Stratemeier J, Rotter N, et al. Augmented reality with Hololens® in parotid tumor surgery: a prospective feasibility study. 2021.
   Google Scholar
• Saito Y, Sugimoto M, Imura S, et al. Intraoperative 3D hologram support with mixed reality techniques in liver surgery. Ann Surg. 2020;271(1):e4–e7. doi: 10.1097/SLA.0000000000003552.
   PubMed Web of Science ®Google Scholar
• Buch VP, Mensah-Brown KG, Germi JW, et al. Development of an intraoperative pipeline for holographic mixed reality visualization during spinal fusion surgery. Surg Innov. 2021;28(4):427–437. doi: 10.1177/1553350620984339.
   PubMed Web of Science ®Google Scholar
• Guerrera F, Nicosia S, Costardi L, et al. Proctor-guided virtual reality–enhanced three-dimensional video-assisted thoracic surgery: an excellent tutoring model for lung segmentectomy. Tumori J. 2021;107(6):NP1–NP4. doi: 10.1177/0300891620972173.
   PubMed Web of Science ®Google Scholar
• Shuhaiber J. Augmented reality in surgery. Arch Surg. 2004;139(2):170–174. doi: 10.1001/archsurg.139.2.170.
   PubMedGoogle Scholar
• Umebayashi D, Yamamoto Y, Nakajima Y, et al. Augmented reality visualization-guided microscopic spine surgery: transvertebral anterior cervical foraminotomy and posterior foraminotomy. J Am Acad Orthop Surg Glob Res Rev. 2018;2(4):e008. doi: 10.5435/JAAOSGlobal-D-17-00008.
   PubMedGoogle Scholar
• Mishvelov AE, Ibragimov AK, Amaliev IT, et al. Computer-assisted surgery: virtual- and augmented-reality displays for navigation during planning and performing surgery on large joints. Pharmacophore. 2021;12(2):32–38. doi: 10.51847/50jmUfdufI.
   Web of Science ®Google Scholar
• Balci D, Kirimker EO, Raptis DA, et al. Uses of a dedicated 3D reconstruction software with augmented and mixed reality in planning and performing advanced liver surgery and living donor liver transplantation (with videos). Hepatob Pancreat Dis Int. 2022;21:455–461.
   PubMed Web of Science ®Google Scholar
• Linxweiler M, Pillong L, Kopanja D, et al. Augmented reality-enhanced navigation in endoscopic sinus surgery: a prospective, randomized, controlled clinical trial. Laryngosc Investig Otol. 2020;5(4):621–629. doi: 10.1002/lio2.436.
   PubMed Web of Science ®Google Scholar
• Medzhidova SA, Koshenov NA, Nahusheva LR, et al. Application of augmented reality technology in abdominal surgery in patients with liver diseases. Int Trans J Eng Manag Appl Sci Technol. 2022;13:1–9.
   Google Scholar
• De Backer P, Van Praet C, Simoens J, et al. Improving augmented reality through deep learning: real-time instrument delineation in robotic renal surgery. Eur Urol. 2023;84(1):86–91. doi: 10.1016/j.eururo.2023.02.024.
   PubMed Web of Science ®Google Scholar
• Dongheon L, Hyoun-Joong K, Donguk K, et al. Preliminary study on application of augmented reality visualization in robotic thyroid surgery. ASTR. 2018;95:297–302.
   Google Scholar
• Gouveia PF, Costa J, Morgado P, et al. Breast cancer surgery with augmented reality. Breast. 2021;56:14–17. doi: 10.1016/j.breast.2021.01.004.
   PubMed Web of Science ®Google Scholar
• Porpiglia F, Checcucci E, Amparore D, et al. Augmented-reality robot-assisted radical prostatectomy using hyper-accuracy three-dimensional reconstruction (HA 3DTM) technology: a radiological and pathological study. BJU Int. 2019;123(5):834–845.
   Google Scholar
• Diana M, Soler LP, Agnus VP, et al. Prospective evaluation of precision multimodal gallbladder surgery navigation: virtual reality, near-infrared fluorescence, and x-ray-based intraoperative cholangiography. Ann Surg. 2017;266(5):890–897. doi: 10.1097/SLA.0000000000002400.
   PubMed Web of Science ®Google Scholar
• Aoyama R, Anazawa U, Hotta H, et al. Augmented reality device for preoperative marking of spine surgery can improve the accuracy of level identification. Spine Surg Relat Res. 2022;6(3):303–309. doi: 10.22603/ssrr.2021-0168.
   PubMed Web of Science ®Google Scholar
• Incekara F, Smits M, Dirven C, et al. Clinical feasibility of a wearable mixed-reality device in neurosurgery. World Neurosurg. 2018;118:e422–e427. doi: 10.1016/j.wneu.2018.06.208.
   PubMed Web of Science ®Google Scholar
• Zhu M, Liu F, Chai G, et al. A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep. 2017;7(1):42365. doi: 10.1038/srep42365.
   PubMedGoogle Scholar
• Hu MH, Chiang CC, Wang ML, et al. Clinical feasibility of the augmented reality computer-assisted spine surgery system for percutaneous vertebroplasty. Eur Spine J. 2020;29(7):1590–1596. doi: 10.1007/s00586-020-06417-4.
   PubMed Web of Science ®Google Scholar
• Molina CAM, Sciubba DMM, Greenberg JK, et al. Clinical accuracy, technical precision, and workflow of the first in human use of an augmented-reality head-mounted display stereotactic navigation system for spine surgery. 2021.
   Google Scholar
• Condino S, Carbone M, Piazza R, et al. Perceptual limits of optical see-through visors for augmented reality guidance of manual tasks. IEEE Trans Biomed Eng. 2019;67:411–419.
   PubMed Web of Science ®Google Scholar
• Sasaki T, Dehari H, Ogi K, et al. Application of a mixed reality device to oral surgery. Adv Oral Maxillofac Surg. 2022;8:100331. doi: 10.1016/j.adoms.2022.100331.
   Google Scholar
• Gregory TM, Gregory J, Sledge J, et al. Surgery guided by mixed reality: presentation of a proof of concept. Acta Orthop. 2018;89(5):480–483. doi: 10.1080/17453674.2018.1506974.
   PubMed Web of Science ®Google Scholar
• Lysenko A, Razumova A, Andrey Y, et al. The use of augmented reality navigation technology in combination with endoscopic surgery for the treatment of an odontogenic cyst of the upper jaw: a technical report. Imaging Sci Dent. 2022;52(2):225–230. doi: 10.5624/isd.20210256.
   PubMed Web of Science ®Google Scholar
• Nguyen T, Plishker W, Matisoff A, et al. Holous: augmented reality visualization of live ultrasound images using Hololens for ultrasound-guided procedures. Int J Comput Assist Radiol Surg. 2022;17(2):385–391. doi: 10.1007/s11548-021-02526-7.
   PubMed Web of Science ®Google Scholar
• Farshad-Amacker NA, Kubik-Huch RA, Kolling C, et al. Learning how to perform ultrasound-guided interventions with and without augmented reality visualization: a randomized study. Eur Radiol. 2023;33(4):2927–2934. doi: 10.1007/s00330-022-09220-5.
   PubMed Web of Science ®Google Scholar
• Gerard IJ, Kersten-Oertel M, Drouin S, et al. Combining intraoperative ultrasound brain shift correction and augmented reality visualizations: a pilot study of eight cases. J Med Imaging. 2018;5(2):021210. doi: 10.1117/1.JMI.5.2.021210.
   Google Scholar