Minimally trained analysts can perform fast, objective assessment of orthopedic technical skill from fluoroscopic images

References

• A. Albanese, M. (2001). Challenges in using rater judgements in medical education. Journal of Evaluation in Clinical Practice, 6(3), 305–319. https://doi.org/10.1046/j.1365-2753.2000.00253.x
   Web of Science ®Google Scholar
• Alvand, A., Logishetty, K., Middleton, R., Khan, T., Jackson, W. F., Price, A. J., & Rees, J. L. (2013). Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair. Arthroscopy, 29(5), 906–912.
   Google Scholar
• Atesok, K., Macdonald, P., Leiter, J., Dubberley, J., Satava, R., Vanheest, A., Hurwitz, S., & Marsh, J. L. (2017). Orthopaedic education in the era of surgical simulation: Still at the crawling stage. World Journal of Orthopedics, 8(4), 290. https://doi.org/10.5312/wjo.v8.i4.290
   Web of Science ®Google Scholar
• Azari, D. P., Frasier, L. L., Quamme, S. R. P., Greenberg, C. C., Pugh, C. M., Greenberg, J. A., & Radwin, R. G. (2019). Modeling surgical technical skill using expert assessment for automated computer rating. Annals of Surgery, 269(3), 574–581. https://doi.org/10.1097/SLA.0000000000002478
   PubMed Web of Science ®Google Scholar
• Baumgaertner, M., Curtin, S. L., Lindskog, D., & Keggi, J. (1995). The value of the tip-apex distance in predicting failure of fixation of peritrochanteric fractures of the hip. The Journal of Bone and Joint Surgery. American Volume, 77(7), 1058–1064. https://doi.org/10.2106/00004623-199507000-00012
   PubMed Web of Science ®Google Scholar
• Bilgic, E., Valanci-Aroesty, S., & Fried, G. M. (2020). Video assessment of surgeons and surgery. Advances in Surgery, 54, 205–214. https://doi.org/10.1016/j.yasu.2020.03.002
   Google Scholar
• Boudissa, M., Oliveri, H., Chabanas, M., & Tonetti, J. (2018). Computer-assisted surgery in acetabular fractures: Virtual reduction of acetabular fracture using the first patient-specific biomechanical model simulator. Orthopaedics & Traumatology, Surgery & Research, 104(3), 359–362. https://doi.org/10.1016/j.otsr.2018.01.007
   Web of Science ®Google Scholar
• Celi, L. A., Ippolito, A., Montgomery, R. A., Moses, C., & Stone, D. J. (2014). Crowdsourcing knowledge discovery and innovations in medicine. Journal of Medical Internet Research, 16(9), e216. https://doi.org/10.2196/jmir.3761
   PubMedGoogle Scholar
• Chong, A. C. M., Pate, R. C., Prohaska, D. J., Bron, T. R., & Wooley, P. H. (2016). Validation of improvement of basic competency in arthroscopic knot tying using a bench top simulator in orthopaedic residency education. Arthroscopy: The Journal of Arthroscopic & Related Surgery, 32(7), 1389–1399. https://doi.org/10.1016/j.arthro.2016.01.059
   PubMed Web of Science ®Google Scholar
• Dai, J. C., Lendvay, T. S., & Sorensen, M. D. (2017). Crowdsourcing in surgical skills acquisition: A developing technology in surgical education. Journal of Graduate Medical Education, 9(6), 697–705. https://doi.org/10.4300/JGME-D-17-00322.1
   Google Scholar
• Darzi, A., Smith, S., & Taffinder, N. (1999). Assessing operative skill. Bmj, 318(7188), 887–888. https://doi.org/10.1136/bmj.318.7188.887
   PubMed Web of Science ®Google Scholar
• Fahim, C., Wagner, N., Nousiainen, M. T., & Sonnadara, R. (2018). Assessment of technical skills competence in the operating room. Academic Medicine: Medicine, 93(5), 794–808. https://doi.org/10.1097/ACM.0000000000001902
   Web of Science ®Google Scholar
• Gaba, D. M. (2004). The future vision of simulation in health care. Quality and Safety in Health Care, 13(suppl_1), i2–i10. https://doi.org/10.1136/qshc.2004.009878
   PubMedGoogle Scholar
• Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors [Review]. Academic Medicine: journal of the Association of American Medical Colleges, 86(10 Suppl), S1–S7. https://doi.org/10.1097/acm.0b013e31822a6cf8
   PubMed Web of Science ®Google Scholar
• Gofton, W. T., Dudek, N. L., Wood, T. J., Balaa, F., & Hamstra, S. J. (2012). The Ottawa surgical competency operating room evaluation (O-SCORE). Academic Medicine, 87(10), 1401–1407. https://doi.org/10.1097/ACM.0b013e3182677805
   PubMed Web of Science ®Google Scholar
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
   Web of Science ®Google Scholar
• Hazlehurst, B., Gorman, P. N., & McMullen, C. K. (2008). Distributed cognition: An alternative model of cognition for medical informatics. International Journal of Medical Informatics, 77(4), 226–234. https://doi.org/10.1016/j.ijmedinf.2007.04.008
   PubMed Web of Science ®Google Scholar
• Hettinger, A. Z., Roth, E. M., & Bisantz, A. M. (2017). Cognitive engineering and health informatics: Applications and intersections. Journal of Biomedical Informatics, 67, 21–33. https://doi.org/10.1016/j.jbi.2017.01.010
   Web of Science ®Google Scholar
• Iramaneerat, C., & Yudkowsky, R. (2007). Rater errors in a clinical skills assessment of medical students. Evaluation & the Health Professions, 30(3), 266–283. https://doi.org/10.1177/0163278707304040
   PubMed Web of Science ®Google Scholar
• Johnson, L. J., Cope, M. R., Shahrokhi, S., & Tamblyn, P. (2008). Measuring tip–apex distance using a picture archiving and communication system (PACS). Injury, 39(7), 786–790. https://doi.org/10.1016/j.injury.2007.12.019
   PubMed Web of Science ®Google Scholar
• Karam, M. D., Thomas, G. W., Koehler, D. M., Westerlind, B. O., Lafferty, P. M., Ohrt, G. T., Marsh, J. L., Van Heest, A. E., & Anderson, D. D. (2015). Surgical coaching from head-mounted video in the training of fluoroscopically guided articular fracture surgery. Journal of Bone and Joint Surgery, 97(12), 1031–1039. Volume, https://doi.org/10.2106/JBJS.N.00748
   Google Scholar
• Karam, M. D., Thomas, G. W., Taylor, L., Liu, X., Anthony, C. A., & Anderson, D. D. (2016). Value added: The case for point-of-view camera use in orthopedic surgical education. The Iowa Orthopaedic Journal, 36, 7–12. https://pubmed.ncbi.nlm.nih.gov/27528828
   Google Scholar
• Katz, A. J. (2016). The role of crowdsourcing in assessing surgical skills. Surgical Laparoscopy, Endoscopy & Percutaneous Techniques, 26(4), 271–277. https://doi.org/10.1097/SLE.0000000000000293
   Web of Science ®Google Scholar
• Koehler, R. J., Amsdell, S., Arendt, E. A., Bisson, L. J., Bramen, J. P., Butler, A., Cosgarea, A. J., Harner, C. D., Garrett, W. E., Olson, T., Warme, W. J., & Nicandri, G. T. (2013). The arthroscopic surgical skill evaluation tool (ASSET). The American Journal of Sports Medicine, 41(6), 1229–1237. https://doi.org/10.1177/0363546513483535
   PubMed Web of Science ®Google Scholar
• Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Acad Med, 89(5), 721–727. https://doi.org/10.1097/acm.0000000000000221
   PubMed Web of Science ®Google Scholar
• Kowalewski, T. M., Comstock, B., Sweet, R., Schaffhausen, C., Menhadji, A., Averch, T., Box, G., Brand, T., Ferrandino, M., Kaouk, J., Knudsen, B., Landman, J., Lee, B., Schwartz, B. F., McDougall, E., & Lendvay, T. S. (2016). Crowd-sourced assessment of technical skills for validation of basic laparoscopic urologic skills tasks. The Journal of Urology, 195(6), 1859–1865. https://doi.org/10.1016/j.juro.2016.01.005
   Web of Science ®Google Scholar
• Lendvay, T. S., White, L., & Kowalewski, T. (2015). Crowdsourcing to assess surgical skill. JAMA Surgery, 150(11), 1086–1087. https://doi.org/10.1001/jamasurg.2015.2405
   Web of Science ®Google Scholar
• Long, S., Thomas, G. W., Karam, M. D., Marsh, J. L., & Anderson, D. D. (2021). Surgical Skill Can be Objectively Measured From Fluoroscopic Images Using a Novel Image-based Decision Error Analysis (IDEA) Score. Clinical Orthopaedics & Related Research, 479(6), 1386–1394. https://doi.org/10.1097/CORR.0000000000001623
   Web of Science ®Google Scholar
• Mackay, S., Datta, V., Chang, A., Shah, J., Kneebone, R., & Darzi, A. (2003). Multiple objective measures of skill (MOMS): A new approach to the assessment of technical ability in surgical trainees. Annals of Surgery, 238(2), 291–300. https://doi.org/10.1097/01.sla.0000080829.29028.c4
   Web of Science ®Google Scholar
• Martin, J. A., Regehr, G., Reznick, R., Macrae, H., Murnaghan, J., Hutchison, C., & Brown, M. (1997). Objective structured assessment of technical skill (OSATS) for surgical residents [Article]. British Journal of Surgery, 84(2), 273–278. ]. https://doi.org/10.1002/bjs.1800840237
   PubMed Web of Science ®Google Scholar
• MATLAB (2019). R2019a. The MathWorks Inc.
   Google Scholar
• Middleton, R. M., Baldwin, M. J., Akhtar, K., Alvand, A., & Rees, J. L. (2016). Which global rating scale? a comparison of the ASSET, BAKSSS, and IGARS for the assessment of simulated arthroscopic skills. J Bone Joint Surg Am, 98(1), 75–81.
   Google Scholar
• Miller, B., Azari, D., Radwin, R., & Le, B. (2018). MP01-07 use of computer vision motion analysis to aid in surgical skill assessment of suturing tasks. Journal of Urology, 199(4S). https://doi.org/10.1016/j.juro.2018.02.113
   Google Scholar
• Nemeth, C., O'Connor, M., Klock, P. A., & Cook, R. (2005). Cognitive artifacts’ implications for health care information technology: revealing how practitioners create and share their understanding of daily work. In K. Henriksen, J. B. Battles, E. S. Marks, & D. I. Lewin (Eds.), Advances in patient safety: From research to implementation (Volume 2: Concepts and methodology). Agency for Healthcare Research and Quality (US).
   Google Scholar
• Niitsu, H., Hirabayashi, N., Yoshimitsu, M., Mimura, T., Taomoto, J., Sugiyama, Y., Murakami, S., Saeki, S., Mukaida, H., & Takiyama, W. (2013). Using the objective structured assessment of technical skills (OSAT S) global rating scale to evaluate the skills of surgical trainees in the operating room. Surgery Today, 43(3), 271–275.
   Google Scholar
• Perron, N. J., Louis-Simonet, M., Cerutti, B., Pfarrwaller, E., Sommer, J., & Nendaz, M. (2016). Feedback in formative OSCEs: Comparison between direct observation and video-based formats. Medical Education Online, 21(1), 32160. https://doi.org/10.3402/meo.v21.32160
   PubMed Web of Science ®Google Scholar
• Rosset, A., Spadola, L., & Ratib, O. (2004). OsiriX: An open-source software for navigating in multidimensional DICOM images. Journal of Digital Imaging, 17(3), 205–216. https://doi.org/10.1007/s10278-004-1014-6
   PubMed Web of Science ®Google Scholar
• Schleicher, I., Leitner, K., Juenger, J., Moeltner, A., Ruesseler, M., Bender, B., Sterz, J., Schuettler, K.-F., Koenig, S., & Kreuder, J. G. (2017). Examiner effect on the objective structured clinical exam – a study at five medical schools. BMC Medical Education, 17(1), 71. https://doi.org/10.1186/s12909-017-0908-1
   Google Scholar
• Tashiro, Y., Miura, H., Nakanishi, Y., Okazaki, K., & Iwamoto, Y. (2009). Evaluation of skills in arthroscopic training based on trajectory and force data. Clinical Orthopaedics and Related Research, 467(2), 546–552. https://doi.org/10.1007/s11999-008-0497-8
   PubMed Web of Science ®Google Scholar
• Taylor, L. K., Thomas, G. W., Karam, M. D., Kreiter, C. D., & Anderson, D. D. (2016). Assessing wire navigation performance in the operating room. Journal of Surgical Education, 73(5), 780–787. https://doi.org/10.1016/j.jsurg.2016.03.018
   Web of Science ®Google Scholar
• Taylor, L. K., Thomas, G. W., Karam, M. D., Kreiter, C. D., & Anderson, D. D. (2018). Developing an objective assessment of surgical performance from operating room video and surgical imagery. IISE Transactions on Healthcare Systems Engineering, 8(2), 110–116. https://doi.org/10.1080/24725579.2017.1418767
   Google Scholar
• Thomas, G. W., Long, S., Tatum, M., Kowalewski, T., Mattioli, D., Marsh, J. L., Kowalski, H. R., Karam, M. D., Bechtold, J. E., & Anderson, D. D. (2020). A Vision for Using Simulation & Virtual Coaching to Improve the Community Practice of Orthopedic Trauma Surgery. The Iowa Orthopaedic Journal, 40(1), 25–34. https://pubmed.ncbi.nlm.nih.gov/32742205
   Google Scholar
• Tofte, J. N., Rojas, E. O., Anthony, C. A., Holte, A. J., Volkmar, A., Karam, M. D., Caldwell, L. S., & Lawler, E. A. (2019). Intraoperative point of view video capture and surgical segmentation in carpal tunnel release: a feasibility analysis. Journal of Surgical Education, 76(6), 1663–1668. https://doi.org/10.1016/j.jsurg.2019.06.004
   Web of Science ®Google Scholar
• Toprak, A., Luhanga, U., Jones, S., Winthrop, A., & McEwen, L. (2016). Validation of a novel intraoperative assessment tool: The Surgical Procedure Feedback Rubric. American Journal of Surgery, 211(2), 369–376. https://doi.org/10.1016/j.amjsurg.2015.08.032
   Web of Science ®Google Scholar
• Van Heest, A. E., Agel, J., Ames, S. E., Asghar, F. A., Harrast, J. J., Marsh, J. L., Patt, J. C., Sterling, R. S., & Peabody, T. D. (2019). Resident surgical skills web-based evaluation. The Journal of Bone and Joint Surgery. American Volume, 101(5), e18. https://doi.org/10.2106/JBJS.17.01512
   Web of Science ®Google Scholar
• Van Heest, A., Kuzel, B., Agel, J., Putnam, M., Kalliainen, L., & Fletcher, J. (2012). Objective structured assessment of technical skill in upper extremity surgery. The Journal of Hand Surgery, 37(2), 332–337.e334. https://doi.org/10.1016/j.jhsa.2011.10.050
   Google Scholar
• Vedula, S. S., Ishii, M., & Hager, G. D. (2017). Objective assessment of surgical technical skill and competency in the operating room. Annual Review of Biomedical Engineering, 19(1), 301–325. https://doi.org/10.1146/annurev-bioeng-071516-044435
   Google Scholar
• Zhang, M., Cheng, X., Copeland, D., Desai, A., Guan, M. Y., Brat, G. A., Yeung, S. (2020). Using computer vision to automate hand detection and tracking of surgeon movements in videos of open surgery. AMIA … Annual Symposium Proceedings. AMIA Symposium, 2020. http://europepmc.org/abstract/MED/34025905
   Google Scholar