Process-oriented metrics to provide feedback and assess the performance of students who are learning surgical procedures: The percutaneous dilatational tracheostomy case

Abstract

Purpose

Assessing competency in surgical procedures is key for instructors to distinguish whether a resident is qualified to perform them on patients. Currently, assessment techniques do not always focus on providing feedback about the order in which the activities need to be performed. In this research, using a Process Mining approach, process-oriented metrics are proposed to assess the training of residents in a Percutaneous Dilatational Tracheostomy (PDT) simulator, identifying the critical points in the execution of the surgical process.

Materials and methods

A reference process model of the procedure was defined, and video recordings of student training sessions in the PDT simulator were collected and tagged to generate event logs. Three process-oriented metrics were proposed to assess the performance of the residents in training.

Results

Although the students were proficient in classic metrics, they did not reach the optimum in process-oriented metrics. Only in 25% of the stages the optimum was achieved in the last session. In these stages, the four more challenging activities were also identified, which account for 32% of the process-oriented metrics errors.

Conclusions

Process-oriented metrics offer a new perspective on surgical procedures performance, providing a more granular perspective, which enables a more specific and actionable feedback for both students and instructors.

Keywords:

• Medical education
• process mining
• technical skills
• surgical procedures

Disclosure statement

No potential competing interest was reported by the authors.

Glossary

 

Process mining: move text below to here

Is an emerging discipline providing comprehensive sets of tools to provide fact-based insights and to support process improvements. This new discipline bridges the gap between traditional model based process analysis (e.g. simulation and other business process management techniques) and data-centric analysis techniques such as machine learning and data mining. Aalst, W. van der. Process Mining, Data Science in Action (Springer, 2016). doi:10.1007/978-3-662-49851-4.

Additional information

Funding

This work was partially supported by Agencia Nacional de Investigación y Desarrollo de Chile (501100020884), grant ids: ANID-P FCHA/Doctorado Nacional/2019-21190116; ANID-P FCHA/Doctorado Nacional/2020-21201411 and FONDECYT 1220202; and Dirección de Investigación de la Vicerrectoría de Investigación de la Pontificia Universidad Católica de Chile (501100009610), grant id: P UENT E 026/2021.

Notes on contributors

Juan José Martínez

Juan José Martínez is a Master student at Department of Computer Science, School of Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile.

Víctor Galvez-Yanjari

Victor Galvez-Yanjari is a PhD candidate at Department of Computer Science, School of Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile.

Rene de la Fuente

Rene de la Fuente, M.D., M.Sc., PhD (C) is an Associate Professor at Division of Anaesthesiology, School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile.

Catalina Kychenthal

Catalina Kychenthal is a MD, School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile.

Eduardo Kattan

Eduardo Kattan, M.D., M.M.Ed., is an Assistant Professor at Departamento de Medicina Intensiva, Facultad de Medicina, Pontificia Universidad Católica de Chile, Santiago, Chile.

Sebastián Bravo

Sebastian Bravo, M.D., is an Assistant Professor at Departamento de Medicina Intensiva, Facultad de Medicina, Pontificia Universidad Católica de Chile, Santiago, Chile.

Jorge Munoz-Gama

Jorge Munoz-Gama, Ph.D., is an Associate Professor at Department of Computer Science, School of Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile.

Marcos Sepúlveda

Marcos Sepúlveda, Ph.D., is an Associate Professor at Department of Computer Science, School of Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile.