Surgical data strengthening in Ethiopia: results of a Kirkpatrick framework evaluation of a data quality intervention

Figures & data

Table 1. Surgical key performance indicators (KPIs) developed by PGSSC and FMoH SaLTS team

Surgical Key Performance Indicators
1	Surgical Bed Occupancy Rate	9	Protection Against Catastrophic Expenditure
2	Delay for Elective Surgical Admission	10	Surgery, Anesthesia, Obstetrics (SAO) Provider Density
3	Mean Duration of In-Hospital, Pre-Operative Stay	11	Peri-Operative Mortality Rate (POMR)
4	Patient Satisfaction	12	Surgical Volume
5	Rate of First Elective Case On-Time Theater Performance	13	Surgical Site Infection
6	Rate of Cancellation of Elective Surgery	14	Anesthetic Adverse Outcome
7	Blood Unavailability Ratio	15	Rate of Safe Surgery Checklist (SSC) Utilization
8	Emergency Surgical Access	16	Surgical Referrals Out

Figure 1. Amhara and Tigray pre- and post-test scores comparison (Kirkpatrick level 2)

Table 2. Key performance indicator (KPI) calculation comparison, REDCap v. registry (Kirkpatrick level 2)

Percent Difference Between Calculated Values of Key Performance Indicators (KPIs) (REDCap v. Registry)
	Difference (%)
Hospital	Anesthetic Adverse Outcome	Post-operative Mortality Rate	Surgical Site Infection	Surgical Volume	Referrals Out
1	0.00%	0.00%	0.00%	4.10%	5.40%
2	0.00%	0.00%	0.00%	0.00%	9.10%
3	0.00%	0.00%	0.00%	33.30%	38.00%
4	0.00%	0.00%	100.00%	22.00%	12.80%
5	100.00%	0.00%	0.00%	7.70%	100.00%
6	0.00%	0.00%	100.00%	2.94%	76.09%
7	0.00%	0.00%	0.00%	2.78%	2.44%
8	100.00%	0.00%	0.00%	3.57%	0.00%
9	0.00%	0.00%	0.00%	6.00%	38.89%
10	0.00%	0.00%	100.00%	0.00%	100.00%

Figure 2. Percentage of participants who agree and strongly agree with program success, individual behavior change, and institutional behavior change items following the intervention (Kirkpatrick levels 3 & 4)

Table 3. Qualitative impact of training as perceived by participants, grouped by theme (Kirkpatrick level 4)

Theme	Description	Quotes
KPI Knowledge Gain and Data Utilization	Many participants expressed increased knowledge gain regarding the surgical KPIs and how the hospital utilizes data better because of the intervention.	“The introduction of KPIs has improved the quality of care. Before the training, we were reporting [data] but it was lacking completeness. Besides, there was no calculation of the KPIs. However, after the training, we calculated the KPIs and understood its benefits. This provides us an opportunity to measure the indicators and monitor our progress objectively. We can now determine the safety checklist [use], the mortality related to the service, SSI and the like. Before the training, there was no measurement of the pre-operative care, safety checklist and even blood availability. The calculation of blood availability would help us to identify the number of referral outs associated with blood availability. – Tigray, Hospital 2 There has been improvement in acknowledging the value of data since the training. The training also makes people focus on very key performance indicators that are important in monitoring and evaluating the activities. – Amhara, Hospital 2 We also became aware of the advantages of properly managed data … such as for planning, research and decision – Amhara, Hospital 1 … the intervention helped us to identify our gaps and knowing the gaps leads us to strive towards the solution.” – Amhara, Hospital 2
Agency	Leadership and accountability growth in staff were reported by many, especially regarding an increase in confidence or agency around data and their professional role because of the training.	“The training has brought remarkable changes on the following activities … it has created sense of ownership and accountability among staff members – Amhara, Hospital 2 I and my colleagues perceived that we are responsible for the success and failure of data quality in this hospital. With this in mind, we have improved in data documentation after training. – Amhara, Hospital 3 Even recording and filling up of the data was based on the level of willingness of the professionals: some may fill while some others may not. But now, all staffs are responsible to record the data of what they did.” – Tigray, Hospital 1
Teamwork and Communication	Over three-quarters of the hospitals reported that teamwork and communication within the surgical team and the hospital improved because of the intervention.	“Thus, the training comes with the concept that safe surgery is the role of all staff in the hospital and should be approached as a team. What makes a hospital is when the hospital works [as a] team … the team work and team sprit has improved.” – Tigray, Hospital 5 “Surgical Data quality has been increasing by using standardized registries and well oriented staff. Lessons are also obtained and we are trying to scale up this documentation system to non- surgical units as well.” – Amhara, Hospital 4 “Regarding the teamwork, the SaLTs members consist of staff from higher level personnel of the hospital to the bottom level, which gives strength and attention to the issues. Especially, the CEO is a member of the team and he raises it as an agenda always, which keeps the issue closer [for] follow up.” – Tigray, Hospital 4 “Health is team work and everybody from the cleaner to the higher officials is involved in patient management even if the role and depth of involvement varies.” – Amhara, Hospital 1
Data Quality	Data quality improvements were cited by all hospitals, indicating that the quality of data in their hospital was positively impacted by the program	“There is also much change in the data quality. There was a weakness and carelessness to record what is done but now it has changed. After[ward] the training staff became curious [about] recording what they did.” – Tigray, Hospital 5 “Data quality has been improved and the following have made their own contribution in this aspect: One, things are critically evaluated periodically by team members and corrective actions are being taken accordingly.” – Amhara, Hospital 2 I can say that data quality in this hospital has improved well. For example, surgical site infection, which did not have ownership before the training, got its own registry and responsible person after the training. Moreover, anesthetic complications, referral system, recording and reporting of canceled surgical cases were all improved after the training.” – Amhara, Hospital 1
Surgical Services	Respondents indicated that the training and knowledge gained have allowed for more discussions around improving quality of surgical services and efficiency of hospital administration	“One component of the training was surgical site infection. In our context, the attention given to the issues was limited. But, after the training we are giving attention to it through training. We are looking at the reasons that cause the infection and training for personnel regarding the issues is provided accordingly.” – Tigray, Hospital 1 “Data collection and reporting has become uniform across all units of the hospital. Until the training we were recording data using different formats. We also have started processing, interpreting and utilizing data for our consumption.” – Amhara, Hospital 2 “ … the intervention helped us to identify our gaps and knowing the gaps leads us to strive towards the solution.”- Tigray, Hospital 1

Table 4. Reported challenges to implementation, grouped by theme (Kirkpatrick level 4)

Theme	Description	Quotes
Buy-In Issues	There were some difficulties in the beginning of the intervention, as reported by some facilities, with getting buy-in from providers for the new registry system, especially those that did not attend the training.	“ … initially there were attitude problems among some staff members in anesthesia department. As you know timing of registration is critical for patients. In this aspect, there are checklists to be filled before, during and after operation.” – Amhara, Hospital 1
Difficulties Collecting Data	All hospitals reported difficulties collecting data. These issues ranged from lack of knowledge from members who didn’t attend trainings and issues with human resources.	“As a data collector, it was not an easy task in registering, compiling and online reporting of data using RED Cap software from seven registries weekly. I was using extra time- lunch, night and week end- especially during the first two to three weeks after the training” – Amhara, Hospital 4 “Though the final compiled activities are reported using electronics, still data are recorded manually using hard copies. However, if the registry had been computerized, it could be possible to save time, reduce errors, and retrieve data easily. Thus, there needs to be computer access in each unit to register activities routinely. Fortunately, this hospital has wireless internet access, and if computers are availing in each department it is possible to give comprehensive services.” – Amhara, Hospital 1 “ … we have no separated surgical ward. Medical ward and surgical ward are both in one. Similarly, all the patients admitted in the GYN [ward] may also be surgical cases. Therefore, the SaLTs team should be further expanded to improve the reach of the intervention.” Tigray, Hospital 1
Continued Training and Follow-Up	Some hospitals described the need for continued training and follow-up in order to reassess data quality and provide refresher trainings to combat turnover rates in the staff.	“There are also challenges related to data recording and reporting because not all staff are trained and there is also a high turnover of the trained staff. I am observing lack of ownership among non-trained staff … especially among nurses. Therefore, unless mechanisms are designed for continuous onsite training, there might be interruption on data quality management.” – Amhara, Hospital 3 “I suggest regular mentoring, at least every three months, for timely identification of the strong and weak parts of our activities” – Amhara, Hospital 1 “There should also be regular review meetings to look at progress to prevent relapse to the old system” – Tigray, Hospital 1