Strengthening routine health data analysis in Ethiopia: the Operational Research and Coaching for Analysts (ORCA) experience

Figures & data

Figure 1. ORCA capacity development model

Table

Specific aims To foster critical thinking regarding data quality, including validation and triangulation of data across sources To increase interdisciplinary discussions on how to reduce the presence of bias in key health metrics that are reported and used for planning and policy formation To incubate and promote the retention of an internal team of experts who will drive improvements in the generation, management, and use of data within the Ethiopian health system. To generate improved and more robust measurements on an agreed set of routine health information management indicators. Recruitment of participants Participation was competitive. The opportunity was circulated throughout directorates at MOH, EPHI and EPSA that directly handle routine health information and employ analysts. The application included a short data analysis and interpretation test accompanied by questions on candidates’ experience and motivation for participating. The number of participants was capped at 40, and quotes were set at 20 from MOH and 10 each from EPHI and EPSA, with 50% male/female distribution. Oversight and Staffing ORCA management team included a principal investigator, contracted by LSHTM and administrator and IT-lead contracted by WHO. Academic staff included ‘coaches’, who were experienced researchers and PhD:s. A Technical Advisory Group (TAG) ensured alignment with MOH strategy and consisted of directors from the 3 government agencies and ORCA leadership and was chaired by an MOH directorate head. The TAG was tasked with monitoring ORCA progress, suggesting modifications and liaising between ORCA and relevant MOH bodies. Financial and Material incentives The programme budgeted included a grant (USD 10,000) and personal laptop for each individual participant as an incentive to apply to the ORCA project and contribute free time to its deliverables. To reduce attrition of staff during the programme, the grant was paid in 4 instalments based on deliverables. There was a further budget to finance fieldwork costs during the mini-studies conducted by each small group.

Thematic Group Work Selection of six thematic areas that formed the basis of small working groups to which all participants were allocated for the remainder of the programme: Maternal Health, Neonatal Survival, Child Nutrition, Immunization, Malaria and Tuberculosis. Groups were organised to ensure diversity by discipline, agency and directorate, as well as to have a genderbalanced composition. Groups worked together throughout organised workshops and between them during their own time. Residential training workshop A combination of 3- and 5-day training workshops were organised outside of Addis Ababa to facilitate group interaction and reduce office-related distractions. Workshops provided intensive training and practicals covering: R-software, study protocol design, qualitative methods and analysis, writing up mixed method studies for dissemination. Research Proposal Each thematic group developed a research proposal for submission to the EPHI Institutional Review Board for ethical approval. Two objectives were adapted for each group as follows: a) to analyse regional and national existing data using the concepts of outliers, international consistency, consistency between related indicators and external consistency between data sources. b) to analyse facility level data and use, applying quantitative and qualitative methods to explore reasons for data quality issues at regional and district levels. Data Collection and Analysis All groups planned and conducted field studies conducted over 6–11 days with a shorter pilot phase during a feasibility visit. Each group gathered district level quantitative and qualitative data. For the first study objective teams conducted shared analysis. For the second objective groups analysed data based on tailored plans identified during protocol development. Coaching and Mentorship Each group was assigned an academic team member for fortnightly supervisory meetings to discuss progress during the applied work, accompanied by regular email support. Other LSHTM mentors participated in training workshops to deliver key skill-building components (e.g. qualitative analysis, writing support).

Table 1. ORCA monitoring framework

Measure	Documentation
Fidelity	Schedule of activities Project narrative kept by ORCA staff, including reflections on implementation progress Annual reports
Feasibility	Project narrative kept by ORCA staff, including reflections on implementation progress Semi-structured interviews with key health system stakeholders
Reach	Application records Attendance registers
Acceptability	Semi-structured interviews with participants at two time periods Semi-structured interviews with key health system stakeholders at two time periods
Plausible effect	Pre- Mid- and Post- survey using the Evidence Based Practice questionnaire Semi-structured interviews with participants and directorate managers at two time periods Number and quality of outputs demonstrating improved data interpretation and presentation Retention of participants within their work

Table 2. Results from the Evidence-Based Practice (EBP), Questionnaire distributed at start (June 2018), mid (September 2019) and end (June 2020) showing expressed relevance for EBP, sympathy for EBP, knowledge of EBP terminology, using EBP in daily work, and showing confidence in EBP activities among ORCA participants, and differences between males and females and MSc qualified and not MSc qualified, respectively

DOMAINS/SECTIONS (No of items, Scoring min-max)	SCORING 2018 mean (SD)	p-value*		SCORING 2019 mean (SD)	p-value*		SCORING 2020 mean (SD)	p-value*
Relevance (14 items,14–70)
Total (n = 35, NA = 2)	60.7 (5.9)		Total (n = 29, NA = 0)	62.5 (4.7)		Total (n = 25, NA = 3)	63.2 (4.0)
Males (21/35, NA = 1)	61.2 (6.0)	0.3322	Males (15/29)	61.9 (4.7)	0.3685	Males (17/25, NA = 0)	62.4 (3.8)	0.2929
Females (14/35, NA = 1)	59.3 (5.9)		Females (14/29)	61.9 (4.7)		Females (8/25, NA = 3)	64.9 (4.3)
Masters (25/35, NA = 1)	59.9 (6.0)	0.3827	Masters (25/29)	62.2 (4.9)	0.2525	Masters (24/25, NA = 1)	63.0 (4.0)	0.3302
Other education (10/35, NA = 1)	62.2 (6.3)		Other education (4/29)	64.8 (1.0)		Other education (1/25, NA = 2)	67.0 (NA)
Sympathy (7 items, 7–35)
Total (n = 35, NA = 2)	21.9 (3.9)		Total (n = 26, NA = 3)	22.8 (4.7)		Total (n = 27, NA = 1)	22.9 (3.8)
Males (21/35, NA = 1)	22.8 (3.9)	0.7955	Males (14/26, NA = 1)	23.1 (4.9)	0.6233	Males (17/27, NA = 0)	22.4 (4.2)	0.6138
Females (14/35, NA = 1)	21.4 (4.0)		Females (12/26, NA = 2)	22.6 (4.8)		Females (10/27, NA = 1)	23.7 (3.0)
Masters (25/35, NA = 1)	21.6 (4.0)	0.2083	Masters (22/26, NA = 3)	23.3 (4.8)	0.1336	Masters (24/27, NA = 1)	23.0 (3.9)	0.35
Other education (10/35, NA = 1)	22.7 (3.9)		Other education (4/26, NA = 0)	20.5 (3.7)		Other education (3/27, NA = 0)	21.7 (2.5)
Terminology (17 items, 17–85)
Total (n = 33, NA = 4)	48.7 (9.6)		Total (n = 26, NA = 3)	54.3 (8.8)		Total (n = 26, NA = 2)	61.5 (10.6)
Males (17/31, NA = 4)	45.8 (8.7)	0.103	Males (14/26, NA = 1)	50.2 (7.0)	0.03931	Males (16/26, NA = 1)	59.0 (9.2)	0.1323
Females (14/31, NA = 0)	51.6 (10.4)		Females (12/26, NA = 2)	59.0 (8.8)		Females (10/26, NA = 1)	65.4 (12.0)
Masters (22/31, NA = 3)	49.0 (10.8)	0.8867	Masters (22/26, NA = 3)	54.6 (9.3)	0.7487	Masters (23/26, NA = 2)	60.2 (10.3)	0.0913
Other education (9/31, NA = 1)	46.3 (6.6)		Other education (4/26, NA = 0)	52.3 (7.8)		Other education (3/26, NA = 0)	71.3 (8.6)
Practice (9 items, 9–45)
Total (n = 32, NA = 5)	19.8 (6.5)		Total (n = 26, NA = 3)	24.6 (6.2)		Total (n = 25, NA = 3)	24.5 (5.7)
Males (18/30, NA = 3)	17.9 (4.6)	0.06166	Males (12/26, NA = 3)	24.0 (6.0)	0.5884	Males (17/25, NA = 0)	23.7 (6.0)	0.9302
Females (12/30, NA = 2)	23.2 (8.1)		Females (14/26, NA = 0)	25.3 (6.5)		Females (8/25, NA = 3)	24.8 (5.4)
Masters (22/30, NA = 3)	19.7 (7.0)	0.8272	Masters (22/26, NA = 3)	25.2 (6.4)	0.226	Masters (22/25, NA = 3)	24.2 (6.1)	0.4758
Other education (8/30, NA = 2)	19.9 (6.5)		Other education (4/26, NA = 0)	21.2 (2.3)		Other education (3/25, NA = 0)	22.3 (1.6)
Confidence (11 items, 11–55)
Total (n = 35, NA = 2)	36.0 (6.8)		Total (n = 28, NA = 1)	37.9 (5.9)		Total (n = 27, NA = 1)	43.5 (5.7)
Males (20/34, NA = 0)	36.0 (7.4)	1	Males (15/28, NA = 0)	39.3 (4.7)	0.3799	Males (17/27, NA = 0)	42.2 (5.9)	0.1245
Females (13/34, NA = 1)	35.5 (6.2)		Females (13/28, NA = 1)	37.6 (5.7)		Females (10/27, NA = 1)	45.6 (4.9)
Masters (24/34, NA = 1)	35.7 (7.2)	0.7893	Masters (24/28, NA = 1)	37.7 (5.7)	0.5101	Masters (24/27, NA = 1)	43.2 (6.0)	0.2156
Other education (10/34, NA = 0)	37.1 (6.6)		Other education (4/28, NA = 0)	39.3 (7.8)		Other education (3/27, NA = 0)	46.0 (1.0)

* Mann-Whitney-Wilcoxon test.