Using routine data for long term impact evaluation: Methodological reflections from a complex health system intervention in a low-income context

Figures & data

Table 1. Missing data by service indicators

Service indicator	Missing values
	Observations	Percent
Antenatal care	3,925	7.71
Normal deliveries	4,012	7.88
Postnatal care	9,178	18.02
N (919 facilities)	50,919

Table 2. Stationarity test

Type of service	Intervention facilities					Control facilities
	Autocorrelation		Trend stationarity			Autocorrelation		Trend stationarity
	ACF	PACF	Systematic	Periodic	Random	ACF	PACF	Systematic	Periodic	Random
Antenatal care	AR (1)	MA (32)	−12***	−2	−10***	AR (1)	MA (39)	−15***	−3	−11***
Normal deliveries	AR (1)	MA (1)	−14***	−3**	−12***	AR (1)	MA (34)	−12***	−3**	−10***
Postnatal care	AR (1)	MA (40)	−13***	−3	−10***	AR (1)	MA (39)	−14***	−3	−11***

Figure 1. Trend graphs for services indicators.

Table 3. Selection of model functional form

Outcomes	Linear		Quadratic		Semi-log
	AIC	BIC	AIC	BIC	AIC	BIC
Antenatal care	−655.7	−649.2	−656.0	−649.6	−146.0	−139.6
Normal deliveries	−953.1	−946.6	−952.9	−946.5	19.8	26.3
Postnatal care	−994.1	−987.6	−994.1	−987.6	−116.7	−110.2

Notes: AIC, Akaike information criterion; BIC, Bayesian information criterion.

Bold denotes selected model.

Table 4. GLS-based estimates (imputed data)

Maternal care services	Delivery	Antenatal	Postnatal
Baseline situation
Level (control group) (${\theta }_0$)	0.069***	0.237***	0.089***
Trend (control group) (${\theta }_1$)	0.001	0.002	1.39e-04
Difference in level (${\theta }_4$)	−0.008	0.001	−3.73e-04
Difference in trend (${\theta }_5$)	4.013e-04	−0.002	−7.38e-04
Effects of original intervention (PBF)
Level change (control group) (${\theta }_2$)	0.008	−0.008	0.005*
Change in trend (control group) (${\theta }_3$)	−1.61e-04	−0.002	−3.04e-04
Difference in level (${\theta }_6$)	−0.004	−0.002	−0.002
Difference in trend (${\theta }_7$)	2.64e-04	0.004**	0.002**
Effects of original + parallel interventions
Level change (control group) (${\theta }_8$)	0.002	0.002	−0.001
Change in trend (control group) (${\theta }_9$)	0.003**	0.003	0.001
Difference in level (${\theta }_{10}$)	0.005	−0.019	−0.005
Difference in trend (${\theta }_{11}$)	−0.003***	−0.003	−0.001
Quadratic term	−2.06e-05	−1.03e-05	−3.67e-06
Autocorrelation
rho	−0.198	0.697	0.711
DW statistic (original)	2.365	0.777	0.879
DW statistic (transformed)	2.025	2.064	1.912

Table 5. GLS-based estimates (unimputed data)

Maternal care services	Delivery	Antenatal	Postnatal
Baseline situation
Level (control group) (${\theta }_0$)	0.069***	0.239***	0.064***
Trend (control group) (${\theta }_1$)	0.001	1.03e-04	0.002***
Difference in level (${\theta }_4$)	0.004	0.013	−0.001
Difference in trend (${\theta }_5$)	1.49e-04	−0.001	4.39e-04
Effects of original intervention (PBF)
Level change (control group) (${\theta }_2$)	5.03e-04	0.025***	−0.004***
Change in trend (control group) (${\theta }_3$)	−0.001	−0.007***	2.34e-04
Difference in level (${\theta }_6$)	−0.001	0.003	−0.004**
Difference in trend (${\theta }_7$)	−1.22e-04	0.002	4.31e-04
Effects of original +parallel interventions
Level change (control group) (${\theta }_8$)	−0.003	−0.002	0.001
Change in trend (control group) (${\theta }_9$)	0.001	−0.002	0.001
Difference in level (${\theta }_{10}$)	0.001	0.002	5.59e-04
Difference in trend (${\theta }_{11}$)	0.001	−0.001	−0.001
Quadratic term	2.95e-08	1.13e-04	−2.49e-04**
Autocorrelation
rho	0.317	0.704	0.775
DW statistic (original)	1.735	0.704	0.555
DW statistic (transformed)	1.578	1.701	1.074

Figure 2. Interrupted time series graphs (imputed data).

Figure 3. Interrupted time series graphs (unimputed data).