Three Statistical Approaches for Assessment of Intervention Effects: A Primer for Practitioners

Figures & data

Table 1 Summary of Study Design, Data Requirements, Assumptions and Extensions of Three Statistical Approaches for Modelling the Impact of an Intervention

Statistical Approach	Study Design	Data Requirements	Underlying Assumptions	Model Extensions
DID	Two-group, two-period	- Repeated cross-sectional data - Panel data - A minimum of 1 observation per subject per pre/post intervention time period	- Parallel trend in the absence of intervention - No spillover effects - Independent residuals	-Serial correlation - Group-specific time trend - Propensity score-based analysis
Segmented regression of ITS	One-group, two-period	- Interrupted time series - Requires at least 12 observations pre/post - Intervention time period	- Linearity - No systematic change in the characteristics of cohort studied over time - Independent residuals	- Multiple change points - Autocorrelation - Multiple groups - Seasonality
Interventional ARIMA	One-group, two-period design	- Interrupted time series	- Stationarity	- Seasonality or cyclic pattern
		- Requires at least 50 observations	- No systematic change in the characteristics of cohort studied	- Flexible intervention function
			- No factor other than the intervention

Table 2 Estimated Impact of Medicaid Expansion on Health Insurance Coverage Under Different DID Models

DID Model	Estimated Impact of Expansion, , (95% CI) ^a	P value ^a	Estimated Coefficient of Pre-Common Trend Test ^b	P value ^b
State fixed effect	5.93 (3.99 to 7.89)	<0.001	0.64	0.436
Linear time trend	4.35 (2.32 to 6.38)	<0.001	0.90	0.236
State fixed effect with linear time trend	5.47 (3.58 to 7.35)	<0.001	0.62	0.446
Propensity score-weighted with state fixed effect	5.95 (4.05 to 7.86)	<0.001	0.96	0.182

Notes: ^aThe estimated impact of the Medicaid expansion is the DID estimate of interest. ^bThe results of pre-trend tests of interaction, to examine the pre-intervention parallel trend assumption, are shown. A p value <0.05 can indicate that the pre-intervention trends are not parallel.

Abbreviations: CI, confidence interval; DID, difference-in-differences.

Figure 1 Trend in health insurance coverage in a low-income population in the United States from 2012 to 2017.

Table 3 Estimates from a Comparative Segmented Regression of ITS Analysis for Assessing the Impact of an ACR Select Tool in Increasing the Mean Appropriateness Score of High-Cost Imaging Orders

Parameter	Interpretation	Estimate (95% CI)	P value
	Pre-intervention level for control group	6.58 (6.50 to 6.66)	<0.001
	Pre-intervention trend for control group	0.01 (0.01 to 0.01)	<0.001
	Post-intervention level change for control group	−0.24 (−0.36 to −0.12)	<0.001
	Post-intervention trend change for control group	−0.01 (−0.02 to −0.01)	0.015
	Pre-intervention level difference between groups	−0.22 (−0.31 to −0.12)	<0.001
	Pre-intervention trend difference between groups	−0.01 (−0.02 to −0.01)	<0.001
	Post-intervention change in the level difference	0.63 (0.53 to 0.73)	<0.001
	Post-intervention change in the trend difference	0.02 (0.01 to 0.03)	0.019

Abbreviation: CI, confidence interval.

Figure 2 Mean scores and predicted regression line for appropriateness scores of imaging orders in an emergency department setting versus an inpatient setting over time.

Abbreviation: ED, emergency department.

Table 4 Parameter Estimates from a SARIMA Model with Intervention in Examining the Effects of eGFR Reporting with Prompts on Physician Requests for Creatinine Clearance Collection in Ontario, Canada

Parameter	Interpretation	Estimate, 95% CI	P value
	Magnitude of change	−0.93 (−1.22 to −0.64)	<0.001
	Rate of change	0.97 (0.95 to 0.99)	<0.001

Abbreviation: CI, confidence interval.

Figure 3 Monthly numbers of 24-hour creatinine clearance collections in Ontario, Canada (adjusted for age and sex) and post-intervention forecast value.