Using Ensemble-Based Methods for Directly Estimating Causal Effects: An Investigation of Tree-Based G-Computation

Figures & data

TABLE 1 Monte Carlo Simulations: Relative Bias (%) in Estimated Average Treatment Effect (ATE) for Continuous Outcomes Using Different Methods Across the Seven Scenarios

Estimation Method	A	C	E	F	G	E2	G2
Regression methods to directly impute potential outcomes
Crude	−44.3	−37.7	−44.1	−46.8	−38.3	−85.3	−129.4
Bagging	−7.1	−6.0	−6.9	−6.2	−5.1	−25.0	−32.3
Random forests	−8.6	−7.6	−8.0	−7.8	−6.7	−30.2	−44.0
Boosting—interaction depth 1	−10.5	−9.2	−10.0	−9.5	−8.1	−34.1	−47.8
Boosting—interaction depth 2	−5.0	−4.6	−4.5	−3.5	−2.8	−25.2	−34.1
Boosting—interaction depth 3	−3.8	−3.6	−3.3	−2.1	−1.5	−22.2	−29.2
Boosting—interaction depth 4	−3.3	−3.0	−2.9	−1.6	−1.1	−20.8	−26.9
Imbens method	−1.4	−1.3	−1.0	0.2	0.0	−50.8	−77.4
G-computation	−1.4	−1.3	−1.1	0.1	0.1	−47.0	−73.7
Inverse probability of treatment weighting
Bagging	−15.2	−13.8	−57.1	6.8	11.2	−85.6	−31.6
Random forests	−37.2	−41.4	−116.7	29.3	30.3	−143.2	−11.5
Boosting—interaction depth 1	−18.4	−8.0	−47.4	−5.7	5.0	−82.4	−51.3
Boosting—interaction depth 2	−11.9	−5.6	−48.6	1.0	10.0	−77.5	−35.4
Boosting—interaction depth 3	−11.6	−6.5	−51.8	4.7	12.0	−77.5	−27.9
Boosting—interaction depth 4	−12.9	−8.7	−55.7	7.5	13.7	−79.9	−23.6
Logistic regression	−1.4	26.0	−24.1	−24.9	−13.4	−78.7	−96.0

TABLE 2 Monte Carlo Simulations: Standard Deviation of Estimated Average Treatment Effect (ATE) for Continuous Outcomes Using Different Methods Across the Seven Scenarios

Estimation Method	A	C	E	F	G	E2	G2
Regression methods to directly impute potential outcomes
Crude	0.1262	0.1285	0.1231	0.1249	0.1290	0.1608	0.2278
Bagging	0.1316	0.1339	0.1350	0.1358	0.1387	0.1746	0.2425
Random forests	0.1276	0.1320	0.1307	0.1325	0.1344	0.1703	0.2383
Boosting—interaction depth 1	0.1265	0.1297	0.1279	0.1300	0.1312	0.1682	0.2326
Boosting—interaction depth 2	0.1287	0.1337	0.1312	0.1314	0.1344	0.1719	0.2380
Boosting—interaction depth 3	0.1298	0.1357	0.1328	0.1323	0.1360	0.1739	0.2408
Boosting—interaction depth 4	0.1304	0.1369	0.1338	0.1329	0.1372	0.1749	0.2428
Imbens method	0.1319	0.1301	0.1353	0.1361	0.1330	0.1813	0.2412
G-computation	0.1318	0.1293	0.1339	0.1357	0.1312	0.1795	0.2374
Inverse probability of treatment weighting
Bagging	0.1318	0.1435	0.1352	0.1338	0.1368	0.1686	0.2187
Random forests	0.1412	0.1447	0.1432	0.1432	0.1464	0.1642	0.1949
Boosting—interaction depth 1	0.1249	0.1282	0.1252	0.1246	0.1252	0.1622	0.2128
Boosting—interaction depth 2	0.1294	0.1334	0.1324	0.1317	0.1300	0.1692	0.2155
Boosting—interaction depth 3	0.1294	0.1360	0.1351	0.1336	0.1331	0.1712	0.2174
Boosting—interaction depth 4	0.1287	0.1367	0.1354	0.1334	0.1348	0.1706	0.2177
Logistic regression	0.1870	0.1629	0.2032	0.1982	0.1679	0.2567	0.2790

TABLE 3 Monte Carlo Simulations: MSE of Estimated Average Treatment Effect (ATE) for Continuous Outcomes Using Different Methods Across the Seven Scenarios

Estimation Method	A	C	E	F	G	E2	G2
Regression methods to directly impute potential outcomes
Crude	0.0473	0.0392	0.0462	0.0506	0.0400	0.1423	0.3199
Bagging	0.0181	0.0185	0.0190	0.0190	0.0196	0.0405	0.0755
Random forests	0.0175	0.0183	0.0181	0.0185	0.0188	0.0436	0.0878
Boosting—interaction depth 1	0.0177	0.0182	0.0180	0.0183	0.0182	0.0468	0.0906
Boosting—interaction depth 2	0.0170	0.0182	0.0175	0.0175	0.0182	0.0397	0.0752
Boosting—interaction depth 3	0.0171	0.0186	0.0178	0.0176	0.0185	0.0381	0.0715
Boosting—interaction depth 4	0.0172	0.0189	0.0180	0.0177	0.0188	0.0375	0.0705
Imbens method	0.0174	0.0169	0.0183	0.0185	0.0177	0.0742	0.1539
G-computation	0.0174	0.0167	0.0179	0.0184	0.0172	0.0675	0.1431
Inverse probability of treatment weighting
Bagging	0.0210	0.0236	0.0703	0.0186	0.0207	0.1457	0.0637
Random forests	0.0420	0.0483	0.2385	0.0342	0.0360	0.3552	0.0401
Boosting—interaction depth 1	0.0210	0.0174	0.0517	0.0160	0.0160	0.1348	0.0873
Boosting—interaction depth 2	0.0190	0.0183	0.0553	0.0174	0.0185	0.1247	0.0665
Boosting—interaction depth 3	0.0189	0.0192	0.0611	0.0182	0.0200	0.1253	0.0596
Boosting—interaction depth 4	0.0192	0.0199	0.0680	0.0187	0.0211	0.1312	0.0562
Logistic regression	0.0350	0.0373	0.0506	0.0492	0.0310	0.1649	0.2251

TABLE 4 Monte Carlo Simulations: Relative Bias (%) in Estimated Average Treatment Effect (ATE) for Binary Outcomes Using Different Methods Across the Seven Scenarios

Estimation Method	A	C	E	F	G	E2	G2
Regression methods to directly impute potential outcomes
Crude	−129.9	−154.7	−139.7	−129.4	−153.1	−112.7	−89.3
Bagging	−4.0	2.4	−18.8	9.3	9.2	−45.7	−43.6
Random forests	−3.7	14.6	−22.1	5.3	15.9	−48.2	−29.5
Boosting—interaction depth 1	−8.8	−5.8	−16.8	−1.8	−3.1	−44.6	−56.2
Boosting—interaction depth 2	−3.6	2.4	−8.1	−0.3	1.6	−32.6	−43.8
Boosting—interaction depth 3	−4.9	3.5	−5.4	−4.1	0.0	−28.1	−42.6
Boosting—interaction depth 4	−7.1	3.0	−3.8	−8.5	−3.1	−25.7	−44.8
Imbens method	−0.6	5.2	−3.1	−0.7	2.2	−51.4	−73.3
G-computation	−0.5	4.3	−1.9	−1.3	0.6	−47.6	−72.6
Inverse probability of treatment weighting
Bagging	−32.1	−19.8	−9.3	−44.0	−35.9	−32.1	−72.9
Random forests	−77.4	−61.3	−36.5	−107.9	−94.8	−52.7	−129.7
Boosting—interaction depth 1	−165.6	−176.7	−184.4	−162.4	−171.1	−154.1	−122.5
Boosting—interaction depth 2	−175.2	−179.2	−192.7	−169.9	−170.4	−168.4	−130.9
Boosting—interaction depth 3	−174.6	−177.8	−193.1	−167.5	−166.8	−173.0	−133.1
Boosting—interaction depth 4	−171.8	−175.4	−191.3	−162.3	−162.2	−173.5	−131.4
Logistic regression	−2.2	7.2	25.3	25.2	43.3	−39.3	−53.9

TABLE 5 Monte Carlo Simulations: Standard Deviation of Estimated Average Treatment Effect (ATE) for Binary Outcomes Using Different Methods Across the Seven Scenarios

Estimation Method	A	C	E	F	G	E2	G2
Regression methods to directly impute potential outcomes
Crude	0.0176	0.0171	0.0179	0.0180	0.0174	0.0172	0.0173
Bagging	0.0195	0.0193	0.0198	0.0202	0.0189	0.0194	0.0182
Random forests	0.0206	0.0203	0.0210	0.0212	0.0201	0.0204	0.0194
Boosting—interaction depth 1	0.0191	0.0183	0.0190	0.0196	0.0183	0.0187	0.0176
Boosting—interaction depth 2	0.0197	0.0193	0.0198	0.0202	0.0190	0.0193	0.0183
Boosting—interaction depth 3	0.0200	0.0197	0.0202	0.0205	0.0194	0.0197	0.0188
Boosting—interaction depth 4	0.0202	0.0199	0.0204	0.0208	0.0197	0.0199	0.0191
Imbens method	0.0201	0.0184	0.0200	0.0207	0.0188	0.0200	0.0181
G-computation	0.0200	0.0182	0.0201	0.0204	0.0187	0.0197	0.0180
Inverse probability of treatment weighting
Bagging	0.0168	0.0165	0.0167	0.0171	0.0162	0.0159	0.0158
Random forests	0.0119	0.0113	0.0120	0.0121	0.0116	0.0115	0.0113
Boosting—interaction depth 1	0.0173	0.0163	0.0172	0.0180	0.0164	0.0166	0.0163
Boosting—interaction depth 2	0.0176	0.0165	0.0176	0.0183	0.0165	0.0169	0.0162
Boosting—interaction depth 3	0.0175	0.0164	0.0176	0.0181	0.0165	0.0169	0.0161
Boosting—interaction depth 4	0.0172	0.0162	0.0174	0.0177	0.0163	0.0166	0.0159
Logistic regression	0.0215	0.0197	0.0229	0.0257	0.0224	0.0211	0.0200

TABLE 6 Monte Carlo Simulations: MSE of Estimated Average Treatment Effect (ATE) for Binary Outcomes Using Different Machine Learning Methods Across the Seven Scenarios

Estimation Method	A	C	E	F	G	E2	G2
Regression methods to directly impute potential outcomes
Crude	0.00099	0.00125	0.00110	0.00099	0.00124	0.00080	0.00062
Bagging	0.00038	0.00037	0.00041	0.00041	0.00036	0.00046	0.00041
Random forests	0.00042	0.00042	0.00046	0.00045	0.00041	0.00051	0.00041
Boosting—interaction depth 1	0.00037	0.00034	0.00037	0.00039	0.00033	0.00043	0.00044
Boosting—interaction depth 2	0.00039	0.00037	0.00039	0.00041	0.00036	0.00042	0.00041
Boosting—interaction depth 3	0.00040	0.00039	0.00041	0.00042	0.00038	0.00042	0.00042
Boosting—interaction depth 4	0.00041	0.00040	0.00042	0.00043	0.00039	0.00042	0.00044
Imbens method	0.00040	0.00034	0.00040	0.00043	0.00035	0.00051	0.00054
G-computation	0.00040	0.00033	0.00040	0.00041	0.00035	0.00048	0.00053
Inverse probability of treatment weighting
Bagging	0.00032	0.00029	0.00028	0.00037	0.00031	0.00029	0.00046
Random forests	0.00038	0.00028	0.00020	0.00061	0.00049	0.00024	0.00080
Boosting—interaction depth 1	0.00140	0.00152	0.00166	0.00138	0.00144	0.00122	0.00087
Boosting—interaction depth 2	0.00154	0.00156	0.00179	0.00149	0.00143	0.00142	0.00095
Boosting—interaction depth 3	0.00152	0.00153	0.00180	0.00145	0.00139	0.00148	0.00097
Boosting—interaction depth 4	0.00147	0.00149	0.00177	0.00137	0.00132	0.00148	0.00094
Logistic regression	0.00046	0.00039	0.00055	0.00068	0.00058	0.00051	0.00051

TABLE 7 Baseline Characteristics of Treated and Untreated Subjects in the Study Sample

Variable	No Smoking Cessation Counseling (N = 754)	Smoking Cessation Counseling (N = 1,588)	Overall Sample (N = 2,342)	Absolute Standardized Difference of the Mean	p Value
Demographic characteristics
Age	60.48 ± 13.26	56.24 ± 11.26	57.61 ± 12.10	0.35	< .001
Female	220 (29.2%)	397 (25.0%)	617 (26.3%)	0.09	.032
Note. Continuous variables are presented as means ± standard deviation; dichotomous variables are presented as N (%); ASA = acetylsalicylic acid.
Presenting signs and symptoms
Acute pulmonary edema	34 (4.5%)	48 (3.0%)	82 (3.5%)	0.08	.067
Vital signs on admission
Systolic blood pressure	146.99 ± 31.82	146.93 ± 29.92	146.95 ± 30.53	0.00	.966
Diastolic blood pressure	84.81 ± 18.99	85.84 ± 18.51	85.50 ± 18.67	0.06	.213
Heart rate	83.28 ± 22.75	81.10 ± 22.54	81.80 ± 22.63	0.10	.029
Respiratory rate	21.18 ± 5.75	20.18 ± 4.64	20.50 ± 5.05	0.20	< .001
Classic cardiac risk factors
Diabetes	179 (23.7%)	260 (16.4%)	439 (18.7%)	0.19	< .001
Hyperlipidemia	238 (31.6%)	539 (33.9%)	777 (33.2%)	0.05	.254
Hypertension	295 (39.1%)	541 (34.1%)	836 (35.7%)	0.11	.017
Family history of coronary artery disease	253 (33.6%)	754 (47.5%)	1,007 (43.0%)	0.28	< .001
Comorbid conditions and vascular history
Cerebrovascular accident/Transient ischemic attack	62 (8.2%)	67 (4.2%)	129 (5.5%)	0.18	<.001
Angina	198 (26.3%)	412 (25.9%)	610 (26.0%)	0.01	.871
Cancer	22 (2.9%)	20 (1.3%)	42 (1.8%)	0.13	.005
Dementia	21 (2.8%)	6 (0.4%)	27 (1.2%)	0.23	< .001
Previous myocardial infarction	161 (21.4%)	241 (15.2%)	402 (17.2%)	0.16	< .001
Asthma	40 (5.3%)	98 (6.2%)	138 (5.9%)	0.04	.406
Depression	76 (10.1%)	131 (8.2%)	207 (8.8%)	0.06	.145
Peptic ulcer disease	39 (5.2%)	111 (7.0%)	150 (6.4%)	0.07	.093
Peripheral vascular disease	77 (10.2%)	90 (5.7%)	167 (7.1%)	0.18	< .001
Previous coronary revascularization	50 (6.6%)	92 (5.8%)	142 (6.1%)	0.04	.427
Chronic congestive heart failure	24 (3.2%)	24 (1.5%)	48 (2.0%)	0.12	.008
Laboratory tests
Glucose	9.35 ± 5.63	8.57 ± 4.79	8.82 ± 5.09	0.15	< .001
White blood count	11.01 ± 4.49	10.77 ± 3.55	10.85 ± 3.88	0.06	.171
Hemoglobin	141.71 ± 19.33	145.83 ± 15.47	144.50 ± 16.92	0.24	< .001
Sodium	138.75 ± 4.54	139.40 ± 3.32	139.19 ± 3.77	0.17	< .001
Potassium	4.10 ± 0.58	4.01 ± 0.49	4.04 ± 0.52	0.16	< .001
Creatinine	99.59 ± 62.86	89.24 ± 30.24	92.57 ± 43.75	0.24	< .001
Prescriptions for cardiovascular medications at hospital discharge
Statin	193 (25.6%)	637 (40.1%)	830 (35.4%)	0.31	< .001
Beta-blocker	460 (61.0%)	1,192 (75.1%)	1,652 (70.5%)	0.31	< .001
ACE inhibitor/Angiotensin receptor blockers	344 (45.6%)	850 (53.5%)	1,194 (51.0%)	0.16	< .001
Plavix	29 (3.8%)	74 (4.7%)	103 (4.4%)	0.04	.37
ASA	544 (72.1%)	1,341 (84.4%)	1,885 (80.5%)	0.31	< .001

TABLE 8 Estimated Effects of Smoking Cessation Counseling on 3-year Mortality

Regression Method	ATE	ATT	ATC
Regression methods to directly impute potential outcomes
Logistic regression (Imbens)	−0.023 (−0.05, 0.003)	−0.015 (−0.039, 0.009)	−0.042 (−0.079, −0.004)
Logistic regression (G-computation)	−0.027 (−0.054, 0)	−0.023 (−0.047, 0)	−0.034 (−f0.068, −0.001)
Bagging	−0.036 (−0.06, −0.011)	−0.028 (−0.052, −0.005)	−0.052 (−0.083, −0.02)
Boosting—interaction depth 1	−0.046 (−0.072, −0.02)	−0.04 (−0.065, −0.016)	−0.058 (−0.09, −0.026)
Boosting—interaction depth 2	−0.035 (−0.06, −0.01)	−0.027 (−0.051, −0.004)	−0.051 (−0.084, −0.018)
Boosting—interaction depth 3	−0.031 (−0.056, −0.006)	−0.023 (−0.046, 0)	−0.048 (−0.081, −0.015)
Boosting—interaction depth 4	−0.029 (−0.053, −0.004)	−0.02 (−0.043, 0.002)	−0.046 (−0.079, −0.013)
Random forests	−0.028 (−0.052, −0.003)	−0.022 (−0.045, 0.001)	−0.040 (−0.072, −0.008)
Note. ATE = average treatment effect; ATT = average treatment effect in the treated; ATC = average treatment effect in the controls.
Inverse probability of treatment weighting using the propensity score
Logistic regression	−0.025 (−0.053, 0.003)	−0.011 (−0.027, 0.005)	−0.014 (−0.028, 0.001)
Bagging:	−0.046 (−0.07, −0.023)	−0.022 (−0.036, −0.007)	−0.024 (−0.035, −0.014)
Boosting—interaction depth 1	−0.037 (−0.062, −0.012)	−0.015 (−0.031, 0)	−0.022 (−0.033, −0.011)
Boosting—interaction depth 2	−0.028 (−0.052, −0.004)	−0.007 (−0.022, 0.007)	−0.021 (−0.031, −0.01)
Boosting—interaction depth 3	−0.025 (−0.048, −0.002)	−0.003 (−0.017, 0.01)	−0.022 (−0.032, −0.011)
Boosting—interaction depth 4	−0.023 (−0.045, −0.001)	0 (−0.013, 0.013)	−0.023 (−0.034, −0.013)
Random forests	−0.007 (−0.022, 0.008)	0.045 (0.035, 0.054)	−0.052 (−0.062, −0.041)