A Principled Approach to Characterize and Analyze Partially Observed Confounder Data from Electronic Health Records

Figures & data

Figure 1 Illustration of plasmode data generating process.

Abbreviations: COI, Confounder of interest; DPP-4, dipeptidyl-peptidase-4 (DPP4i) inhibitor; MCAR, Missing completely at random; MAR, Missing at random; MNAR (unmeasured), Missing not at random; missingness depends on an unmeasured confounder; MNAR (value), Missing not at random; missingness depends on the value of the confounder of interest itself; SGLT-2, sodium-glucose-cotransporter-2 (SGLT2i).

Table 1 Overview of All Plasmode Simulation Parameters. In a First Step, Complete Plasmode Datasets with a Known Underlying Outcome-Generating Model Were Simulated Using an Empirical Exposure, Some Fully Observed Prognostic Factors (C1) and a Confounder of Interest (COI) That Missingness Was Subsequently Imposed on. For Each COI, 200 Plasmode Datasets Were Generated. The Simulated Missingness-Generating Models for the COIs Followed Four Different Missingness Mechanisms in Which the Missing Probability Was the Same for Each Patient (MCAR), Dependent on Observed C1 Covariates (MAR), Dependent on an Unmeasured Confounder (MNAR [Unmeasured]) or Dependent on the COI Itself (MNAR [Value]), Respectively. Besides the COI and the Underlying Missingness Mechanisms, Other Simulation Parameters That Were Altered Involved the Proportion of Missingness That Was Imposed on the COI, the Covariates Included in the Imputation Model and the Presence of Exposure Treatment Effect Modification by the COI

Simulation Element	Simulation Parameter
Confounder of interest (COI) and variable type	● HbA1c value in % (continuous) ● Body mass index (ordinal: underweight/normal, overweight, obese) ● Smoking (binary: current/former vs never)
C1 covariates (used for outcome generation together with exposure and COI)	● Demographics: age, sex, race, Medicare subsidy, year of index date ● Comorbidities at baseline: Combined comorbidity score,^Citation15 hyperglycemia, cardiomyopathy, atrial fibrillation, anemia, stroke/ischemia, diabetic neuropathy, diabetes with peripheral circulatory disorders, diabetic nephropathy ● Use of concomitant drugs at baseline: statin, sulfonylureas (current use), anticoagulants, antiplatelets, antihypertensive ● Healthcare utilization: number of internal physician visits, hospital admission, generics, # diabetes medications, # HbA1c tests during baseline period, # emergency room visits
Missingness mechanisms simulated	● Missing completely at random (MCAR) ● Missing at random (MAR) ● Missing not at random - MNAR(unmeasured) ● Missing not at random - MNAR(value)
# resampled datasets and sampled cohort size	200 resampled datasets per COI (100 datasets with and 100 without HTE), 1000 patients each
Missingness proportion	10 –50% (incremental increases by 10%)
Analytic approaches to address missing data	● Complete case analysis (baseline method) ● Inverse probability weighting (IPW) ● Missingness indicator ● missForest^Citation16 (single imputation random forest algorithm) Multiple imputation (m = 5 imputed datasets each): ● MICE defaults for specific variable types: Predictive mean matching (HbA1c, continuous) Proportional odds model (BMI, ordinal) Logistic regression (smoking, binary) ● Classification and regression tree (CART) ● Random forest (RF)
Covariates included in diagnostics and imputation models	● C1 covariates only ● C1 covariates + C0 covariates as auxiliary variables ● + Exposure, Outcome (ie, time-to-event and event indicator)
Modifications to causal effect estimation	± treatment effect modification by EHR COI

Abbreviations: #, Number of; COI, Confounder of interest; EHR, Electronic health records; HbA1c, Hemoglobin A1C; HTE, Heterogeneous treatment effects; MAR, Missing at random mechanism in which the missingness probability depends on observed covariates; MCAR, Missing completely at random mechanism in which each patients has the same missingness probability; MNAR(unmeasured), Missing not at random mechanism in which the missingness can only be explained by a covariate which is not observed in the underlying dataset; MNAR(value), Missing not at random mechanism in which the missingness just depends on the actual value of the partially observed confounder of interest itself; MICE, Multiple imputation by chained equations.

Figure 2 M-graphs depicting structural missing data assumptions and simulated mechanisms. (a) Missing completely at random: MCAR, (b) Missing at random: MAR), (c) Missing not at random, missingness depends on an unmeasured confounder: MNAR(unmeasured) and d) Missing not at random, missingness depends on the value of the confounder of interest itself: MNAR(value). (Notation: C1 = Fully observed C1 confounders used in outcome generation, COI = Confounder of interest [HbA1c, BMI, Smoking], COI_obs = observed portion of COI, E = Exposure, M = Missingness of COI with M=0 fully observed and M=1 fully missing, Y = Outcome).

Table 2 Diagnostics to Empirically Differentiate and Characterize Missing Data Mechanisms. The Three Group Diagnostics are Composed of Analytic Models and Tests That Contextualize and Provide Information to Differentiate and Characterize Potentially Underlying Missingness Mechanisms

	Group 1 Diagnostics		Group 2 Diagnostics	Group 3 Diagnostics
Diagnostic metric	Absolute Standardized Mean Difference (ASMD)	P-value Hoteling^Citation21/ Little^Citation22	Area Under the Receiver Operating Curve (AUC)	Log HR (Missingness Indicator)
Purpose	Comparison of distributions between patients with vs without observed value of the partially observed covariate.		Assessing the ability to predict missingness based on observed covariates.	Check whether missingness of a covariate is associated with the outcome (differential missingness).
Example value	ASMD = 0.1	p-value < 0.001	AUC = 0.5	log HR = 0.1 (0.05 to 0.2)
Interpretation	<0.1^a: no imbalances in observed patient characteristics; missingness may be likely completely at random or not at random (~MCAR, ~MNAR). >0.1^a: imbalances in observed patient characteristics; missingness may be likely at random (~MAR).	High test statistics and low p-values indicate differences in baseline covariate distributions and null hypothesis would be rejected (~MAR).	AUC values ~ 0.5 indicate completely random or not at random prediction (~MCAR, ~MNAR). Values meaningfully above 0.5 indicate stronger relationships between covariates and missingness (~MAR).	No association in either univariate or adjusted model and no meaningful difference in the log HR after full adjustment (~MCAR). Association in univariate but not fully adjusted model (~MAR). Meaningful difference in the log HR also after full adjustment (~MNAR).

Note: ^aAnalogous to propensity score-based balance measures.²³

Abbreviations: ASMD, Median absolute standardized mean difference across all covariates; AUC, Area under the curve; CI, Confidence interval; MAR, Missing at random mechanism in which the missingness probability depends on observed covariates; MCAR, Missing completely at random mechanism in which each patients has the same missingness probability; MNAR(unmeasured), Missing not at random mechanism in which the missingness can only be explained by a covariate which is not observed in the underlying dataset; MNAR(value), Missing not at random mechanism in which the missingness just depends on the actual value of the partially observed confounder of interest itself.

Table 3 Averaged Simulation Results of Missingness diagnostics Across All Simulated Plasmode Datasets

Mechanism	ASMD (95% CI)^a	p(Hotelling)^a	p(Little)^a	AUC (95% CI)^b	log HR(univariate) (95% CI)^c	log HR (95% CI)^c
MCAR	0.05 (0.05, 0.05)	0.50	0.50	0.50 (0.50, 0.50)	0.00 (−0.01, 0.00)	0.00 (−0.01, 0.00)
MAR	0.20 (0.20, 0.20)	<0.001	<0.001	0.59 (0.58, 0.59)	0.53 (0.53, 0.54)	0.00 (−0.01, 0.00)
MNAR(unmeasured)	0.09 (0.09, 0.09)	0.02	0.02	0.54 (0.54, 0.54)	0.43 (0.43, 0.44)	0.31 (0.31, 0.32)
MNAR(value)	0.06 (0.06, 0.06)	0.10	0.10	0.53 (0.53, 0.53)	0.05 (0.04, 0.05)	0.10 (0.09, 0.11)

Notes: ^aGroup 1 diagnostic: Differences in patient characteristics between patients with and without covariate. ^bGroup 2 diagnostic: Ability to predict missingness. ^cGroup 3 diagnostic: Assessment if missingness is associated with the outcome (univariate, adjusted).

Abbreviations: ASMD, Median absolute standardized mean difference across all covariates; AUC, Area under the curve; CI, Confidence interval; Confidence intervals are based on the empirical Monte-Carlo standard error.

Table 4 Averaged Simulation Results of Missingness Diagnostics by Proportion Missingness

Mechanism	ASMD (95% CI)^a	p(Hotelling)^a	p(Little)^a	AUC (95% CI)^b	log HR(univariate) (95% CI)^c	log HR (95% CI)^c
Proportion missing: 10%
MCAR	0.07 (0.07, 0.07)	0.52	0.52	0.50 (0.50, 0.50)	−0.01 (−0.02, 0.01)	0.00 (−0.02, 0.01)
MAR	0.22 (0.22, 0.23)	<0.001	<0.001	0.53 (0.53, 0.53)	0.61 (0.60, 0.62)	0.00 (−0.01, 0.02)
MNAR(unmeasured)	0.11 (0.11, 0.11)	0.06	0.06	0.52 (0.51, 0.52)	0.48 (0.47, 0.49)	0.35 (0.34, 0.37)
MNAR(value)	0.08 (0.08, 0.09)	0.18	0.18	0.52 (0.51, 0.52)	0.06 (0.05, 0.08)	0.12 (0.11, 0.14)
Proportion missing: 20%
MCAR	0.05 (0.05, 0.05)	0.50	0.50	0.50 (0.50, 0.50)	0.00 (−0.01, 0.01)	0.00 (−0.01, 0.01)
MAR	0.20 (0.20, 0.21)	<0.001	<0.001	0.56 (0.56, 0.56)	0.55 (0.54, 0.56)	0.00 (−0.02, 0.01)
MNAR(unmeasured)	0.10 (0.10, 0.10)	0.02	0.02	0.53 (0.52, 0.53)	0.45 (0.44, 0.46)	0.33 (0.32, 0.34)
MNAR(value)	0.07 (0.07, 0.07)	0.11	0.11	0.53 (0.53, 0.53)	0.05 (0.04, 0.06)	0.11 (0.10, 0.13)
Proportion missing: 30%
MCAR	0.05 (0.05, 0.05)	0.50	0.49	0.50 (0.50, 0.50)	0.00 (−0.01, 0.01)	0.00 (−0.01, 0.01)
MAR	0.19 (0.19, 0.20)	<0.001	<0.001	0.59 (0.59, 0.60)	0.52 (0.51, 0.53)	0.00 (−0.01, 0.01)
MNAR(unmeasured)	0.09 (0.09, 0.09)	0.01	0.01	0.54 (0.54, 0.54)	0.43 (0.42, 0.44)	0.32 (0.31, 0.33)
MNAR(value)	0.06 (0.06, 0.06)	0.07	0.07	0.54 (0.53, 0.54)	0.04 (0.04, 0.05)	0.09 (0.08, 0.10)
Proportion missing: 40%
MCAR	0.04 (0.04, 0.04)	0.49	0.49	0.50 (0.50, 0.50)	0.00 (−0.01, 0.00)	0.00 (−0.01, 0.01)
MAR	0.19 (0.19, 0.19)	<0.001	<0.001	0.62 (0.62, 0.62)	0.49 (0.49, 0.50)	−0.01 (−0.02, 0.01)
MNAR(unmeasured)	0.08 (0.08, 0.08)	0.01	0.01	0.55 (0.55, 0.55)	0.40 (0.39, 0.41)	0.29 (0.28, 0.30)
MNAR(value)	0.06 (0.06, 0.06)	0.07	0.07	0.54 (0.54, 0.54)	0.04 (0.03, 0.05)	0.09 (0.08, 0.10)
Proportion missing: 50%
MCAR	0.04 (0.04, 0.04)	0.50	0.50	0.51 (0.51, 0.51)	0.00 (−0.01, 0.01)	0.00 (−0.01, 0.01)
MAR	0.19 (0.18, 0.19)	<0.001	<0.001	0.63 (0.63, 0.63)	0.48 (0.47, 0.49)	0.00 (−0.01, 0.01)
MNAR(unmeasured)	0.08 (0.08, 0.08)	0.02	0.02	0.56 (0.56, 0.56)	0.39 (0.38, 0.40)	0.29 (0.28, 0.30)
MNAR(value)	0.05 (0.05, 0.05)	0.08	0.08	0.55 (0.54, 0.55)	0.04 (0.03, 0.04)	0.08 (0.07, 0.09)

Notes: ^aGroup 1 diagnostic: Differences in patient characteristics between patients with and without covariate. ^bGroup 2 diagnostic: Ability to predict missingness. ^cGroup 3 diagnostic: Assessment if missingness is associated with the outcome (univariate, adjusted).

Abbreviation: ASMD, Median absolute standardized mean difference across all covariates; AUC, Area under the curve; CI, Confidence interval.

Figure 3 Panel illustrating (a) root mean squared error (RMSE), (b) % bias and (c) Variance for each analytic missing data method averaged across all simulation scenarios.

Figure 4 Root mean square error (RMSE) metrics by simulated missingness mechanism.

Table 5 Missingness Diagnostics of the Exemplary Base Case Study for Three Blood Laboratory Measurements Representing a Patient’s Baseline Kidney Function. Due to the Expected Monotone Missingness Pattern, the Missingness Diagnostics for One EHR Lab Were Performed in Absence of the Other Two EHR Labs, Except for Little’s Test Which Globally Tests for Presence of MCAR Considering All EHR Labs Jointly

Covariate	ASMD (Min/Max)^a	p(Hotelling)^a	AUC^b	Log HR(Univariate) (95% CI)^c	log HR (95% CI)^c
Blood urea nitrogen (BUN)	0.056 (0.001, 0.389)	<0.001	0.621	−0.13 (95% CI −0.23, −0.02)	0.03 (95% CI −0.08, 0.14)
Serum creatinine	0.056 (0.000, 0.380)	<0.001	0.622	−0.13 (95% CI −0.24, −0.02)	0.03 (95% CI −0.09, 0.14)
Estimated glomerular filtration rate (eGFR)	0.047 (0.001, 0.359)	<0.001	0.611	−0.13 (95% CI −0.24, −0.03)	0.01 (95% CI −0.11, 0.12)

Notes: p little: <0.001, ^aGroup 1 diagnostic: Differences in patient characteristics between patients with and without covariate. ^bGroup 2 diagnostic: Ability to predict missingness. ^cGroup 3 diagnostic: Assessment if missingness is associated with the outcome (univariate, adjusted).

Abbreviations: ASMD, Median absolute standardized mean difference across all covariates; AUC, Area under the curve; beta, beta coefficient; CI, Confidence interval; max, Maximum; min, Minimum.

Figure 5 Missingness pattern²⁵ of Blood urea nitrogen (bun_results_NA), estimated glomerular filtration rate (egfr_results_NA) and serum creatinine (creatinine_result_NA) in the base case cohort. The set size displays the count of missing observations for each lab result individually while the intersection size illustrates the count of intersecting missing observations across the three labs.

Figure 6 Comparison of results from the empirical case example assessing opioid versus non-steroidal anti-inflammatory drug use on the time to acute kidney injury dependent on level of adjustment.

Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S. A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol. 2011;64(7):749–759. doi:10.1016/j.jclinepi.2010.10.004

 PubMed Web of Science ®Google Scholar

Stekhoven DJ, Bühlmann P. MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics. 2012;28(1):112–118. doi:10.1093/bioinformatics/btr597

 PubMed Web of Science ®Google Scholar

Hotelling H. The Generalization of Student’s Ratio. Ann Math Stat. 1931;2(3):360–378. doi:10.1214/aoms/1177732979

 Google Scholar

Little RJA. A test of missing completely at random for multivariate data with missing values. J Am Stat Assoc. 1988;83(404):1198–1202. doi:10.1080/01621459.1988.10478722

 Web of Science ®Google Scholar

Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar Behav Res. 2011;46(3):399–424. doi:10.1080/00273171.2011.568786

 PubMed Web of Science ®Google Scholar

Ruddle RA, Adnan M, Hall M. Using set visualisation to find and explain patterns of missing values: a case study with NHS hospital episode statistics data. BMJ Open. 2022;12(11):e064887. doi:10.1136/bmjopen-2022-064887

 PubMed Web of Science ®Google Scholar

Data Sharing Statement

Patient-level data cannot be shared due to restriction under our data use agreement with CMS. However, to test the missing data diagnostics and imputation analyses, the smdi R package includes a simulated dataset (smdi_data) with similar missingness structures as the data used in this manuscript. Patient cohorts used in this study were queried using SAS version 9.4 and all analyses were conducted in R version 4.1.2. Detailed information on used packages and versions can be found in the Supplementary Methods and code used in this study is available at https://gitlab-scm.partners.org/drugepi/missingehr. The missing data diagnostics presented in this manuscript can be implemented using the smdi R package, available from https://janickweberpals.gitlab-pages.partners.org/smdi/ and CRAN via install.packages(“smdi”).