A Practical Approach to Proper Inference with Linked Data

Abstract

Entity resolution (ER), comprising record linkage and deduplication, is the process of merging noisy databases in the absence of unique identifiers to remove duplicate entities. One major challenge of analysis with linked data is identifying a representative record among determined matches to pass to an inferential or predictive task, referred to as the downstream task. Additionally, incorporating uncertainty from ER in the downstream task is critical to ensure proper inference. To bridge the gap between ER and the downstream task in an analysis pipeline, we propose five methods to choose a representative (or canonical) record from linked data, referred to as canonicalization. Our methods are scalable in the number of records, appropriate in general data scenarios, and provide natural error propagation via a Bayesian canonicalization stage. The proposed methodology is evaluated on three simulated datasets and one application – determining the relationship between demographic information and party affiliation in voter registration data from the North Carolina State Board of Elections. We first perform Bayesian ER and evaluate our proposed methods for canonicalization before considering the downstream tasks of linear and logistic regression. Bayesian canonicalization methods are empirically shown to improve downstream inference in both settings through prediction and coverage.

Keywords:

• Bayesian methods
• Canonicalization
• Entity resolution
• Error propagation
• Record linkage

Supplementary Materials

The supplement contains additional details concerning data for the simulation study and NCVD application, additional results, diagnostics, and hyperparameters for the fitted models, as well as further advice for practitioners. The R code for reproducing the results is also available.

Notes

1 Note that while there are three GeCO datasets, they only differ in the noise level for the relationship between explanatory and response variables. Thus, the noise differences do not affect the record linkage, which is performed just once with the linkage variables.

Additional information

Funding

This work was partially supported by NSF-1652431, NSF-1534412, NSF-2051911, and the Laboratory for Analytic Sciences.