Fast Bayesian Record Linkage for Streaming Data Contexts

Abstract

Record linkage is the task of combining records from multiple files which refer to overlapping sets of entities when there is no unique identifying field. In streaming record linkage, files arrive sequentially in time and estimates of links are updated after the arrival of each file. This problem arises in settings such as longitudinal surveys, electronic health records, and online events databases, among others. The challenge in streaming record linkage is to efficiently update parameter estimates as new data arrive. We approach the problem from a Bayesian perspective with estimates calculated from posterior samples of parameters and present methods for updating link estimates after the arrival of a new file that are faster than fitting a joint model with each new data file. In this article, we generalize a two-file Bayesian Fellegi-Sunter model to the multi-file case and propose two methods to perform streaming updates. We examine the effect of prior distribution on the resulting linkage accuracy as well as the computational tradeoffs between the methods when compared to a Gibbs sampler through simulated and real-world survey panel data. We achieve near-equivalent posterior inference at a small fraction of the compute time. Supplementary materials for this article are available online.

KEYWORDS:

• Bayesian online learning
• Entity resolution
• Filtering
• Streaming inference

Supplemental Materials

All supplemental materials are contained in a single compressed (zipped) archive.

Appendix to “Fast Bayesian Record Linkage for Streaming Data Contexts”: Appendices that include supplemental tables and figures; posterior and full conditional distributions; supplemental definitions, theorems, and proofs; and simulation details. (streaming-record-linkage-appendix.pdf, PDF document)

R-package for streaming record linkage: R-package “bstrl”, implementing the streaming record linkage model and the PPRB-within-Gibbs and SMCMC streaming updates. (bstrl_1.0.2.tar.gz, GNU zipped tar file)

Reproducible code repository: R code that can be used to reproduce the numerical results in this article, including tables and figures. (streamingrl-reproducible-main.zip, compressed folder)

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

The authors acknowledge the support of the Laboratory for Analytic Sciences at North Carolina State University. B. Betancourt acknowledges the support of NSF DMS-2310222.