Multifile Partitioning for Record Linkage and Duplicate Detection

Abstract

Merging datafiles containing information on overlapping sets of entities is a challenging task in the absence of unique identifiers, and is further complicated when some entities are duplicated in the datafiles. Most approaches to this problem have focused on linking two files assumed to be free of duplicates, or on detecting which records in a single file are duplicates. However, it is common in practice to encounter scenarios that fit somewhere in between or beyond these two settings. We propose a Bayesian approach for the general setting of multifile record linkage and duplicate detection. We use a novel partition representation to propose a structured prior for partitions that can incorporate prior information about the data collection processes of the datafiles in a flexible manner, and extend previous models for comparison data to accommodate the multifile setting. We also introduce a family of loss functions to derive Bayes estimates of partitions that allow uncertain portions of the partitions to be left unresolved. The performance of our proposed methodology is explored through extensive simulations. Supplementary materials for this article are available online.

Keywords:

• Data matching
• Data merging
• Entity resolution
• Microclustering

Supplementary Material

Appendices: Supplementary appendices A–E. (.pdf file)

R-package multilink: Code implementing the proposed methodology. (.tar.gz file)

Simulation code and data: Code and data used to run simulations. (.zip file)

Acknowledgments

The authors thank Jorge A. Restrepo for providing the Colombian homicide data.

Additional information

Funding

This research was supported by the NSF under grant SES-1852841 and by the NIH–NIDA under award R21DA051756. The authors thank Jorge A. Restrepo for providing the Colombian homicide data.