Method G: Uncertainty Quantification for Distributed Data Problems Using Generalized Fiducial Inference

Abstract

It is not unusual for a data analyst to encounter datasets distributed across several computers. This can happen for reasons such as privacy concerns, efficiency of likelihood evaluations, or just the sheer size of the whole dataset. This presents new challenges to statisticians as even computing simple summary statistics such as the median becomes computationally challenging. Furthermore, if other advanced statistical methods are desired, then novel computational strategies are needed. In this article, we propose a new approach for distributed analysis of massive data that is suitable for generalized fiducial inference and is based on a careful implementation of a “divide-and-conquer” strategy combined with importance sampling. The proposed approach requires only small amount of communication between nodes, and is shown to be asymptotically equivalent to using the whole dataset. Unlike most existing methods, the proposed approach produces uncertainty measures (such as confidence intervals) in addition to point estimates for parameters of interest. The proposed approach is also applied to the analysis of a large set of solar images. Supplementary materials for this article are available online.

Keywords:

• Divide and conquer
• Importance sampling
• Massive data
• Parallel processing
• Solar imaging

Supplementary Material

An R implementation of this algorithm can be available as a supplementary material.

Division of Mathematical Sciences;Division of Information and Intelligent Systems;

Acknowledgments

The authors are most grateful to the reviewers and the associate editor for their most constructive comments which led to a much improved version of the article.

Additional information

Funding

Hannig’s research was supported by the National Science Foundation under grant nos. IIS-1633074 and DMS-1916115. Lee’s research was supported by the National Science Foundation under grant nos. DMS-1811405, DMS-1811661 and DMS-1916125.