A review of distributed statistical inference

Abstract

The rapid emergence of massive datasets in various fields poses a serious challenge to traditional statistical methods. Meanwhile, it provides opportunities for researchers to develop novel algorithms. Inspired by the idea of divide-and-conquer, various distributed frameworks for statistical estimation and inference have been proposed. They were developed to deal with large-scale statistical optimization problems. This paper aims to provide a comprehensive review for related literature. It includes parametric models, nonparametric models, and other frequently used models. Their key ideas and theoretical properties are summarized. The trade-off between communication cost and estimate precision together with other concerns is discussed.

Keywords:

• Distributed computing
• divide-and-conquer
• communication-efficiency
• shrinkage methods
• nonparametric estimation
• principal component analysis
• feature screening
• bootstrap

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work is supported by National Natural Science Foundation of China (No. 11971171), the 111 Project (B14019) and Project of National Social Science Fund of China (15BTJ027). Weidong Liu's research is supported by National Program on Key Basic Research Project (973 Program, 2018AAA0100704), National Natural Science Foundation of China (No. 11825104, 11690013), Youth Talent Support Program, and a grant from Australian Research Council. Hansheng Wang's research is partially supported by National Natural Science Foundation of China (No. 11831008, 11525101, 71532001). It is also supported in part by China's National Key Research Special Program (No. 2016YFC0207704).

Notes on contributors

Yuan Gao

Mr. Yuan Gao is a Ph.D. candidate in school of statistics at East China Normal University.

Weidong Liu

Dr. Weidong Liu is the Distinguished Professor in school of mathematical sciences at Shanghai Jiao Tong University.

Hansheng Wang

Dr. Hansheng Wang is a professor in Guanghua School of Management at Peking University.

Xiaozhou Wang

Dr. Xiaozhou Wang is an assistant professor in school of statistics at East China Normal University.

Yibo Yan

Mr. Yibo Yan is a Ph.D. candidate in school of statistics at East China Normal University.

Riquan Zhang

Dr. Riquan Zhang is a professor in school of statistics at East China Normal University.