Discussion of: ‘A review of distributed statistical inference’

First of all, we would like to congratulate Dr Gao et al. for their excellent paper, which provides a comprehensive overview of amounts of existing work on distributed estimation (learning). Different from related work Gu et al. (2019); Liu et al. (2021); Verbraeken et al. (2020) that focus on computing, storage and communication architecture, the current paper leverages how to guarantee statistical efficiency of a given distributed method from a statistical viewpoint.

In the following, we divide our discussion into three parts:

1. The principle of variance-bias trade off.

2. The specific difficulty of multivariate-nonparametric distributed learning.

3. Robustness problems for various adversarial models.

1. The principle of variance-bias trade off

In the sequel, we start by general parametric models which can illustrate the interplay between the variance and the bias of an estimator.

Let ${\text{β}}_0\in {\mathbb{R}}^d$ be the true parameter that we want to learn from all the observations. We denote by $\widehat{\text{β}}$ an estimator generated by a given learning rule. According to the classical variance-bias decomposition, we informally have (1) $\Vert \widehat{\text{β}}-{\text{β}}_0\Vert \le \text{Variance}\left(\widehat{\text{β}}\right)+\text{Bias}\left(\widehat{\text{β}}\right).$(1) Variance describes how much a random variable differs from its expected value, while bias is the amount that a model's prediction differs from the target value. The correct balance of bias and variance is crucial to building statistical methodologies that create accurate results from their models.

Under the classical master/workers distributed framework, all the works with the number m produce local estimators based on their individual data, and then the master machine merges all the local estimators into a global estimator. In the case of linear Lasso estimation, the j-th worker runs the Lasso estimation to generate a local linear estimator (${\widehat{\text{β}}}_j$), and the master takes a simple average ($\overline{\text{β}}=\frac{1}{m}\sum _{j=1}^m{\widehat{\text{β}}}_j$) to form a global linear estimator.

In such an estimation procedure, each ${\widehat{\text{β}}}_j$ has a greater variance than the centralized Lasso estimator, mainly because of using a smaller sample size. It is known that the simple average can generate a global estimator with a small variance. In contrast to the variance term, the bias of the global estimator cannot be reduced significantly, since the simple average is an unbiased estimation. On one hand, this insight tells us that, the simple average strategy may be feasible for any less biased distributed estimation. On the other hand, when the bias term dominates the variance, the global estimator is no longer applicable under the distributed framework.

This paper follows the principle of variance-bias trade off and provides a full overview for classical parametric models, high-dimensional models and nonparametric models. In particular, it was shown in Zhang et al. (2013, Corollary 2) that, under appropriate regularity conditions for fixed dimensional parametric models, $\Vert \overline{\text{β}}-{\text{β}}_0\Vert =O_p\left(\frac{1}{N}\right),$ provided that the local sample size satisfies $n\gg \sqrt{N}$, where N = nm refers to the total number of sample size. This means that data splitting has no negative effect in a minimax senses and also just one-round communication among machines is enough in sense that the above divide and conquer strategy is quite communication efficient. However, these one-shot distributed learning suffer from the following three issues: (1) the size constraint $n\gg \sqrt{N}$ is often so restrictive that local machines cannot run such an amount data under some big data setting; (2) some interesting structures (e.g. sparsity) cannot be reserved well by the simple average; and (3) the achievable minimax rate is just one angle of measuring statistics, and some additional indexes such as asymptotic normality need to be considered as well in some situations.

To this end, this paper spends much spaces in reviewing a class of iterative distributed estimation, proposed originally by Shamir et al. (2014). This iterative distrusted learning can be viewed as one Newton-type approximate optimization, which makes full use of local high-order information over the master machine and the first-order information over the workers. Instead of the simple average as mentioned above, the global estimator is formulated by an objective over the master machine. Despite of some little loss on communication rounds, this iterative distributed algorithm can yield satisfactory solutions under suitable conditions, e.g., sparsity reservation and the local sample size can be taken a far smaller value than $\sqrt{N}$. Subsequently, this idea motivates the rapid development of many modern distributed algorithms.

2. The specific difficulty of multivariate-nonparametric distributed learning

For genetic multivariate-nonparametric models, developing good distributed learning algorithm has to face with some particular challenges, since those classical spline-base tool or local constant/linear smooth methods may not be a good choice. Instead, reproducing kernel methods and neural network ones are two class of popular tools for high dimensional nonparametric models.

This paper also reviews some distributed learning based on kernel methods, such as two representative work in Lin et al. (2017) and Zhang et al. (2013). Let $K(\cdot ,\cdot )$ be a given reproducing kernel and ${\hat{f}}_j$ be an nonparametric estimator at the j-th worker. Under kernel-based framework, ${\hat{f}}_j$ based on the local data $\{x_{ij},y_{ij}{\}}_{i=1}^n$ has the form ${\hat{f}}_j\left(x\right)=\sum _{i=1}^n{\alpha }_{ij}K(x_{ij},x),\mathrm{\forall }x\in \mathcal{X},$ where $\alpha =({\alpha }_{1j},\dots ,{\alpha }_{nj})\in {\mathbb{R}}^n$ is an estimated vector parameter. The global nonparametric estimator by the simple average is expressed as follows: $\overline{f}\left(x\right)=\frac{1}{m}\sum _{j=1}^m{\hat{f}}_j\left(x\right).$ Although kernel-based distributed algorithm does not result in any computational issue, the communication issue in the prediction phase becomes a major barrier observed from $\overline{f}$, where N parameters and all samples need to be passed to the host. Alternatively, every input point x is sent to all local workers, and then send back the result of ${\hat{f}}_j\left(x\right)$. However, this requires that all the workers always run normally and specially are communication efficient at any time. In other words, developing kernel-based distributed learning still has a lot of space to be explored in the future.

We discuss several related distributed design using neural networks, which has been regarded as the state of art in modern artificial intelligence. However, we observe that neural network function is highly non-convex, and so the simple average cannot make sure stationary consistency of the global estimator. On the other hand, it seems that the iterative approximate methods mentioned as above still work. Yet, the communication issues become a major bottleneck since typical neural networks contain too many weight parameters, even its size is far larger than N. Therefore, how to develop competitive distributed algorithms for neural network models is an interesting topic.

3. Robustness problems for various adversarial models

The paper gives some related important works about distributed learning in Section 4: principal component analysis, feature screening and Bootstrap. It presents a comprehensive overview on the three issues. In addition, robustness is also an important topic in the distributed inference, especially Byzantine-robust distributed learning. In distributed learning, data and computation generally come from individual computer/smartphones or worker machines of units. These devices can be easily reprogrammed and controlled by external attackers and thus behave adversarially. It may also due to crashes, faulty hardware, stalled computation or unreliable communication channels. Such worker machines are often unpredictable, which can incur major degradation in learning performance. Therefore, it is necessary to develop Byzantine-robust distributed learning algorithm to cope with adversarial attacks. For statisticians, there are some questions to consider: what kind of robust learning algorithm or the adversarial models can confront Byzantine failures, what is the best achievable statistical performance while being Byzantine-robust, how to design a learning algorithm that can achieve this performance and communication efficiency.

For adversarial models, various Byzantine-robust distributed algorithms have been considered recently, for example, Blanchard et al. (2017), Chen et al. (2017), Ghosh et al. (2020), Su and Vaidya (2016), Tu et al. (2021), Yin et al. (2018), and Zhou et al. (2021). Specially, Yin et al. (2018) proposed two robust distributed gradient descent algorithms based on coordinate-wise median and coordinate-wise trimmed mean and established order-optimal statistical error rates $\tilde{\mathcal{O}}\left(\frac{1}{\sqrt{n}}\alpha +\frac{1}{\sqrt{m}}\right)$ under mild conditions, where α is the fraction of Byzantine machines. It is a new statistical result under the robustness framework. Developing robust distributed learning algorithms cannot sacrifice the quality of learning, meanwhile pursue the best possible statistical accuracy in the presence of adversarial attacks. Robust distributed learning deserves attention and research in the era of big data.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Shaogao Lv's work is partially supported by NSFC [Grant Number 11871277]; Xingcai Zhou's work is partially supported by NSFC [Grant Number 12171242] and National Social Science Foundation of China [Grant Number 19BTJ034].