Characterizing variation of nonparametric random probability measures using the Kullback–Leibler divergence

ABSTRACT

This work characterizes the dispersion of some popular random probability measures, including the bootstrap, the Bayesian bootstrap, and the Pólya tree prior. This dispersion is measured in terms of the variation of the Kullback–Leibler divergence of a random draw from the process to that of its baseline centring measure. By providing a quantitative expression of this dispersion around the baseline distribution, our work provides insight for comparing different parameterizations of the models and for the setting of prior parameters in applied Bayesian settings. This highlights some limitations of the existing canonical choice of parameter settings in the Pólya tree process.

KEYWORDS:

• Bayesian nonparametrics
• Kullback–Leibler divergence
• bootstrap methods
• Pólyatrees

Acknowledgments

We are grateful to Judith Rousseau for helpful comments. This work was done whilst Nieto-Barajas was visiting the Department of Statistics at the University of Oxford.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. The digamma function is defined as the logarithmic derivative of the gamma function, that is, ${\psi }_0\left(x\right)=\left(\mathrm{d}/\mathrm{d}x\right)\log \mathrm{\Gamma }\left(x\right)={\mathrm{\Gamma }}^{\prime }\left(x\right)/\mathrm{\Gamma }\left(x\right)$. In similar fashion, the trigamma function is defined as the second derivative.

2. The variance of each element is defined in terms of first and second moments and rely on independence properties to compute them. Working out the algebra with patience and noting that $\mathrm{E}\left\{\right(\log Y_{mj}\left)Y_{mj}\right\}={\lambda }_2\left(m\right)/2$, $\mathrm{E}\left\{\right(\log Y_{mj}\left)Y_{mj}^2\right\}=\frac{1}{2}\left(\right(\alpha \rho \left(m\right)+1\left)/\right(2\alpha \rho \left(m\right)+1\left)\right){\lambda }_3\left(m\right)$, $\mathrm{E}\left\{\right(\log Y_{mj}\left)Y_{mj}\right(1-Y_{mj}\left)\right\}=\frac{1}{2}\left(\alpha \rho \left(m\right)/\right(2\alpha \rho \left(m\right)+1\left)\right){\lambda }_4\left(m\right)$, $\mathrm{E}\left\{\right(\log Y_{mj}{)}^2{Y}_{mj}^2\}=\frac{1}{2}(\left(\alpha \rho \left(m\right)+1\right)/\left(2\alpha \rho \left(m\right)+1\right)\left){\lambda }_5\right(m)$ and $\mathrm{E}\left\{\right(\log Y_{mj})\log (1-Y_{mj}\left)Y_{mj}\right(1-Y_{mj}\left)\right\}=\frac{1}{2}\left(\alpha \rho \left(m\right)/\right(2\alpha \rho \left(m\right)+1\left)\right){\lambda }_6\left(m\right)$, the result is obtained.

3. Figure 1 appears to show that $\mathrm{E}\left\{\mathrm{KL}\right(f||f_0\left)\right\}$ remains constant, but this is an artefact due to the scale.

4. We use this notation to emphasize the fact that $\mathbf{w}$ represents a random probability mass function, but taking values on the set $\{0,1/n,2/n,\dots ,1\}$. A factor of n is needed for the vector to be distributed according to a multinomial distribution.

5. It is interesting to note that in the original work they only consider this special case.

6. $\mathrm{Var}\left(w_i\right)=p_i(1-p_i)/({\alpha }_n+1)$, $\mathrm{Cov}(w_i,w_j)=-p_i{p}_j/({\alpha }_n+1)$, $\mathrm{Var}(w_i\log w_i)=p_i({\alpha }_n{p}_i+1)/({\alpha }_n+1)\left\{{\psi }_1\right({\alpha }_n{p}_i+2)-{\psi }_1({\alpha }_n+2)+[{\psi }_0({\alpha }_n{p}_i+2)-{\psi }_0({\alpha }_n+2){]}^2\}-p_i^2\{{\psi }_0({\alpha }_n{p}_i+1)-{\psi }_0({\alpha }_n+1){\}}^2$, $\mathrm{Cov}(w_i\log w_i,w_i)=p_i({\alpha }_n{p}_i+1)/({\alpha }_n+1)\left\{{\psi }_0\right({\alpha }_n{p}_i+2)-{\psi }_0({\alpha }_n+2\left)\right\}-p_i^2\left\{{\psi }_0\right({\alpha }_n{p}_i+1)-{\psi }_0({\alpha }_n+1\left)\right\}$, $\mathrm{Cov}(w_i\log w_i,w_j)=p_i{p}_j\{-{\psi }_0({\alpha }_n{p}_i+1)/({\alpha }_n+1)+{\psi }_0({\alpha }_n+1)-{\alpha }_n{\psi }_0({\alpha }_n+2)/({\alpha }_n+1)\}$, $\mathrm{Cov}(w_i\log w_i,w_j\log w_j)={\alpha }_n{p}_i{p}_j/({\alpha }_n+1)\left[\right\{{\psi }_0({\alpha }_n{p}_i+1)-{\psi }_0({\alpha }_n+2)\left\}\right\{{\psi }_0({\alpha }_n{p}_j+1)-{\psi }_0({\alpha }_n+2)\}-{\psi }_1({\alpha }_n+2\left)\right]-p_i{p}_j\left\{{\psi }_0\right({\alpha }_n{p}_i+1)-{\psi }_0({\alpha }_n+1\left)\right\}\left\{{\psi }_0\right({\alpha }_n{p}_j+1)-{\psi }_0({\alpha }_n+1\left)\right\}$.

7. Proposition 3 of Ferguson [4] states that any fixed density (measure) g absolutely continuous with respect to $f_0$ can be arbitrarily approximated pointwise with a draw f from a Dirichlet process.

Additional information

Funding

Watson is supported by the Industrial Doctoral Training Centre (SABS-IDC) at Oxford University and Hoffman-La Roche. Nieto-Barajas is supported by CONACYT grant 244459 and Asociación Mexicana de Cultura, A.C.–Mexico. Holmes gratefully acknowledges support for this research from the Oxford-Man Institute, the EPSRC i-Like programme grant EP/K014463/1 and the Medical Research Council program leader's award MC_UP_A390_1107.