Asymptotically Exact Data Augmentation: Models, Properties, and Algorithms

References

• Aguirregabiria, J. M. , Hernández, A. , and Rivas, M. (2002), “δ-Function Converging Sequences,” American Journal of Physics , 70, 180–185.
   Web of Science ®Google Scholar
• Albert, J. H. , and Chib, S. (1993), “Bayesian Analysis of Binary and Polychotomous Response Data,” Journal of the American Statistical Association , 88, 669–679.
   Web of Science ®Google Scholar
• Baragatti, M. , Grimaud, A. , and Pommeret, D. (2013), “Likelihood-Free Parallel Tempering,” Statistics and Computing , 23, 535–549.
   Web of Science ®Google Scholar
• Barbos, A.-C. , Caron, F. , Giovannelli, J.-F. , and Doucet, A. (2017), “Clone MCMC: Parallel High-Dimensional Gaussian Gibbs Sampling,” in Advances in Neural Information Processing Systems , pp. 5020–5028.
   Google Scholar
• Beaumont, M. A. , Zhang, W. , and Balding, D. J. (2002), “Approximate Bayesian Computation in Population Genetics,” Genetics , 162, 2025–2035.
   PubMed Web of Science ®Google Scholar
• Beck, A. , and Teboulle, M. (2009), “A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems,” SIAM Journal on Imaging Sciences , 2, 183–202.
   Web of Science ®Google Scholar
• Ben-Tal, A. , Margalit, T. , and Nemirovski, A. (2001), “The Ordered Subsets Mirror Descent Optimization Method With Applications to Tomography,” SIAM Journal on Optimization , 12, 79–108.
   Web of Science ®Google Scholar
• Besag, J. , and Green, P. J. (1993), “Spatial Statistics and Bayesian Computation,” Journal of the Royal Statistical Society, Series B, 55, 25– 37.
   Google Scholar
• Boyd, S. , Parikh, N. , Chu, E. , Peleato, B. , and Eckstein, J. (2011), “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers,” Foundations and Trends in Machine Learning , 3, 1–122.
   Google Scholar
• Bredies, K. , Kunisch, K. , and Pock, T. (2010), “Total Generalized Variation,” SIAM Journal on Imaging Sciences , 3, 492–526.
   Web of Science ®Google Scholar
• Canny, J. (2004), “GaP: A Factor Model for Discrete Data,” in Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 122–129.
   Google Scholar
• Chambolle, A. (2004), “An Algorithm for Total Variation Minimization and Applications,” Journal of Mathematical Imaging and Vision , 20, 89–97.
   Web of Science ®Google Scholar
• Chambolle, A. , Novaga, M. , Cremers, D. , and Pock, T. (2010), “An Introduction to Total Variation for Image Analysis,” in Theoretical Foundations and Numerical Methods for Sparse Recovery , ed. Massimo Fornasier , Berlin: De Gruyter, pp. 263–340.
   Google Scholar
• Choi, H. M. , and Hobert, J. P. (2013), “The Pólya-Gamma Gibbs Sampler for Bayesian Logistic Regression Is Uniformly Ergodic,” Electronic Journal of Statistics , 7, 2054–2064.
   Web of Science ®Google Scholar
• Combettes, P. L. , and Pesquet, J.-C. (2011), “Proximal Splitting Methods in Signal Processing,” in Fixed-Point Algorithms for Inverse Problems in Science and Engineering , eds. H. H. Bauschke , R. Burachik , P. Combettes , V. Elser , D. Luke , and H. Wolkowicz , New York: Springer, pp. 185–212.
   Google Scholar
• Damien, P. , Wakefield, J. , and Walker, S. (1999), “Gibbs Sampling for Bayesian Non-Conjugate and Hierarchical Models by Using Auxiliary Variables,” Journal of the Royal Statistical Society, Series B, 61, 331–344.
   Google Scholar
• Dang, Q. A. , and Ehrhardt, M. (2012), “On Dirac Delta Sequences and Their Generating Functions,” Applied Mathematics Letters , 25, 2385–2390.
   Web of Science ®Google Scholar
• Del Moral, P. , Doucet, A. , and Jasra, A. (2012), “An Adaptive Sequential Monte Carlo Method for Approximate Bayesian Computation,” Statistics and Computing , 22, 1009–1020.
   Web of Science ®Google Scholar
• Dempster, A. P. , Laird, N. M. , and Rubin, D. B. (1977), “Maximum Likelihood From Incomplete Data via the EM Algorithm,” Journal of the Royal Statistical Society, Series B, 39, 1–38.
   Google Scholar
• Dobson, A. J. , and Barnett, A. G. (2008), An Introduction to Generalized Linear Models , Texts in Statistical Science (3rd ed.), Boca Raton, FL: Chapman & Hall/CRC Press.
   Google Scholar
• Doucet, A. , Godsill, S. J. , and Robert, C. P. (2002), “Marginal Maximum a Posteriori Estimation Using Markov Chain Monte Carlo,” Statistics and Computing , 12, 77–84.
   Web of Science ®Google Scholar
• Duane, S. , Kennedy, A. , Pendleton, B. J. , and Roweth, D. (1987), “Hybrid Monte Carlo,” Physics Letters B , 195, 216–222.
   Web of Science ®Google Scholar
• Dümbgen, L. , and Rufibach, K. (2009), “Maximum Likelihood Estimation of a Log-Concave Density and Its Distribution Function: Basic Properties and Uniform Consistency,” Bernoulli , 15, 40–68.
   Web of Science ®Google Scholar
• Durmus, A. , and Moulines, E. (2017), “Nonasymptotic Convergence Analysis for the Unadjusted Langevin Algorithm,” The Annals of Applied Probability , 27, 1551–1587.
   Web of Science ®Google Scholar
• Durmus, A. , Moulines, E. , and Pereyra, M. (2018), “Efficient Bayesian Computation by Proximal Markov Chain Monte Carlo: When Langevin Meets Moreau,” SIAM Journal on Imaging Sciences , 11, 473–506.
   Web of Science ®Google Scholar
• Edwards, R. G. , and Sokal, A. D. (1988), “Generalization of the Fortuin-Kasteleyn-Swendsen-Wang Representation and Monte Carlo Algorithm,” Physical Review D , 38, 2009–2012. DOI: 10.1103/physrevd.38.2009.
   PubMed Web of Science ®Google Scholar
• Fearnhead, P. , and Prangle, D. (2012), “Constructing Summary Statistics for Approximate Bayesian Computation: Semi-Automatic Approximate Bayesian Computation,” Journal of the Royal Statistical Society, Series B, 74, 419–474.
   Google Scholar
• Fellows, M. , Mahajan, A. , Rudner, T. G. J. , and Whiteson, S. (2019), “VIREL: A Variational Inference Framework for Reinforcement Learning,” in Advances in Neural Information Processing Systems , pp. 7120–7134.
   Google Scholar
• Geman, D. , and Reynolds, G. (1992), “Constrained Restoration and the Recovery of Discontinuities,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 14, 367–383.
   Web of Science ®Google Scholar
• Geman, D. , and Yang, C. (1995), “Nonlinear Image Recovery With Half-Quadratic Regularization,” IEEE Transactions on Image Processing , 4, 932–946. DOI: 10.1109/83.392335.
   PubMed Web of Science ®Google Scholar
• Giovannelli, J. F. (2008), “Unsupervised Bayesian Convex Deconvolution Based on a Field With an Explicit Partition Function,” IEEE Transactions on Image Processing , 17, 16–26. DOI: 10.1109/tip.2007.911819.
   PubMed Web of Science ®Google Scholar
• Hartley, H. O. (1958), “Maximum Likelihood Estimation From Incomplete Data,” Biometrics , 14, 174–194.
   Web of Science ®Google Scholar
• Higdon, D. (2007), A Primer on Space-Time Modeling From a Bayesian Perspective , Boca Raton, FL: Chapman & Hall/CRC, pp. 217–279.
   Google Scholar
• Higdon, D. M. (1998), “Auxiliary Variable Methods for Markov Chain Monte Carlo With Applications,” Journal of the American Statistical Association , 93, 585–595.
   Web of Science ®Google Scholar
• Holmes, C. C. , and Mallick, B. K. (2003), “Generalized Nonlinear Modeling With Multivariate Free-Knot Regression Splines,” Journal of the American Statistical Association , 98, 352–368.
   Web of Science ®Google Scholar
• Hurn, M. (1997), “Difficulties in the Use of Auxiliary Variables in Markov Chain Monte Carlo Methods,” Statistics and Computing , 7, 35–44.
   Web of Science ®Google Scholar
• Krichene, W. , Bayen, A. , and Bartlett, P. L. (2015), “Accelerated Mirror Descent in Continuous and Discrete Time,” in Advances in Neural Information Processing Systems , pp. 2845–2853.
   Google Scholar
• Kyung, M. , Gill, J. , Ghosh, M. , and Casella, G. (2010), “Penalized Regression, Standard Errors, and Bayesian Lassos,” Bayesian Analysis , 5, 369–411.
   Web of Science ®Google Scholar
• Liechty, M. W. , Liechty, J. C. , and Müller, P. (2009), “The Shadow Prior,” Journal of Computational and Graphical Statistics , 18, 368–383.
   Web of Science ®Google Scholar
• Marnissi, Y. , Chouzenoux, E. , Benazza-Benyahia, A. , and Pesquet, J.-C. (2018), “An Auxiliary Variable Method for Markov Chain Monte Carlo Algorithms in High Dimension,” Entropy , 20, 110.
   Web of Science ®Google Scholar
• Meng, X.-L. , and van Dyk, D. (1997), “The EM Algorithm—An Old Folk-Song Sung to a Fast New Tune,” Journal of the Royal Statistical Society, Series B, 59, 511–567.
   Google Scholar
• Nesterov, Y. , and Spokoiny, V. (2017), “Random Gradient-Free Minimization of Convex Functions,” Foundations of Computational Mathematics , 17, 527–566.
   Web of Science ®Google Scholar
• Papandreou, G. , and Yuille, A. L. (2010), “Gaussian Sampling by Local Perturbations,” in Advances in Neural Information Processing Systems , pp. 1858–1866.
   Google Scholar
• Park, T. , and Casella, G. (2008), “The Bayesian Lasso,” Journal of the American Statistical Association , 103, 681–686.
   Web of Science ®Google Scholar
• Pereyra, M. (2016), “Proximal Markov Chain Monte Carlo Algorithms,” Statistics and Computing , 26, 745–760.
   Web of Science ®Google Scholar
• Polson, N. G. (1996), “Convergence of Markov Chain Monte Carlo Algorithms,” in Bayesian Statistics (Vol. 5), eds. J. M. Bernardo , J. O. Berger , A. P. Dawid , and A. F. M. Smith . Oxford: Oxford University Press, pp. 297–321.
   Google Scholar
• Polson, N. G. , Scott, J. G. , and Windle, J. (2013), “Bayesian Inference for Logistic Models Using Polya-Gamma Latent Variables,” Journal of the American Statistical Association , 108, 1339–1349.
   Web of Science ®Google Scholar
• Rendell, L. J. , Johansen, A. M. , Lee, A. , and Whiteley, N. (2020), “Global Consensus Monte Carlo,” Journal of Computational and Graphical Statistics .
   Web of Science ®Google Scholar
• Robert, C. P. (2001), The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation (2nd ed.), New York: Springer.
   Google Scholar
• Robert, C. P. , and Casella, G. (2004), Monte Carlo Statistical Methods (2nd ed.), Berlin: Springer.
   Google Scholar
• Salim, A. , Koralev, D. , and Richtarik, P. (2019), “Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates,” in Advances in Neural Information Processing Systems , pp. 6649– 6661.
   Google Scholar
• Scheffé, H. (1947), “A Useful Convergence Theorem for Probability Distributions,” The Annals of Mathematical Statistics , 18, 434–438.
   Google Scholar
• Scott, S. L. , Blocker, A. W. , Bonassi, F. V. , Chipman, H. A. , George, E. I. , and McCulloch, R. E. (2016), “Bayes and Big Data: The Consensus Monte Carlo Algorithm,” International Journal of Management Science and Engineering Management , 11, 78–88.
   Web of Science ®Google Scholar
• She, Y. , and Owen, A. B. (2011), “Outlier Detection Using Nonconvex Penalized Regression,” Journal of the American Statistical Association , 106, 626–639.
   Web of Science ®Google Scholar
• Sisson, S. A. , Fan, Y. , and Beaumont, M. A. (eds.) (2018a), Handbook of Approximate Bayesian Computation (1st ed.), Boca Raton, FL: Chapman and Hall/CRC Press.
   Google Scholar
• Sisson, S. A. , Fan, Y. , and Beaumont, M. A. (eds.) (2018b), “Overview of Approximate Bayesian Computation,” in Handbook of Approximate Bayesian Computation (1st ed.), eds. S. Sisson , Y. Fan , and M. Beaumont , Boca Raton, FL: Chapman and Hall/CRC Press, pp. 3–54.
   Google Scholar
• Swendsen, R. H. , and Wang, J.-S. (1987), “Nonuniversal Critical Dynamics in Monte Carlo Simulations,” Physical Review Letters , 58, 86–88. DOI: 10.1103/PhysRevLett.58.86.
   PubMed Web of Science ®Google Scholar
• Tanner, M. A. , and Wong, W. H. (1987), “The Calculation of Posterior Distributions by Data Augmentation,” Journal of the American Statistical Association , 82, 528–540.
   Web of Science ®Google Scholar
• Tanner, M. A. , and Wong, W. H. (2010), “From EM to Data Augmentation: The Emergence of MCMC Bayesian Computation in the 1980s,” Statistical Science , 25, 506–516.
   Web of Science ®Google Scholar
• van Dyk, D. A. , and Meng, X.-L. (2001), “The Art of Data Augmentation,” Journal of Computational and Graphical Statistics , 10, 1–50.
   Web of Science ®Google Scholar
• Vono, M. , Dobigeon, N. , and Chainais, P. (2019), “Split-and-Augmented Gibbs Sampler—Application to Large-Scale Inference Problems,” IEEE Transactions on Signal Processing , 67, 1648–1661.
   Web of Science ®Google Scholar
• Wand, M. P. , and Jones, M. C. (1995), Kernel Smoothing , London: Chapman & Hall/CRC.
   Google Scholar
• Wang, C. , and Blei, D. M. (2018), “A General Method for Robust Bayesian Modeling,” Bayesian Analysis , 13, 1163–1191.
   Web of Science ®Google Scholar
• Wang, X. , and Dunson, D. B. (2013), “Parallelizing MCMC via Weierstrass Sampler,” Technical Report, arXiv no. 1312.4605.
   Google Scholar
• Wang, Y. , Yang, J. , Yin, W. , and Zhang, Y. (2008), “A New Alternating Minimization Algorithm for Total Variation Image Reconstruction,” SIAM Journal on Imaging Sciences , 1, 248–272.
   Web of Science ®Google Scholar
• Wilkinson, R. (2013), “Approximate Bayesian Computation (ABC) Gives Exact Results Under the Assumption of Model Error,” Statistical Applications in Genetics and Molecular Biology , 12, 1–13.
   PubMed Web of Science ®Google Scholar