Thank God That Regressing Y on X is Not the Same as Regressing X on Y: Direct and Indirect Residual Augmentations

Abstract

What does regressing Y on X versus regressing X on Y have to do with Markov chain Monte Carlo (MCMC)? It turns out that many strategies for speeding up data augmentation (DA) type algorithms can be understood as fostering independence or “de-correlation” between a regression function and the corresponding residual, thereby reducing or even eliminating dependence among MCMC iterates. There are two general classes of algorithms, those corresponding to regressing parameters on augmented data/auxiliary variables and those that operate the other way around. The interweaving strategy of Yu and Meng provides a general recipe to automatically take advantage of both, and it is the existence of two different types of residuals that makes the interweaving strategy seemingly magical in some cases and promising in general. The concept of residuals—which depends on actual data—also highlights the potential for substantial improvements when DA schemes are allowed to depend on the observed data. At the same time, there is an intriguing phase transition type of phenomenon regarding choosing (partially) residual augmentation schemes, reminding us once more of the prevailing issue of trade-off between robustness and efficiency. This article reports on these latest theoretical investigations (using a class of normal/independence models) and empirical findings (using a posterior sampling for a probit regression) in the search for effective residual augmentations—and ultimately more MCMC algorithms—that meet the 3-S criterion: simple, stable, and speedy. Supplementary materials for the article are available online.

Key Words:

• Ancillary-Sufficient Interweaving Strategy (ASIS)
• Conditional augmentation
• MCMC
• Marginal augmentation
• Phase transition
• Probit regression
• PX-DA

SUPPLEMENTARY MATERIALS

Dataset: The supplementary files for this article include the lupus nephritis dataset used in probit regression. (xumengyu.zip, zip archive)

R Code: The supplementary files include the R code that can be used to replicate the algorithm proposed in the article. (xumengyu.zip, zip archive)

Appendix: The supplementary files include the Appendix that gives proofs needed in the article:

A.  Proof of the bound for the normal model.

B.  Proof of the bound (2.10) for the normal/independence model.

C.  Proof of the limits of . (xumengyu_appendix.pdf)

ACKNOWLEDGMENTS

This article is based on a keynote address (by Meng) at the ICMS Workshop on Advances in Markov chain Monte Carlo held in Edinburgh during April 23–25, 2012. We thank the organizers, Mark Girolami, Antonietta Mira, and Christian Robert for the invitation and for helpful discussions, many participants—especially Jim Hobert, Omiros Papaspiliopoulos, Gareth Roberts, and David van Dyk—for stimulating exchanges, and the National Science Foundation for partial financial support. We also thank Steven Finch for very helpful proofreading and comments, and Stefan Wilhelm for his timely help regarding the R package tmvtnorm.