Mean-field approximations for high-dimensional Bayesian Regression

Abstract:
Variational approximations provide an attractive computational alternative to MCMC-based strategies for approximating the posterior distribution in Bayesian inference. Despite their popularity in applications, supporting theoretical guarantees are limited, particularly in high-dimensional settings.

In the first part of the talk, we will study bayesian inference in the context of a linear model with product priors, and derive sufficient conditions for the correctness (to leading order) of the naive mean-field approximation. To this end, we will utilize recent advances in the theory of non-linear large deviations (Chatterjee and Dembo 2014). Next, we analyze the naive mean-field variational problem, and precisely characterize the asymptotic properties of the posterior distribution in this setting.

In the second part of the talk, we will turn to linear regression with iid gaussian design under a proportional asymptotic setting. The naive mean- field approximation is conjectured to be inaccurate in this case|instead, the Thouless-Anderson-Palmer approximation from statistical physics is expected to provide a tight approximation. We will rigorously establish the TAP formula under a uniform spherical prior on the regression coefficients.  This is based on joint work with Sumit Mukherjee (Columbia University) and Jiaze Qiu (Harvard University).

–

Bio:
Subhabrata Sen is an assistant professor in the Department of Statistics, Harvard University. His research interests span Applied Probability, Statistics, and Machine Learning.  He was a Schramm postdoc at Microsoft Research New England and MIT Mathematics from 2017-2019. He graduated from the Stanford Statistics Department in 2017, where he was jointly advised by Prof Amir Dembo and Prof Andrea Montanari. Prior to joining Stanford, he received his undergraduate and Masters degrees in Statistics from the Indian Statistical Institute, Kolkata.