On Variational Bayes Estimation and Model Selection For Linear Regression Using Spike and Slab Priors
Chong You (University of Wollongong)
In applied statistics model selection is one of the most fundamental tasks. The selection of models in the linear regression context has been well-studied. However, the efficiency and effectiveness of these methods in high- dimensional settings is limited. We present a model selection method to addresses this. Variational Bayes (VB) is known as a fast alternative to Markov chain Monte Carlo for performing approximate Bayesian inference. However, VB is often criticised, typically based on empirical grounds, for being unable to produce valid statistical inferences. In You et al (2013), asymptotic properties are proved for VB based estimators in Bayesian linear models, partially contradicting this criticism.
Encouraged by You et al (2013), we extend the VB approach to the more complicated spike and slab priors. We show under mild regularity conditions, that: (i) VB based estimators for the coefficients are consistent estimators of the true parameters; and (ii) the VB estimators of the model indicator variables shrink towards zero rapidly if the corresponding true value of the coefficient is zero and one otherwise. This property allows us to use VB estimates of indicator variables to select models.
Simulations results support that our method is competitive in terms of efficiency and effectiveness in comparison to various alternative model selection procedures considered.