3
$\begingroup$

I am currently reading about Bayesian Lasso Regression. It was stated that the Lasso Penalty is non Bayesian because the posterior does not contract at the same rate as the posterior mode. I have looked for a non technical definition of bayesian posterior contraction rate but found none. Can anyone provide me with an explanation?

$\endgroup$

2 Answers 2

5
$\begingroup$

Posterior contraction is about the contraction of the posterior distribution (of some random variable of interest), not the posterior mode.

Under Bayesian models that obey standard regularity conditions, as data are observed, the posterior distribution of a random variable morphs from the prior distribution to a point mass (Dirac measure) centered at the true value of the random variable.

Posterior contraction is the movement of probability density from the prior distribution of a random variable ($\theta$) to a point mass centered at the true value ($\theta_0$) as data are observed. The contraction is described by a distance metric between the posterior distribution and the point mass centered at $\theta_0$, as the number of data points ($n$) increases, and the contraction rate is measured by a sequence of numbers that decreasing to 0 ($\epsilon_n$).

The distinction between posterior distribution and posterior mode is important, because posterior contraction is often discussed in the context of Bayesian nonparametrics. Under Bayesian nonparametric models, we are typically interested in modeling the data density. The posterior distribution of the data density ($f$) approaches the true data density ($f_0$), which is usually a collection of observed data points and not just a point mass.

Regarding lasso, I am not sure whether it's productive to argue over whether lasso is Bayesian. Bayesian methods represent a spectrum from empirical Bayes to full Bayesian hierarchical models.

Nonetheless, lasso has a key deficiency: If you use the posterior mode to estimate the coefficients (consistent with common practice), these estimates will often be biased towards 0 in practice, because the posterior contraction rate is slow (for model with a Laplace prior as compared to a Gaussian prior).

You may consider the use of non-local priors for variable selection, where coefficients are either shrunk toward a point mass at 0 or away from 0. Practically, only coefficients with small values will be forcibly shrunk toward zeros, instead of most coefficients.

$\endgroup$
0
$\begingroup$

The definition I know, is close to what is described in this preprint (Schad et al, arxiv:1904.12765):

posterior contraction estimates how much prior uncertainty is reduced in the posterior estimation:

$$ s = 1 - \frac{\sigma^2_{post}}{\sigma^2_{prior}} $$

Here, the variance of the posterior distribution, $\sigma^2_{post}$, is divided by the prior variance, $\sigma^2_{prior}$. In general, additional information from the likelihood will reduce uncertainty, such that the posterior variance will be smaller than the prior variance. If the data is highly informative, then the variance in the estimate is strongly reduced, and there will be strong posterior contraction $s$ close to $1$. However, when the data provide little information, then the posterior variance will be of similar size as the prior variance, and posterior contractions will be close to $0$

However, I don't see how would this be related to the notions about lasso, that you mentioned.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.