Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have heard that ridge regression can be derived as the mean of a posterior distribution, if the prior is adequately chosen. Is the intuition that the constraints as set on the regression coefficients by the prior (e.g. standard normal distributions around 0) are identical / replace the penalty set on the squared size of the coefficients? Does the prior have to be Gaussian for this equivalence to hold?

share|improve this question

2 Answers 2

No, in the sense that other priors do logically relate to other penalties. In general you do want more mass near zero effect ($\beta=0$) to reduce overfitting/over-interpretation. Ridge is a quadratic (L2, Gaussian) penalty, lasso is an $|\beta|$ (L1, Laplace or double exponential distribution) penalty. Many other penalties (priors) are available. The Bayesian approach has the advantage of yielding a solid interpretation (and solid credible intervals) whereas penalized maximum likelihood estimation (ridge, lasso, etc.) yields $P$-values and confidence intervals that are hard to interpret, because the frequentist approach is somewhat confused by biased (shrunk towards zero) estimators.

share|improve this answer

Two points:

The posterior distribution in the Bayesian case is a distribution. The ridge regression estimate is simply a vector $\hat{\beta}$ and not a distribution. Thus they're not completely equivalent.

It is true that in the case of a multivariate normal prior and multivariate normal likelihood, the posterior is multivariate normal with a mean that is the ridge regression estimate for an appropriately chosen ridge parameter.

The proof of this depends on the particular form of the prior and likelihood and doesn't work for more general priors or likelihood functions.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.