I searched about Bayesian Ridge Regression on Internet but most of the result I got is about Bayesian Linear Regression. I wonder if it's both the same things because the formula look quite similar


Ridge regression uses regularization with $L_2$ norm, while Bayesian regression, is a regression model defined in probabilistic terms, with explicit priors on the parameters. The choice of priors can have the regularizing effect, e.g. using Laplace priors for coefficients is equivalent to $L_1$ regularization. They are not the same, because ridge regression is a kind of regression model, and Bayesian approach is a general way of defining and estimating statistical models that can be applied to different models.

Ridge regression model is defined as

$$ \underset{\beta}{\operatorname{arg\,min}}\; \|y - X\beta\|^2_2 + \lambda \|\beta\|^2_2 $$

In Bayesian setting, we estimate the posterior distribution by using Bayes theorem

$$ p(\theta|X) \propto p(X|\theta)\,p(\theta) $$

Ridge regression means assuming Normal likelihood and Normal prior for the parameters. After droping the normalizing constant, the log-density function of normal distribution is

$$\begin{align} \log p(x|\mu,\sigma) &= \log\Big[\frac{1}{\sigma \sqrt{2\pi} } e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2}\Big] \\ &= \log\Big[\frac{1}{\sigma \sqrt{2\pi} }\Big] + \log\Big[e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2}\Big] \\ &\propto -\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2 \\ &\propto -\frac{1}{\sigma^2} \|x - \mu\|^2_2 \end{align}$$

Now you can see that maximizing normal log-likelihood, with normal priors is equivalent to minimizing the squared loss, with ridge penalty

$$\begin{align} \underset{\beta}{\operatorname{arg\,max}}& \; \log\mathcal{N}(y|X\beta, \sigma) + \log\mathcal{N}(0, \tau) \\ = \underset{\beta}{\operatorname{arg\,min}}&\; -\Big\{\log\mathcal{N}(y|X\beta, \sigma) + \log\mathcal{N}(0, \tau)\Big\} \\ = \underset{\beta}{\operatorname{arg\,min}}&\; \frac{1}{\sigma^2}\|y - X\beta\|^2_2 + \frac{1}{\tau^2} \|\beta\|^2_2 \end{align}$$

For reading more on ridge regression and regularization see the threads: Why does ridge estimate become better than OLS by adding a constant to the diagonal?, and What problem do shrinkage methods solve?, and When should I use lasso vs ridge?, and Why is ridge regression called "ridge", why is it needed, and what happens when $\lambda$ goes to infinity?, and many others we have.

  • $\begingroup$ Thanks for the answer ! i tried to understand what are the advantages of $L_2$ norm, the explanation on Scikit is a bit complicated for me. It would be nice if you could point out the problem with normal Ordinary Least Squares $\endgroup$ – Thien Feb 13 '18 at 10:43
  • 1
    $\begingroup$ @Thien see the edit for some links $\endgroup$ – Tim Feb 13 '18 at 10:51
  • $\begingroup$ So, it is called as Bayesian Ridge Regression if using Normal priors for coefficients(which is equivalent to $L_2$ ) ? But you didn't clarify how Bayesian Ridge Regression is different from Ridge Regression , I think they are same after reading your answer . $\endgroup$ – Mithril Jul 10 '20 at 9:49
  • $\begingroup$ @Mithril the difference is that Ridge Regression minimizes loss, while Bayesian version maximizes the posterior probability by fitting a probabilistic model. So it is the same as the difference between Bayesian linear regression vs linear regression or any other Bayesian counterpart of the classical model. $\endgroup$ – Tim Jul 10 '20 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.