What's the advantages of bayesian version of linear regression, logistic regression etc

For many regression/classification algorithms, we have the bayesian version of it. Like bayesian linear regression, bayesian logistic regression, bayesian neuron network. I do not fully understand the math in them, but what are its advantages compared with the original algorithm? Is is of great practical use?

Doing Bayesian regression is not an algorithm but a different approach to statistical inference. The major advantage is that, by this Bayesian processing, you recover the whole range of inferential solutions, rather than a point estimate and a confidence interval as in classical regression. (I can only recommend you to read a statistics manual to understand the difference between an algorithm and statistical inference.)

• Could you kindly elaborate on "the whole range of inferential solutions"? That would help the OP (and me!) to better understand how you see the contrast with classical methods. – Assad Ebrahim Jul 31 '13 at 16:04
• @Xi'an The OP's question seems to concern estimation whereas your answer seems to concern inference. – AdamO Feb 18 '16 at 22:17
• @AdamO: given that the OP has not been seen on X validated since Nov. 12, 2012, (s)he does not seem very concerned by the question! – Xi'an Feb 19 '16 at 9:06

In general, the advantage of Bayesian estimation is that you can incorporate the use of a prior, or assumed knowledge about the current state of "beliefs", and how the evidence might update those beliefs.

• Can you elaborate a little bit please on this? For example, when it is better to use bayesian regression than common regression? – Poete Maudit Aug 30 '18 at 8:44

Maximum Likelihood Estimation(MLE) of the parameters of a Non Bayesian Regression model or simply a linear regression model overfits the data, meaning the unknown value for a certain value of independent variable becomes too precise when calculated. Bayesian Linear Regression relaxes this fact, saying that there is uncertainty involved by incorporating "Predictive Distribution".

The Difference

Let's do a small thought experiment with regards to regression. Let's make this simple regression:

$y = \beta_0 + \beta_1 x$

We can apply solve for the best possible weights and linear algebra states that the best weights can be found via:

$\beta^* = [\beta_0, \beta_1]^T = (X'X)^{-1} X'y$

Now imagine that we run the regression with a small dataset and with a large dataset. I think it should be safe to argue that we are much more certain that our estimate $\beta^*$ is sensible when we have 1000 points of data opposed to if we only have 10 points of data. Because $\beta^*$ is a single datapoint and not a distribution, we cannot quantify our certainty with it.

This is where and why bayesians interpret $\beta$ differently. Bayesians look at $\beta$ and think that depending on the dataset, we can be more or less certain about it. If this feels very confusing, you may appreciate this and this blogpost where the difference is mentioned in more detail. [Disclaimer, these blogposts are written by myself]

The Benefit

Now we will assume that we've learned our $\beta^*$ and this is a distribution instead of a mere number. You'll notice that our prediction now becomes stochastic.

$\beta_0 + \beta_1 x_i \to \hat{y_i}$

Our prediction $y_i$ is now a distribution too. This means that we have confidence bounds on our prediction. If you care about the uncertainty of your prediction, this is a very very nice thing.