Bayesian Linear Regression vs Least Squares

Question

Suppose $X,Y$ are random variables and we wish to use linear regression $Y=aX+b + \epsilon$. We can determine $a,b$ by using a very straightforward least squares computation. Alternatively, we can give $a,b$ prior distributions and use Bayesian methods to find the maximum likelihoods for $a,b$ and get a linear regression that way instead. It appears that the Bayesian approach here is certainly a lot more difficult to do computationally. However, I have seen statistics books reference this method. My question is, why would someone use the Bayesian approach instead?

Answer 1

Whether you use a Bayesian or a Frequentist method, in general, should depend entirely upon what type of question you are answering or what usage you will make of the result.

Let us assume that all the assumptions required for the use of ordinary least squares are met fully.

In general, null hypothesis methods produce an answer where there is a social convention that most, if not all, parties will accept as valid. Ordinary least squares fall into that category. Everybody seeing the same data and using the same cutoff value for $\alpha$ will agree on the result assuming there are no methodological issues haunting the project.

Its virtue is that it is guaranteed to be perfectly accurate upon infinite repetition and that there is no unbiased estimator that can dominate it unless prior information exists. It is also the most efficient estimator. It can also be a very fast estimator to create computationally. Its virtues are the cause of its extensive usage throughout the sciences.

It has one other virtue that is very important, all information about the location of the parameters will come from the data itself. It works equally well no matter what the real value of $a$ or $b$ happens to be.

So, given all these good things, why use a Bayesian method?

Bayesian methods cannot be dominated by non-Bayesian methods but they can be tied. It is relatively common for a Bayesian estimate to be closer to the value in nature, on average, than the ordinary least squares estimator. Bayesian methods tend to be biased but more precise.

They are also subjective. The best reason to use a Bayesian method is to update your beliefs about the location of $a$ or $b$.

For example, if another researcher had already performed an experiment on this topic, maybe with a large sample, then you can directly incorporate their results as if it were your data by using it to build your prior distribution. If five other researchers have already done work on these parameters, you can incorporate all of their research results by putting them into your prior distribution.

The problem is that your prior distribution is your prior distribution. If I am doing the same research and only have access to one study result, then my results will not match your results because you incorporated more information. Using the same data, we may not agree.

Likewise, if you have spent five decades doing this type of research, you have a lot of built-up professional knowledge that can be incorporated in the prior distribution that an undergraduate could not know or incorporate, so the results will differ.

There is another difficulty. Your Bayesian result will not work equally well over the entire parameter space. It will work best if your prior beliefs are also near to the correct answer. So, if those five prior studies had results that were due to something like a statistical run or the samples were not otherwise representative, then you are adding bad information to your regression.

Conversely, if the research you include is of very high quality, then you may get a very tight estimate of the actual location when your data is included. The Bayesian estimator is not trapped with the Cramer-Rao lower bound, you can get closer to reality than the ordinary least squares can get because you can use real knowledge from outside your experiment to find the location of the parameters.

There is also one other case for Bayesian solutions when gambling or putting money at risk where there is an opponent. There is a concept called a Dutch Book. If I can build a Dutch Book on your, then what will happen is that you and I will play a game of "heads I win, tails you lose." Bayesian methods with real proper priors cannot be Dutch Booked but non-Bayesian methods can be.

There is another class of issues between null hypothesis and Bayesian methods. They are created by having differing interpretations of probabilities. Sometimes these are very important, other times they could be ignored without risk.

Null hypothesis based confidence, tolerance, and prediction intervals, as well as inference, have quite a different interpretation to Bayesian intervals.

The Frequentist interpretation of probability has to do with long-run frequencies. Suppose you were interested in determining a 95% confidence interval for some research. Any function where the interval covers the actual value of the parameter at least 95% of the time would be a valid 95% confidence interval. There are an infinite number of them, so you need to choose them on other properties beyond simple coverage.

One important thing to remember is that you obtain an interval $b\in[5,7]$ does not imply that there is a 95% chance that $b\in[5,7]$. Either there is a 100% chance it is in the interval or a 0% chance. Whether or not it is inside the interval is a factual statement, though possibly one that could not be resolved. However, what you have confidence in is that at least 95% of your intervals, upon infinite repetition, will cover the parameter.

The equivalent Bayesian interval would be the 95% credible set. It may not be a connected interval. The Bayesian 95% credible set may be $b\in[5.5,5.6]\cup[6.7.6.75]$. If those are the highest density regions, which is not required but what is generally done, then those are the smallest regions that contain 95% of the mass. Of course, under a continuous distribution with support on the real numbers, there is an infinite number of sets that can be cut up to add to 95% of the mass.

Note the slight difference in meaning. There is a 95% chance the parameter is inside the set. There was no statement about coverage over repetitions. It could have poor coverage if you would repeat the experiment. Nonetheless, given only the data that you saw plus any prior information that you included, that set is the best set. It has a 95% chance of being true. Of course, there is a 5% chance that it is wrong and 5% is not 0%. Because it is not a frequency, you cannot say that it will be wrong 5% of the time. What you can say is that you believe there is a 95% chance that the location is inside that set. You cannot say how often your beliefs will lead you astray.

Finally, there is one other place where a Bayesian tool could be preferential. There is a concept called a posterior predictive distribution. The Bayesian posterior predictive distribution minimizes the K-L divergence with nature. In other words, if you were using the K-L divergence to measure how wrong your approximation was, the Bayesian posterior predictive distribution would be the least wrong, assuming your prior was made from good information. It isn't possible to get closer to nature.

There is a close non-Bayesian idea called a predictive interval. As with confidence intervals, there are an infinite number of them. They minimize the loss you would take from a prediction based on your personal loss function. They are not complete probability distributions. They are intervals. There is a class of predictive interval that minimizes the average K-L divergence. Its advantage and disadvantage is that it cannot use outside information. It isn't at risk from a bad prior distribution but has no advantage created by a good one. It is on average good, but it can be quite bad for a specific sample.

Answer 2

Dave Harris gave a detailed answer (+1), so let me just give a more general comment. You seem to worry about computational complexity, why? This should never be your biggest concerns.

In practice, for simple models such as linear regression more computationally demanding method on a modern computer would usually run few seconds, maybe minutes, slower than the less demanding one. It could make a visible difference for more complicated models where the training could take hours or days. In both cases you should however ask yourself what is most important for you for the particular problem? For example, if you have strict business requirements for the algorithm to return predictions in milliseconds, than you would worry about computational complexity and discard any demanding algorithm. On another hand, if you have a research project where in two years you need to write a report and your model would train for a week non stop, it is likely not a major problem.

Many computationally expensive algorithms have additional merits, for example, non-parametric models would not make distributional assumptions that are not appropriate for your data, obtaining uncertainty estimates as with Bayesian models, more accurate results, etc.