What should an uninformative prior be for the slope when doing linear regression?

Question

When performing bayesian linear regression, one needs to assign a prior for the slope $a$ and intercept $b$. Since $b$ is a location parameter it makes sense to assign an uniform prior; however, it seems to me that $a$ is akin to a scale parameter and it seems unnatural to assign an uniform prior to it.

On the other hand, it doesn't quite seem right to assign the usual uninformative Jeffrey prior ($1/a$) for a slope of a linear regression. For one, it can be negative. But I fail to see what else it could be.

So what is the "proper" uninformative prior for the slope of a bayesian linear regression? (Any references would be appreciated.)

Answer 1

From Bayesian Data Analysis 3rd ed., p. 355:

The standard noninformative prior distribution

In the normal regression model, a convenient noninformative prior distribution is uniform on $(\beta, \log \sigma)$ or, equivalently, $$p(\beta,\sigma^2|X) \propto\sigma^{-2}$$

($X$ referring to the regressors.) The book contains useful further discussion beyond the scope of this question: When this prior is useful, when others are better suited, its posterior, and comparison to classical estimates.

Answer 2

Bayesians normally choose priors that make their mathematically challenging lives easier to bear. This means Gaussian priors, unless the model absolutely forbids it. Remember that you need a bivariate prior in your situation, since you have to model the correlation between slope and location, as well as their marginal behaviours. The multivariate normal is your ticket.

A Gaussian prior on the parameters meshes nicely with the (doubtless) Gaussian measurement error that your regression model already has.

By the way, I don't associate slopes with scale parameters, since slopes can be negative and scale parameters cannot.

Now the gaussian distribution is not an uninformative prior, but if you really have no prior information, perhaps you should go frequentist. Or use a Gaussian with a very large variance.

I don't know a modern reference to Bayesian inference. At the risk of using a bazooka to shoot a rabbit, you could look up Rasmussen and Williams, which is available online. The first section of chapter 2 goes through Bayesian regression in some detail.

Answer 3

Typically a uniform prior on the slope and offset is used, however I like the idea of putting flat priors on $\tan^{-1}\left(a\right)$ and $b\cos\theta$ with $\theta$ being the angle between the line and y=0. This gives a prior of $$ p\left(a,b\right) = \left(1+a^2\right)^{-3/2} $$ which favours slopes around zero. This is derived at http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/#The-Prior, and Frequentism and Bayesianism: A Python-driven Primer by Jake Vanderblas