Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When performing bayesian linear regression, one needs to assign a prior for the slope $a$ and intercept $b$. Since $b$ is a location parameter it makes sense to assign an uniform prior; however, it seems to me that $a$ is akin to a scale parameter and it seems unnatural to assign an uniform prior to it.

On the other hand, it doesn't quite seem right to assign the usual uninformative Jeffrey prior ($1/a$) for a slope of a linear regression. For one, it can be negative. But I fail to see what else it could be.

So what is the "proper" uninformative prior for the slope of a bayesian linear regression? (Any references would be appreciated.)

share|improve this question
    
Slope is not really like a scale parameter - for example, it can be negative. There isn't a "proper" uninformative ("low-information" may be a better term) prior. There are some common choices, which may suit different people or different situations. – Glen_b Jan 25 at 23:11
up vote 6 down vote accepted

From Bayesian Data Analysis 3rd ed., p. 355:

The standard noninformative prior distribution

In the normal regression model, a convenient noninformative prior distribution is uniform on $(\beta, \log \sigma)$ or, equivalently, $$p(\beta,\sigma^2|X) \propto\sigma^{-2}$$

($X$ referring to the regressors.) The book contains useful further discussion beyond the scope of this question: When this prior is useful, when others are better suited, its posterior, and comparison to classical estimates.

share|improve this answer

Bayesians normally choose priors that make their mathematically challenging lives easier to bear. This means Gaussian priors, unless the model absolutely forbids it. Remember that you need a bivariate prior in your situation, since you have to model the correlation between slope and location, as well as their marginal behaviours. The multivariate normal is your ticket.

A Gaussian prior on the parameters meshes nicely with the (doubtless) Gaussian measurement error that your regression model already has.

By the way, I don't associate slopes with scale parameters, since slopes can be negative and scale parameters cannot.

Now the gaussian distribution is not an uninformative prior, but if you really have no prior information, perhaps you should go frequentist. Or use a Gaussian with a very large variance.

I don't know a modern reference to Bayesian inference. At the risk of using a bazooka to shoot a rabbit, you could look up Rasmussen and Williams, which is available online. The first section of chapter 2 goes through Bayesian regression in some detail.

share|improve this answer

Typically a uniform prior on the slope and offset is used, however I like the idea of putting flat priors on $\tan^{-1}\left(a\right)$ and $b\cos\theta$ with $\theta$ being the angle between the line and y=0. This gives a prior of $$ p\left(a,b\right) = \left(1+a^2\right)^{-3/2} $$ which favours slopes around zero. This is derived at http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/#The-Prior, and Frequentism and Bayesianism: A Python-driven Primer by Jake Vanderblas

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.