Let me preface this by saying that I am an applied statistician teaching myself Bayesian regression. If we can do this without Greek letters that would be great.

Suppose I am modeling a continuous outcome on some predictor. We'll assume a jeffrey's prior for the variance. Now I need a prior for my predictor and the constant. Essentially what I am doing is setting up a distribution that the values for the parameters (the coefficients for the constant and predictor) will be drawn from, correct?

Assuming I want informative priors, my immediate inclination is to go with a normal prior. I set the mean to my "best guess" of what the effect of the predictor ought to be and then set the variance. I like the normal distribution because it is essentially favoring values near my guess while discounting those far from it (i.e. the tails) and discounting them in a symmetrical way. I'm also specifically wondering if I can make use of the standard deviation to characterize my confidence in my guess. An example will illustrate.

I get that larger variances mean less weight assigned to the prior belief. However, I am curious if I can interpret this in terms of standard deviations. For example, if I believe the effect will be 5, I set the mean at 5. Let's say I'm 95% confident that the effect will be between 15 and -5. This would call for a standard deviation of 5 so I'd set the variance at 25. Is this a proper way to describe my confidence in the prior?

As another example, if I were absolutely positive that the effect was going to be somewhere between -10 and 20 I would use a uniform prior ranging from -10 - 20, correct? Am I also correct in assuming that all this is true regardless of the actual distribution of the independent variable (i.e. it could be continuous or dichotomous)?


Basically, yes. Normal priors are common and your approach to selecting the s.d./variance is perfectly reasonable. I'd recommend reading any of Andrew Gelman's stuff about prior distributions. His book Bayesian Data Analysis is a very good and approachable resource for (almost) all things Bayesian.

Essentially what I am doing is setting up a distribution that the values for the parameters (the coefficients for the constant and predictor) will be drawn from, correct?

In a sense, yes, but I don't think this is the best way to think of priors. Priors are there to dampen/accentuate the likelihood in certain regions of parameter space, i.e. to weight the likelihood in some way.

If we choose improper uniform priors (i.e. completely uninformative), the posterior distribution is just a normalised likelihood, which is very sensitive to the data. The mode of this distribution is the estimate you would get from running any common linear regression procedure (lm in R for example).

This is fine if we have complete faith in our data -- we're sure it reflects the population distribution pretty well.

If we have reason to be suspicious of the data, it makes sense to include a prior. The effect of this is that if the data happen to result in lots of likelihood at some unreasonable parameter value, our prior should cancel it out by attaching a very small weight to that value; remember we multiply the prior and the likelihood (pointwise) to get the posterior distribution.

So you aren't really drawing parameter values from the prior, rather you're expressing a restriction on the parameter values you're willing to accept as reasonable by weighting the likelihood. As a very loose approximation, we're just modifying the likelihood before we run lm to find its maximum to make sure our estimates don't come out as being unreasonable/at odds with our prior understanding of the process.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.