Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm curious as to how putting noninformative priors on the regression coefficients in a GLM compares to maximum likelihood estimation, in frequentist terms. A dispersion parameter $\phi$ is of course a source of trickiness, so perhaps we assume that these are estimated using empirical Bayes.

In the simplest situation, the response $Y_i, i = 1, ..., n$ might be Gaussian distributed with mean $$ E(Y_i | \beta) = x_i ^T \beta $$ have common variance $\sigma^2$, and be independent (given $\beta$). Then the Bayes estimator of $\beta$ is the same as the MLE, and for $\sigma^2$ estimation via empirical Bayes results in $\hat \sigma^2 = \frac{\sum(Y_i - \hat Y_i)^2}{n - p}$ where $p$ is the number of $\beta$'s, which makes an appropriate correction to the denominator that the MLE does not make.

My general sense is that it is a bit easier to get away with flat priors on mean parameters, and that integrating out the mean parameters is generally what you want to be doing to estimate the dispersion parameter $\phi$ (short of putting an informative or "weakly informative" (in Gelman's sense) prior on it). That is to say, we would rather integrate out $\beta$ than max-marginalize (aka profile) it out.

It isn't obvious to me immediately that flat priors on regression coefficients lead to proper posteriors in GLMs; part of me feels that they usually ought to, but part of me also feels that maybe they run into the same sort of issues as MLE's; for example, maybe flat priors break down if the data are perfectly seperated in a logistic regression.

So, general insight or guidance on this is appreciated, as well as references.

share|improve this question

One big difference (philosophical basis aside) - in a Bayesian analysis, if you're interested in inference on the parameters, you're normally looking at the posterior distribution, often the posterior mean; ML is looking at the mode of the likelihood. In large samples with uniformative priors, they'll usually be very similar, but in small samples they can be quite different. For example, likelihood can be distinctly asymmetric.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.