I'm curious as to how putting noninformative priors on the regression coefficients in a GLM compares to maximum likelihood estimation, in frequentist terms. A dispersion parameter $\phi$ is of course a source of trickiness, so perhaps we assume that these are estimated using empirical Bayes.

In the simplest situation, the response $Y_i, i = 1, ..., n$ might be Gaussian distributed with mean $$ E(Y_i | \beta) = x_i ^T \beta $$ have common variance $\sigma^2$, and be independent (given $\beta$). Then the Bayes estimator of $\beta$ is the same as the MLE, and for $\sigma^2$ estimation via empirical Bayes results in $\hat \sigma^2 = \frac{\sum(Y_i - \hat Y_i)^2}{n - p}$ where $p$ is the number of $\beta$'s, which makes an appropriate correction to the denominator that the MLE does not make.

My general sense is that it is a bit easier to get away with flat priors on mean parameters, and that integrating out the mean parameters is generally what you want to be doing to estimate the dispersion parameter $\phi$ (short of putting an informative or "weakly informative" (in Gelman's sense) prior on it). That is to say, we would rather integrate out $\beta$ than max-marginalize (aka profile) it out.

It isn't obvious to me immediately that flat priors on regression coefficients lead to proper posteriors in GLMs; part of me feels that they usually ought to, but part of me also feels that maybe they run into the same sort of issues as MLE's; for example, maybe flat priors break down if the data are perfectly seperated in a logistic regression.

So, general insight or guidance on this is appreciated, as well as references.


1 Answer 1


One big difference (philosophical basis aside) - in a Bayesian analysis, if you're interested in inference on the parameters, you're normally looking at the posterior distribution, often the posterior mean; ML is looking at the mode of the likelihood. In large samples with uniformative priors, they'll usually be very similar, but in small samples they can be quite different. For example, likelihood can be distinctly asymmetric.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.