21
$\begingroup$

Background

One of the most commonly used weak prior on variance is the inverse-gamma with parameters $\alpha =0.001, \beta=0.001$ (Gelman 2006).

However, this distribution has a 90%CI of approximately $[3\times10^{19},\infty]$.

library(pscl)
sapply(c(0.05, 0.95), function(x) qigamma(x, 0.001, 0.001))

[1] 3.362941e+19          Inf

From this, I interpret that the $IG(0.001, 0.001)$ gives a low probability that variance will be very high, and the very low probability that variance will be less than 1 $P(\sigma<1|\alpha=0.001, \beta=0.001)=0.006$.

pigamma(1, 0.001, 0.001)
[1] 0.006312353

Question

Am I missing something or is this actually an informative prior?

update to clarify, the reason that I was considering this 'informative' is because it claims very strongly that the variance is enormous and well beyond the scale of almost any variance ever measured.

follow-up would a meta-analysis of a large number of variance estimates provide a more reasonable prior?


Reference

Gelman 2006. Prior distributions for variance parameters in hierarchical models. Bayesian Analysis 1(3):515–533

$\endgroup$
  • 1
    $\begingroup$ A "true" noninformative prior is not a distribution. So there's no prior probability such as P(sigma<1). $\endgroup$ – Stéphane Laurent Dec 11 '12 at 12:33
38
$\begingroup$

Using the inverse gamma distribution, we get:

$$p(\sigma^2|\alpha,\beta) \propto (\sigma^2)^{-\alpha-1} \exp(-\frac{\beta}{\sigma^2})$$

You can see easily that if $\beta \rightarrow 0$ and $\alpha \rightarrow 0$ then the inverse gamma will approach the Jeffreys prior. This distribution is called "uninformative" because it is a proper approximation to the Jeffreys prior

$$p(\sigma^2) \propto \frac{1}{\sigma^2}$$

Which is uninformative for scale parameters see page 18 here for example, because this prior is the only one which remains invariant under a change of scale (note that the approximation is not invariant). This has a indefinite integral of $\log(\sigma^2)$ which shows that it is improper if the range of $\sigma^2$ includes either $0$ or $\infty$. But these cases are only problems in the maths - not in the real world. Never actually observe infinite value for variance, and if the observed variance is zero, you have perfect data!. For you can set a lower limit equal to $L>0$ and upper limit equal $U<\infty$, and your distribution is proper.

While it may seem strange that this is "uninformative" in that it prefers small variance to large ones, but this is only on one scale. You can show that $\log(\sigma^2)$ has an improper uniform distribution. So this prior does not favor any one scale over any other

Although not directly related to your question, I would suggest a "better" non-informative distribution by choosing the upper and lower limits $L$ and $U$ in the Jeffreys prior rather than $\alpha$ and $\beta$. Usually the limits can be set fairly easily with a bit of thought to what $\sigma^2$ actually means in the real world. If it was the error in some kind of physical quantity - $L$ cannot be smaller than the size of an atom, or the smallest size you can observe in your experiment. Further $U$ could not be bigger than the earth (or the sun if you wanted to be really conservative). This way you keep your invariance properties, and its an easier prior to sample from: take $q_{(b)} \sim \mathrm{Uniform}(\log(L),\log(U))$, and then the simulated value as $\sigma^{2}_{(b)}=\exp(q_{(b)})$.

$\endgroup$
  • 5
    $\begingroup$ +1 for not only answering the question, but also providing useful advice. $\endgroup$ – whuber Mar 10 '11 at 4:57
  • $\begingroup$ +1 - uniform for $log(\sigma)$ on a "big" range is often a good choice. For the variance components of a hierarchical model I think you can still get into near-impropriety of the posterior if the range is too large since you're approaching Jeffrey's again. But of course it's a simpler fix - just don't pick huge intervals :) $\endgroup$ – JMS Mar 10 '11 at 5:04
  • $\begingroup$ @JMS - in a heirarchical setting, the data do not "squash" the singularity at 0 (i.e. a level 2 variance could be zero). So the prior for small values matters. $Beta_{2}(1,1)$ is a good level 2 and higher variance prior (I think it also has been called a "half Cauchy", it is similar to $F_{1,1}$-distribution). It has "fat tails" and is "data-robust" in that, if prior and likelihood conflict, the likelihood wins. Also $Beta_{2}(0,0)$ is the jeffreys prior. $\endgroup$ – probabilityislogic Mar 10 '11 at 5:58
  • 1
    $\begingroup$ @probabilityislogic thanks for the explanation. If I understand, the gamma is nice theoretically because its rage is $[0,\infty]$ and because it is conjugate to the normal, but in application these features are not generally required. But what is the difference between sampling from $\sigma\sim exp(U(log(L),log(U))$ and $\sigma\sim U(L,U)$? $\endgroup$ – David LeBauer Mar 10 '11 at 15:41
  • $\begingroup$ @probabilityislogic Not familiar with your notation, are you referring to the beta prime? If so it's an interesting choice. Not the half Cauchy though; that's just the Cauchy restricted to $(0, \infty)$. But the beta prime with $\alpha=1, \beta=1/2$ has been called the "quasi Cauchy" IIRC $\endgroup$ – JMS Mar 10 '11 at 16:36
10
$\begingroup$

It's pretty close to flat. Its median is 1.9 E298, almost the largest number one can represent in double precision floating arithmetic. As you point out, the probability it assigns to any interval that isn't really huge is really small. It's hard to get less informative than that!

$\endgroup$
  • $\begingroup$ thanks for your explanation. I have been running into convergence problems and I was surprised that so many of the variables I work with have means is < 1000 (i.e. if something is >1000 g it is measured in kg), and the variances are on about the same order of magnitude. So, I am realizing that I need more priors that incorporate this information even if I don't really have a good prior knowledge of its value or how it is partitioned. $\endgroup$ – David LeBauer Mar 10 '11 at 4:28
  • $\begingroup$ Depending on the model, your posterior might be very close to improper using this prior $\endgroup$ – JMS Mar 10 '11 at 4:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.