Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
For a prior of $\sigma^2 \sim IG(0.01, 0.01)$, often recommended as an uninformative prior for the variance parameter in MCMC approaches and other Bayesian methods, which parameterization does this recommendation correspond to?
Is it shape/scale or shape/rate? That is, in R, would it be invgamma::dinvgamma(sigma, shape = 0.01, scale = 0.01) or invgamma::dinvgamma(sigma, shape = 0.01, rate = 0.01)?
TLDR: The recommendation for an Inverse Gamma prior on variance parameters uses the shape/scale parameterization. But take care with the invgamma package which seems to have mixed up rate and scale.
I can see why the parameterization is confusing if you use the invgamma package: In its documentation the Inverse Gamma density is written as
f(x) = rate^shape/Gamma(shape) x^(-1-shape) e^(-rate/x)
In the Wikipedia article on the Inverse Gamma distribution, the pdf is of course the same: $$ f(x) = \beta^\alpha/\Gamma(\alpha) x^{-\alpha-1}\exp(-\beta/x) $$ but the $\beta$ parameter is described as the scale.
Wikipedia gets it right: $\beta$ is the "scale", not the "rate" of the Inverse Gamma. This is the standard parameterization.
It may be helpful to reason about this by simply plotting the density. Note that I use the invgamma package which seems to have mixed up scale and rate.
The pdf on the right doesn't look like the density of a "noninformative" prior as it puts all its mass on large values of $x$. So if you use the invgamma package (better not?), you would specify the "rate"; if you use another implementation, you would specify the scale.
And for more up-to-date prior recommendations, take a look at the Stan documentation: Prior Choice Recommendations. This guide has five levels of priors, from flat to specific informative.
For example the recommendations for a weakly informative prior on a scale parameter, such as the variance, include an exponential with expected value 10, a half-normal (0,10) and a half-Cauchy(0,5).
library("invgamma")
par(mfrow = c(1, 2))
curve(
dinvgamma(x, shape = 0.01, rate = 0.01),
main = "shape = 0.01, rate = 0.01",
from = 0.0001, to = 1.5, n = 10001, font.main = 1,
xlim = c(0, 1.5), ylab = "f(x)", yaxs = "i"
)
curve(
dinvgamma(x, shape = 0.01, scale = 0.01),
main = "shape = 0.01, scale = 0.01",
from = 0.0001, to = 1.5, n = 10001, font.main = 1,
xlim = c(0, 1.5), ylab = "f(x)", yaxs = "i"
)
The inverse gamma is a conjugate prior for $\sigma^2$ in typical Gaussian-data models. The underlying idea is to have a convenient proper-prior approximation to an improper $\sigma^{-2}$ prior, for a weakly informative/low information prior. In the limit as $\epsilon=\beta=\alpha$ goes to $0$, the $\text{inverse gamma}(\epsilon, \epsilon)$ distribution goes to that improper prior. This occurs when $\beta$ is a scale parameter, not a rate parameter.
This use goes back at least to about 1993 give or take, that was when I first encountered it - I believe in conversation - and indeed subsequently used it in an illustrative example. Gelman [1] credits Spiegelhalter et al (1994), so that's possibly its first appearance, though I would not be in the least surprised to discover it's a good deal older than that.
A warning: according to Gelman ([1], again) the $\text{inverse gamma}(\epsilon, \epsilon)$ for small $\epsilon$ (where 0.01 or 0.001 are typical choices of small) distribution is not as weakly informative a prior as was hoped for; he says that the results depend on which $\epsilon$ is chosen.
[1]: Gelman, A., (2006),
"Prior distributions for variance parameters in hierarchical models,"
Bayesian Analysis 1, No. 3, pp. 515–533
http://www.stat.columbia.edu/%7Egelman/research/published/taumain.pdf
