Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Let $$X_i\sim \mathcal{N}(0,\sigma^2)$$ than we know that
$$\sum_{i=1}^N\frac{X_i^2}{N}\sim\Gamma(\frac{N}{2},\frac{2\sigma^2}{N})$$
that the empirical variance follows a Gamma distribution. How do we reconcile this, with the fact the the inverse gamma is being used for the prior on the variance? I know that inverse gamma happens to be conjugate. But would it not make more sense, to use gamma, as this is how empirical variance is distributed?
No reconciliation is needed. In one case you are referring to the sampling distribution of the maximum likelihood estimator, which is a function of the data. In the other, you are referring to the posterior distribution of the actual model parameter. Two different referents; two different solutions.
The advantage of conjugacy is that we get nice closed-form solutions out for the posterior distribution, and this ability depends on the form of the likelihood function, not on the sampling distribution of the maximum likelihood estimator.
If we look at the Normal likelihood function, we see:
$$\mathcal{L}(\sigma^2) \propto \sigma^{-n}\exp\left(-{\sum X_i^2\over 2\sigma^2}\right)$$
Note how the $\sigma^2$ terms are all in the various denominators of ratios, not in the numerators. In order to maintain conjugacy, we need to find a distribution that looks similar:
$$p(\sigma^2) \propto \sigma^{-a}\exp\left(-{b\over\sigma^2}\right)$$
which will lead to a posterior that has the same form:
$$p(\sigma^2|X) \propto \sigma^{-(a+n)}\exp\left(-{b+\sum X_i^2\over\sigma^2}\right)$$
... and that distribution is the inverse-Gamma.
If we were to use the precision $\beta = 1/\sigma^2$ as our parameter of choice, we'd have:
$$\mathcal{L}(\beta) \propto \beta^{n}\exp\left(-{\sum X_i^2\over 2}\beta\right)$$
and evidently the conjugate prior would be a Gamma distribution. Note that in the former case the $\sum X_i^2$ term and $\sigma^2$ terms are in the numerator and denominator of a ratio, respectively; this leads (well, you have to do the math) to the Gamma and inverse-Gamma distributions respectively. In the latter case, the $\beta$ is in the numerator along with $\sum X_i^2$, and we have Gamma distributions in both cases.
