4
$\begingroup$

Let $$X_i\sim \mathcal{N}(0,\sigma^2)$$ than we know that

$$\sum_{i=1}^N\frac{X_i^2}{N}\sim\Gamma(\frac{N}{2},\frac{2\sigma^2}{N})$$

that the empirical variance follows a Gamma distribution. How do we reconcile this, with the fact the the inverse gamma is being used for the prior on the variance? I know that inverse gamma happens to be conjugate. But would it not make more sense, to use gamma, as this is how empirical variance is distributed?

$\endgroup$
6
$\begingroup$

No reconciliation is needed. In one case you are referring to the sampling distribution of the maximum likelihood estimator, which is a function of the data. In the other, you are referring to the posterior distribution of the actual model parameter. Two different referents; two different solutions.

The advantage of conjugacy is that we get nice closed-form solutions out for the posterior distribution, and this ability depends on the form of the likelihood function, not on the sampling distribution of the maximum likelihood estimator.

If we look at the Normal likelihood function, we see:

$$\mathcal{L}(\sigma^2) \propto \sigma^{-n}\exp\left(-{\sum X_i^2\over 2\sigma^2}\right)$$

Note how the $\sigma^2$ terms are all in the various denominators of ratios, not in the numerators. In order to maintain conjugacy, we need to find a distribution that looks similar:

$$p(\sigma^2) \propto \sigma^{-a}\exp\left(-{b\over\sigma^2}\right)$$

which will lead to a posterior that has the same form:

$$p(\sigma^2|X) \propto \sigma^{-(a+n)}\exp\left(-{b+\sum X_i^2\over\sigma^2}\right)$$

... and that distribution is the inverse-Gamma.

If we were to use the precision $\beta = 1/\sigma^2$ as our parameter of choice, we'd have:

$$\mathcal{L}(\beta) \propto \beta^{n}\exp\left(-{\sum X_i^2\over 2}\beta\right)$$

and evidently the conjugate prior would be a Gamma distribution. Note that in the former case the $\sum X_i^2$ term and $\sigma^2$ terms are in the numerator and denominator of a ratio, respectively; this leads (well, you have to do the math) to the Gamma and inverse-Gamma distributions respectively. In the latter case, the $\beta$ is in the numerator along with $\sum X_i^2$, and we have Gamma distributions in both cases.

$\endgroup$
  • $\begingroup$ So, the fact that Gamma is the sampling distribution of the MLE estimator, has nothing to do with the fact that we use the InverseGamma as a prior on the model parameter (as we use IG for the convenience of conjugacy)? And thank you for the indepth answer, I already accepted it, as this question is a follow-up. $\endgroup$ – LeastSquaresWonderer Jun 12 '18 at 2:23
  • 1
    $\begingroup$ Well, excluding the fact that they are related through the Normal distribution, that's correct. It's a little tough when it's all math, because of course the functional form is what causes both outcomes, but other than that, no, the inverse Gamma is in no way chosen because the MLE has a Gamma distribution, but for the convenience of conjugacy. Note that the inverse Gamma is also the conjugate prior for the rate parameter of an Exponential or Gamma distribution, or, more generally, of the inverse of a scale parameter of an exponential family distribution. $\endgroup$ – jbowman Jun 12 '18 at 2:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.