Let $$X_i\sim \mathcal{N}(0,\sigma^2)$$ than we know that


that the empirical variance follows a Gamma distribution. How do we reconcile this, with the fact the the inverse gamma is being used for the prior on the variance? I know that inverse gamma happens to be conjugate. But would it not make more sense, to use gamma, as this is how empirical variance is distributed?


2 Answers 2


No reconciliation is needed. In one case you are referring to the sampling distribution of the maximum likelihood estimator, which is a function of the data. In the other, you are referring to the posterior distribution of the actual model parameter. Two different referents; two different solutions.

The advantage of conjugacy is that we get nice closed-form solutions out for the posterior distribution, and this ability depends on the form of the likelihood function, not on the sampling distribution of the maximum likelihood estimator.

If we look at the Normal likelihood function, we see:

$$\mathcal{L}(\sigma^2) \propto \sigma^{-n}\exp\left(-{\sum X_i^2\over 2\sigma^2}\right)$$

Note how the $\sigma^2$ terms are all in the various denominators of ratios, not in the numerators. In order to maintain conjugacy, we need to find a distribution that looks similar:

$$p(\sigma^2) \propto \sigma^{-a}\exp\left(-{b\over\sigma^2}\right)$$

which will lead to a posterior that has the same form:

$$p(\sigma^2|X) \propto \sigma^{-(a+n)}\exp\left(-{b+\sum X_i^2\over\sigma^2}\right)$$

... and that distribution is the inverse-Gamma.

If we were to use the precision $\beta = 1/\sigma^2$ as our parameter of choice, we'd have:

$$\mathcal{L}(\beta) \propto \beta^{n}\exp\left(-{\sum X_i^2\over 2}\beta\right)$$

and evidently the conjugate prior would be a Gamma distribution. Note that in the former case the $\sum X_i^2$ term and $\sigma^2$ terms are in the numerator and denominator of a ratio, respectively; this leads (well, you have to do the math) to the Gamma and inverse-Gamma distributions respectively. In the latter case, the $\beta$ is in the numerator along with $\sum X_i^2$, and we have Gamma distributions in both cases.

  • $\begingroup$ So, the fact that Gamma is the sampling distribution of the MLE estimator, has nothing to do with the fact that we use the InverseGamma as a prior on the model parameter (as we use IG for the convenience of conjugacy)? And thank you for the indepth answer, I already accepted it, as this question is a follow-up. $\endgroup$ Commented Jun 12, 2018 at 2:23
  • 1
    $\begingroup$ Well, excluding the fact that they are related through the Normal distribution, that's correct. It's a little tough when it's all math, because of course the functional form is what causes both outcomes, but other than that, no, the inverse Gamma is in no way chosen because the MLE has a Gamma distribution, but for the convenience of conjugacy. Note that the inverse Gamma is also the conjugate prior for the rate parameter of an Exponential or Gamma distribution, or, more generally, of the inverse of a scale parameter of an exponential family distribution. $\endgroup$
    – jbowman
    Commented Jun 12, 2018 at 2:43

Eventhough this question was posted 4 years ago, I will make a post since it seems that there is a misconception in the comments of the other post. The question "So, the fact that Gamma is the sampling distribution of the MLE estimator, has nothing to do with the fact that we use the InverseGamma as a prior on the model parameter?" From the perspective of jbowman's answer then the answer to the above question seems to be no. But there is actually a deeper explanation. As LeastSquaresWonderer mentions in his question the maximum likelihood estimator of the variance is gamma distributed: \begin{align*} \sigma^2_{MLE} := \sum^N_{i=1} \frac{X_i^2}{N} \sim \Gamma\left(\frac{N}{2},\frac{2 \sigma^2}{N}\right) \ . \end{align*} The missing part is that you are looking at the distribution of $\sigma^2_{MLE}$, but in Bayesian statistics you are treating the parameters them selfs as the random variables. Thus in the above expression you want to have $\sigma^2$ on the left hand side treating it as a random variable. For a gamma distributed random variable $X \sim \Gamma(k,\theta)$ we have for $c>0$ that $cX \sim \Gamma(k,c\theta)$ in our case we replace $c$ with $\frac{1}{\sigma^2 \sigma^2_{MLE}}$ then we get: \begin{align*} \frac{1}{\sigma^2 \sigma^2_{MLE}} \sigma^2_{MLE} = \frac{1}{\sigma^2} \sim \Gamma\left(\frac{N}{2},\frac{2}{N\sigma^2_{MLE}}\right) \ . \end{align*} Now if a random variable $X \sim \Gamma(k,\theta)$ is gamma distributed then $\frac{1}{X}\sim \Gamma^{-1}(k,1/\theta)$ which is the inverse gamma distribution. Thus we get: \begin{align*} \sigma^2 \sim \Gamma^{-1}\left(\frac{N}{2},\frac{N\sigma^2_{MLE}}{2}\right) \ . \end{align*} This together with the fact that the inverse gamma distribution is a conjugate prior of a normal distribution motivates the use of the inverse gamma as a prior.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.