Why do we use inverse Gamma as prior on variance, when empirical variance is Gamma (chi square) Let
$$X_i\sim \mathcal{N}(0,\sigma^2)$$
than we know that
$$\sum_{i=1}^N\frac{X_i^2}{N}\sim\Gamma(\frac{N}{2},\frac{2\sigma^2}{N})$$
that the empirical variance follows a Gamma distribution. How do we reconcile this, with the fact the the inverse gamma is being used for the prior on the variance? I know that inverse gamma happens to be conjugate. But would it not make more sense, to use gamma, as this is how empirical variance is distributed?
 A: No reconciliation is needed.  In one case you are referring to the sampling distribution of the maximum likelihood estimator, which is a function of the data.  In the other, you are referring to the posterior distribution of the actual model parameter.  Two different referents; two different solutions.
The advantage of conjugacy is that we get nice closed-form solutions out for the posterior distribution, and this ability depends on the form of the likelihood function, not on the sampling distribution of the maximum likelihood estimator. 
If we look at the Normal likelihood function, we see:
$$\mathcal{L}(\sigma^2) \propto \sigma^{-n}\exp\left(-{\sum X_i^2\over 2\sigma^2}\right)$$
Note how the $\sigma^2$ terms are all in the various denominators of ratios, not in the numerators. In order to maintain conjugacy, we need to find a distribution that looks similar:
$$p(\sigma^2) \propto \sigma^{-a}\exp\left(-{b\over\sigma^2}\right)$$ 
which will lead to a posterior that has the same form:
$$p(\sigma^2|X) \propto \sigma^{-(a+n)}\exp\left(-{b+\sum X_i^2\over\sigma^2}\right)$$
... and that distribution is the inverse-Gamma.
If we were to use the precision $\beta = 1/\sigma^2$ as our parameter of choice, we'd have:
$$\mathcal{L}(\beta) \propto \beta^{n}\exp\left(-{\sum X_i^2\over 2}\beta\right)$$
and evidently the conjugate prior would be a Gamma distribution.   Note that in the former case the $\sum X_i^2$ term and $\sigma^2$ terms are in the numerator and denominator of a ratio, respectively; this leads (well, you have to do the  math) to the Gamma and inverse-Gamma distributions respectively.   In the latter case, the $\beta$ is in the numerator along with $\sum X_i^2$, and we have Gamma distributions in both cases.
A: Eventhough this question was posted 4 years ago, I will make a post since it seems that there is a misconception in the comments of the other post. The question "So, the fact that Gamma is the sampling distribution of the MLE estimator, has nothing to do with the fact that we use the InverseGamma as a prior on the model parameter?" From the perspective of jbowman's answer then the answer to the above question seems to be no. But there is actually a deeper explanation. As LeastSquaresWonderer mentions in his question the maximum likelihood estimator of the variance is gamma distributed:
\begin{align*}
\sigma^2_{MLE} := \sum^N_{i=1} \frac{X_i^2}{N} \sim \Gamma\left(\frac{N}{2},\frac{2 \sigma^2}{N}\right) \ .
\end{align*}
The missing part is that you are looking at the distribution of $\sigma^2_{MLE}$, but in Bayesian statistics you are treating the parameters them selfs as the random variables. Thus in the above expression you want to have $\sigma^2$ on the left hand side treating it as a random variable. For a gamma distributed random variable $X \sim \Gamma(k,\theta)$ we have for $c>0$ that $cX \sim \Gamma(k,c\theta)$ in our case we replace $c$ with $\frac{1}{\sigma^2 \sigma^2_{MLE}}$ then we get:
\begin{align*}
 \frac{1}{\sigma^2 \sigma^2_{MLE}} \sigma^2_{MLE} = \frac{1}{\sigma^2} \sim \Gamma\left(\frac{N}{2},\frac{2}{N\sigma^2_{MLE}}\right) \ .
\end{align*}
Now if a random variable  $X \sim \Gamma(k,\theta)$ is gamma distributed then $\frac{1}{X}\sim \Gamma^{-1}(k,1/\theta)$ which is the inverse gamma distribution. Thus we get:
\begin{align*}
\sigma^2 \sim \Gamma^{-1}\left(\frac{N}{2},\frac{N\sigma^2_{MLE}}{2}\right) \ .
\end{align*}
This together with the fact that the inverse gamma distribution is a conjugate prior of a normal distribution motivates the use of the inverse gamma as a prior.
