Bayesian estimation of the variance

Question

The mean of the Gamma distribution is $\alpha/\beta$, while the mean of the Inverse Gamma is $\beta/(\alpha-1)$. Similarly, the mode of the Gamma is $(\alpha-1)/\beta$, but the mode of the Inverse Gamma is $\alpha/(\beta+1)$.

How does this relate to the title of the question?

Well, if we are given data $X$ assumed to be normally distributed, the population variance is given by: $$\sigma_{pop}^2=\frac{\sum(x_i-\bar x)^2}{n}\equiv\frac{s_n^2}{n}$$ However, if we take a Bayesian approach and choose a non-informative Normal-Inverse-Gamma conjugate prior for the variance (i.e. $\alpha_0\rightarrow0, \beta_0\rightarrow0$), we have that the marginal distribution of $\sigma^2$ is also Inverse-Gamma distributed, with $\alpha=n/2, \beta=s_n^2/2$ and mean: $$E[\sigma^2] = \frac{s_n^2/2}{n/2-1}\neq\sigma_{pop}^2 \quad(!)$$ On the other hand, if one uses an uninformative Normal-Gamma prior: $$E[\tau] = \frac{n/2}{s_n^2/2} = \frac{1}{\sigma_{pop}^2}$$

Assuming the above is correct, I have a couple of questions:

1. I realize that $E[1/X]\neq1/E[X]$, yet I'm not sure why $E[\tau]$ specifically should yield the "correct" result. What's wrong with using $E[\sigma^2]$?
2. The frequentist approach would lead to using the sample variance, which seems to match neither approaches with an uninformative prior. What if any, is the significance of the particular prior that would result in the sample variance for both $\tau$ and $\sigma$?
3. What is the significance of the mode, i.e. the MAP estimator of $\sigma^2$ or $\tau$? again, they are both different, and I don't believe I've seen either being used in practice.

Answer 1

First, a correction:

$\sigma_{pop}^2 = \dfrac{\sum (x_i-\bar{x})}{n} \equiv s_n ^2$ or $\dfrac{n \cdot s_n ^2}{n-1}$

depending on whether or not you know a priori the true mean. That doesn't change your conclusion that $E(\sigma ^2)$ is apparently "wrong" and $E(\tau)$ is apparently "right".

On to your questions:

1. Indeed, $E(1/X) \neq 1/E(X)$. In fact, $E(\sigma ^2) = E(1/\tau) = \alpha / \beta$, which reduces to $\dfrac{1}{\sigma_{pop} ^2}$. There is no particular reason why the expected value of $\sigma$ should also reduce to $\sigma_{pop} ^2$.

In the plot above the Gamma distribution is plotted against $1/x$ to account for the fact that $\sigma = \dfrac{1}{\tau}$.

Even after this correction is applied, notice how the distributions shapes are quite different from one another.

Notice also that, while the shape is different (and thus the mean and mode are also different), the percentiles, and thus the area under the curve (each on its original scale), are the same for all $x$. This is what matters most for a direct comparison, because it allow us to calculate the probability of e.g. $\sigma >= 58.8$ (which is $70\%$, as the plot shows). Clearly, the probability that $\tau >= 1/58.8 = 0.017$ is also $70\%$. The plot also shows that. The medians of both distributions (plotted on the same x-axis) are also the same.

2. The frequentist approach is based on a different set of assumptions than the bayesian approach. Before you decide for either approach, you should decide which set of assumptions best matches your problem`s requirements. See this question for a more complete answer to your question.
3. The MAP estimator is sometimes used in MCMC (simulation) contexts. Starting the simulation from the joint distribution MAP can, in some cases, be more efficient. Other than that, I haven't seen them being used in practice either.