# Bayesian estimation of the variance

The mean of the Gamma distribution is $$\alpha/\beta$$, while the mean of the Inverse Gamma is $$\beta/(\alpha-1)$$. Similarly, the mode of the Gamma is $$(\alpha-1)/\beta$$, but the mode of the Inverse Gamma is $$\alpha/(\beta+1)$$.

How does this relate to the title of the question?

Well, if we are given data $$X$$ assumed to be normally distributed, the population variance is given by: $$\sigma_{pop}^2=\frac{\sum(x_i-\bar x)^2}{n}\equiv\frac{s_n^2}{n}$$ However, if we take a Bayesian approach and choose a non-informative Normal-Inverse-Gamma conjugate prior for the variance (i.e. $$\alpha_0\rightarrow0, \beta_0\rightarrow0$$), we have that the marginal distribution of $$\sigma^2$$ is also Inverse-Gamma distributed, with $$\alpha=n/2, \beta=s_n^2/2$$ and mean: $$E[\sigma^2] = \frac{s_n^2/2}{n/2-1}\neq\sigma_{pop}^2 \quad(!)$$ On the other hand, if one uses an uninformative Normal-Gamma prior: $$E[\tau] = \frac{n/2}{s_n^2/2} = \frac{1}{\sigma_{pop}^2}$$

Assuming the above is correct, I have a couple of questions:

1. I realize that $$E[1/X]\neq1/E[X]$$, yet I'm not sure why $$E[\tau]$$ specifically should yield the "correct" result. What's wrong with using $$E[\sigma^2]$$?
2. The frequentist approach would lead to using the sample variance, which seems to match neither approaches with an uninformative prior. What if any, is the significance of the particular prior that would result in the sample variance for both $$\tau$$ and $$\sigma$$?
3. What is the significance of the mode, i.e. the MAP estimator of $$\sigma^2$$ or $$\tau$$? again, they are both different, and I don't believe I've seen either being used in practice.
• I assume $\sigma _{pop} ^2$ is the estimate for the population variance given the sample $X$'s variance, $s_n ^2$. In that case, shouldn't the denominator for the population variance formula be $n-1$, not $n$? Aug 15, 2020 at 20:39
• @LmnICE - if you use $N$ it's the population variance, if you use $N-1$ it's the sample variance, at least those are the terms I'm used to. Aug 16, 2020 at 12:56
• Right, and I assume that dataset $X$ is a sample from the population of whatever you’re trying to model. The point is that perhaps there is nothing wrong with the fact that the expected values of $\sigma$ and $\tau$ are different from what you would expect from a frequentist standpoint (which is just another, unstated, set of assumptions about the prior). Aug 16, 2020 at 15:06
• Could you explain $\tau$? Is it something like $\tau = 1/\sigma$? In that case you already answered your first question "I realize that $E[1/X] \neq 1/E[X]$". If you realize that, then what is the problem? Sure the two approaches are gonna give a different result. Whichever is best depends on your goal. Aug 17, 2020 at 12:21
• @SextusEmpiricus the precision $\tau$ is a fairly common definition. Understanding why things may be different doesn't answer why one should be preferred. Aug 17, 2020 at 12:23

First, a correction:

$$\sigma_{pop}^2 = \dfrac{\sum (x_i-\bar{x})}{n} \equiv s_n ^2$$ or $$\dfrac{n \cdot s_n ^2}{n-1}$$

depending on whether or not you know a priori the true mean. That doesn't change your conclusion that $$E(\sigma ^2)$$ is apparently "wrong" and $$E(\tau)$$ is apparently "right".

1. Indeed, $$E(1/X) \neq 1/E(X)$$. In fact, $$E(\sigma ^2) = E(1/\tau) = \alpha / \beta$$, which reduces to $$\dfrac{1}{\sigma_{pop} ^2}$$. There is no particular reason why the expected value of $$\sigma$$ should also reduce to $$\sigma_{pop} ^2$$. In the plot above the Gamma distribution is plotted against $$1/x$$ to account for the fact that $$\sigma = \dfrac{1}{\tau}$$.

Even after this correction is applied, notice how the distributions shapes are quite different from one another.

Notice also that, while the shape is different (and thus the mean and mode are also different), the percentiles, and thus the area under the curve (each on its original scale), are the same for all $$x$$. This is what matters most for a direct comparison, because it allow us to calculate the probability of e.g. $$\sigma >= 58.8$$ (which is $$70\%$$, as the plot shows). Clearly, the probability that $$\tau >= 1/58.8 = 0.017$$ is also $$70\%$$. The plot also shows that. The medians of both distributions (plotted on the same x-axis) are also the same.

1. The frequentist approach is based on a different set of assumptions than the bayesian approach. Before you decide for either approach, you should decide which set of assumptions best matches your problem`s requirements. See this question for a more complete answer to your question.
2. The MAP estimator is sometimes used in MCMC (simulation) contexts. Starting the simulation from the joint distribution MAP can, in some cases, be more efficient. Other than that, I haven't seen them being used in practice either.
• I must be missing something - What would the posterior be if Beta goes to infinity? Also, it seems a number of sources use $IG(\epsilon, \epsilon) as an uninformative prior. Aug 17, 2020 at 10:08 • Interesting. As I investigate this further, I came across this question in Math.SE where the OP had the same question that I have regarding the shape of the distribution (In his case, it is the Gamma distribution, but the principle still applies). It turns out that, in a sense,$IG(\epsilon, \epsilon)\$ can be considered vague (if rather problematic, see here). I'll correct my answer. Aug 17, 2020 at 22:36