1
$\begingroup$

The mean of the Gamma distribution is $\alpha/\beta$, while the mean of the Inverse Gamma is $\beta/(\alpha-1)$. Similarly, the mode of the Gamma is $(\alpha-1)/\beta$, but the mode of the Inverse Gamma is $\alpha/(\beta+1)$.

How does this relate to the title of the question?

Well, if we are given data $X$ assumed to be normally distributed, the population variance is given by: $$\sigma_{pop}^2=\frac{\sum(x_i-\bar x)^2}{n}\equiv\frac{s_n^2}{n}$$ However, if we take a Bayesian approach and choose a non-informative Normal-Inverse-Gamma conjugate prior for the variance (i.e. $\alpha_0\rightarrow0, \beta_0\rightarrow0$), we have that the marginal distribution of $\sigma^2$ is also Inverse-Gamma distributed, with $\alpha=n/2, \beta=s_n^2/2$ and mean: $$E[\sigma^2] = \frac{s_n^2/2}{n/2-1}\neq\sigma_{pop}^2 \quad(!)$$ On the other hand, if one uses an uninformative Normal-Gamma prior: $$E[\tau] = \frac{n/2}{s_n^2/2} = \frac{1}{\sigma_{pop}^2}$$

Assuming the above is correct, I have a couple of questions:

  1. I realize that $E[1/X]\neq1/E[X]$, yet I'm not sure why $E[\tau]$ specifically should yield the "correct" result. What's wrong with using $E[\sigma^2]$?
  2. The frequentist approach would lead to using the sample variance, which seems to match neither approaches with an uninformative prior. What if any, is the significance of the particular prior that would result in the sample variance for both $\tau$ and $\sigma$?
  3. What is the significance of the mode, i.e. the MAP estimator of $\sigma^2$ or $\tau$? again, they are both different, and I don't believe I've seen either being used in practice.
$\endgroup$
7
  • $\begingroup$ I assume $\sigma _{pop} ^2$ is the estimate for the population variance given the sample $X$'s variance, $s_n ^2$. In that case, shouldn't the denominator for the population variance formula be $n-1$, not $n$? $\endgroup$ – LmnICE Aug 15 '20 at 20:39
  • $\begingroup$ @LmnICE - if you use $N$ it's the population variance, if you use $N-1$ it's the sample variance, at least those are the terms I'm used to. $\endgroup$ – nbubis Aug 16 '20 at 12:56
  • $\begingroup$ Right, and I assume that dataset $X$ is a sample from the population of whatever you’re trying to model. The point is that perhaps there is nothing wrong with the fact that the expected values of $\sigma$ and $\tau$ are different from what you would expect from a frequentist standpoint (which is just another, unstated, set of assumptions about the prior). $\endgroup$ – LmnICE Aug 16 '20 at 15:06
  • $\begingroup$ Could you explain $\tau$? Is it something like $\tau = 1/\sigma$? In that case you already answered your first question "I realize that $E[1/X] \neq 1/E[X]$". If you realize that, then what is the problem? Sure the two approaches are gonna give a different result. Whichever is best depends on your goal. $\endgroup$ – Sextus Empiricus Aug 17 '20 at 12:21
  • $\begingroup$ @SextusEmpiricus the precision $\tau$ is a fairly common definition. Understanding why things may be different doesn't answer why one should be preferred. $\endgroup$ – nbubis Aug 17 '20 at 12:23
2
+50
$\begingroup$

First, a correction:

$\sigma_{pop}^2 = \dfrac{\sum (x_i-\bar{x})}{n} \equiv s_n ^2$ or $\dfrac{n \cdot s_n ^2}{n-1}$

depending on whether or not you know a priori the true mean. That doesn't change your conclusion that $E(\sigma ^2)$ is apparently "wrong" and $E(\tau)$ is apparently "right".

On to your questions:

  1. Indeed, $E(1/X) \neq 1/E(X)$. In fact, $E(\sigma ^2) = E(1/\tau) = \alpha / \beta$, which reduces to $\dfrac{1}{\sigma_{pop} ^2}$. There is no particular reason why the expected value of $\sigma$ should also reduce to $\sigma_{pop} ^2$.

plot of IG and Gamma distributions

In the plot above the Gamma distribution is plotted against $1/x$ to account for the fact that $\sigma = \dfrac{1}{\tau}$.

Even after this correction is applied, notice how the distributions shapes are quite different from one another.

Notice also that, while the shape is different (and thus the mean and mode are also different), the percentiles, and thus the area under the curve (each on its original scale), are the same for all $x$. This is what matters most for a direct comparison, because it allow us to calculate the probability of e.g. $\sigma >= 58.8$ (which is $70\%$, as the plot shows). Clearly, the probability that $\tau >= 1/58.8 = 0.017$ is also $70\%$. The plot also shows that. The medians of both distributions (plotted on the same x-axis) are also the same.

  1. The frequentist approach is based on a different set of assumptions than the bayesian approach. Before you decide for either approach, you should decide which set of assumptions best matches your problem`s requirements. See this question for a more complete answer to your question.
  2. The MAP estimator is sometimes used in MCMC (simulation) contexts. Starting the simulation from the joint distribution MAP can, in some cases, be more efficient. Other than that, I haven't seen them being used in practice either.
$\endgroup$
2
  • $\begingroup$ I must be missing something - What would the posterior be if Beta goes to infinity? Also, it seems a number of sources use $IG(\epsilon, \epsilon) as an uninformative prior. $\endgroup$ – nbubis Aug 17 '20 at 10:08
  • $\begingroup$ Interesting. As I investigate this further, I came across this question in Math.SE where the OP had the same question that I have regarding the shape of the distribution (In his case, it is the Gamma distribution, but the principle still applies). It turns out that, in a sense, $IG(\epsilon, \epsilon)$ can be considered vague (if rather problematic, see here). I'll correct my answer. $\endgroup$ – LmnICE Aug 17 '20 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.