Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Given that the posterior estimate of $\sigma'^{2}$of a normal likelihood and an inverse gamma prior on $\sigma^2$ is:

$$\sigma'^{2}\sim\textrm{IG}\left(\alpha + \frac{n}{2}, \beta +\frac{\sum_{i=1}^n{(y_i-\mu)^2}}{2}\right)$$

which is equivalent to

$$\sigma'^{2}\sim\textrm{IG}\left( \frac{n}{2}, \frac{n\sigma^2}{2}\right)$$

since a weak $\textrm{IG}(\alpha, \beta)$ prior on $\sigma^2$ removes $\alpha$ and $\beta$ from eqn 1:

$$\sigma'^{2}\sim\textrm{IG}\left( \frac{n}{2}, \frac{\sum_{i=1}^n{(y_i-\mu)^2}}{2}\right)$$

It is apparent that the posterior estimate of $\sigma^2$ is a function of the sample size and the sum of squares of the likelihood. But what does this mean? There is a derivation on Wikipedia that I don't quite follow.

I have the following questions

  1. Can I get to this second equation without invoking Bayes' rule? I am curious if there is something inherent in the parameters of an IG that is related to the mean and variance independent of the normal likelihood.
  2. Can I use the sample size and standard deviation from a previous study to estimate an informed prior on $\sigma^2$, and then update the prior with new data? This seems straightforward, but I can not find any examples of doing so, or rationale why this would be a legitmate approach - other than what can be seen in the posterior.
  3. Is there a popular probability or statistics textbook that I can consult for further explanation?
share|improve this question
    
Don't you mean an inverse-gamma likelihood and an inverse-gamma prior? –  Neil G Aug 7 '12 at 5:58
1  
First of all, I see in your question several misunderstandings: from Bayes theorem you get not posterior estimate, but entire posterior distribution.The second point is that this posterior distribution does not depend on "the sum of squares of the likelihood". It simply depends on the size of your sample (namely, n) and the sample values, which is perfectly natural and reasonable. These dependencies affect your posterior estimates of mean, variance, etc. For example you posterior mean ofvariance parameter is equal to $$\frac{1}{n-2}\sum \left ( y_{i}-\mu \right )^{2}$$ –  Tomas Aug 7 '12 at 10:44
    
@thomas by estimate, I meant estimate of the posterior distribution;. Isn't the sum of squares term in the posterior the exact same calculation as the ss term in the normal likelihood? –  Abe Aug 7 '12 at 16:11
    
@Abe I recently asked (and answered) a question related to your question nr. 2. It is given the SD and the SD of the SD how to calculate the corresponding gamma prior on the precision of a normal distribution: Question is here: stats.stackexchange.com/questions/41187/… –  Rasmus Bååth Nov 12 '12 at 22:55

2 Answers 2

up vote 4 down vote accepted
+50

I think it is more correct to speak of the posterior distribution of your parameter $\sigma'^{2}$ rather than its posterior estimate. For clarity of notations, I will drop the prime in $\sigma'^{2}$ in what follows.

Suppose that $X$ is distributed as $\mathcal{N}(0, \sigma^2)$, — I drop $\mu$ for now to make a heuristic example — and $1/\sigma^2 = \sigma^{-2}$ is distributed as $\Gamma(\alpha, \beta)$ and is independent of $X$.

The pdf of $X$ given $\sigma^{-2}$ is Gaussian, i.e.

$$f(x|\sigma^{-2}) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{x^2}{2\sigma^2}\right).$$

The joint pdf of $(X, \sigma^{-2})$, $f(x,\sigma^{-2})$ is obtained by multiplying $f(x|\sigma^{-2})$ by $g(\sigma^{-2})$ — the pdf of $\sigma^{-2}$. This comes out as

$$f(x, \sigma^{-2}) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{x^2}{2\sigma^2}\right) \frac{\beta^{\alpha}}{\Gamma(\alpha)}\exp \left(-\frac{\beta}{ \sigma^2}\right)\frac{1}{\sigma^{2(\alpha-1)}}.$$

We can group similar terms and rewrite this as follows

$$f(x, \sigma^{-2}) \propto \sigma^{-2(\alpha-1/2)} \exp\left(-\sigma^{-2} \left(\beta + x^2/2 \right)\right).$$

The posterior distribution of $\sigma^{-2}$ is by definition the pdf of $\sigma^{-2}$ given $x$, which is $f(x, \sigma^{-2}) / f(x)$ by Bayes' formula. To answer your question 1. I don't think there is a way to express $f(\sigma^{-2}|x)$ from $f(x, \sigma^{-2})$ without using Bayes' formula. On with the computation, we recognize in the formula above something that looks like a $\Gamma$ function, so integrating $\sigma^{-2}$ out to get $f(x)$ is fairly easy.

$$ f(x) \propto (\beta + x^2/2)^{-(\alpha+1/2)}, $$

so by dividing we get

$$ f(\sigma^{-2}|x) \propto \left(\beta + x^2/2 \right) \left( \sigma^{-2} \left(\beta + x^2/2 \right) \right)^{\alpha-1/2} \exp\left(-\sigma^{-2} \left(\beta + x^2/2 \right)\right) \\ \propto \left( \sigma^{-2} \left(\beta + x^2/2 \right) \right)^{\alpha-1/2} \exp\left(-\sigma^{-2} \left(\beta + x^2/2 \right)\right). $$

And here in the last formula we recognize a $\Gamma$ distribution with parameters $(\alpha + 1/2, \beta + x^2/2)$.

If you have an IID sample $((x_1, \sigma_1^{-2}), ..., (x_n, \sigma^{-2}_n))$, by integrating out all the $\sigma_i^{-2}$, you would get $f(x_1, ..., x_n)$ and then $f(\sigma_1^{-2}, ..., \sigma_n^{-2}|x_1, ..., x_n)$ as a product of the following terms:

$$ f(\sigma_1^{-2}, ..., \sigma_n^{-2}|x_1, ..., x_n) \propto \prod_{i=1}^n \left( \sigma_i^{-2} \left(\beta + x_i^2/2 \right) \right)^{\alpha-1/2} \exp\left(-\sigma_i^{-2} \left(\beta + x_i^2/2 \right)\right), $$

Which is a product of $\Gamma$ variables. And we are stuck here because of the multiplicity of the $\sigma_i^{-2}$. Besides, the distribution of the mean of those independent $\Gamma$ variables is not straightforward to compute.

However, if we assume that all the observations $x_i$ share the same value of $\sigma^{-2}$ (which seems to be your case) i.e. that the value of $\sigma^{-2}$ was drawn only once from a $\Gamma(\alpha, \beta)$ and that all $x_i$ were then drawn with that value of $\sigma^{-2}$, we obtain

$$ f(x_1, ..., x_n, \sigma^{-2}) \propto \sigma^{-2 (\alpha + n/2)} \exp\left(-\sigma^{-2} \left(\beta + \frac{1}{2} \sum_{i=1}^n x_i^2\right) \right), $$

from which we derive the posterior distribution of $\sigma^{-2}$ as your equation 1 by applying Bayes' formula.

The posterior distribution of $\sigma^{-2}$ is a $\Gamma$ that depends on $\alpha$ and $\beta$, your prior parameters, the sample size $n$ and the observed sum of squares. The prior mean of $\sigma^{-2}$ is $\alpha/\beta$ and the variance is $\alpha/\beta^2$, so if $\alpha = \beta$ and the value is very small, the prior carries very little information about $\sigma^{-2}$ because the variance becomes huge. The values being small, you can drop them from the above equations and you end up with your equation 3.

In that case the posterior distribution becomes independent of the prior. This formula says that the inverse of the variance has a $\Gamma$ distribution that depends only on the sample size and the sum of squares. You can show that for Gaussian variables of known mean, $S^2$, the estimator of the variance, has the same distribution, except that it is a function of the sample size and the true value of the parter $\sigma^2$. In the Bayesian case, this is the ditribution of the parameter, in the frequentist case, it is the distribution of the estimator.

Regarding your question 2. you can of course use the values obtained in a previous experiment as your priors. Because we established a parallel between Bayesian and frequentist interpretation in the above, we can elaborate and say that it is like computing a variance from a small sample size and then collecting more data points: you would update your estimate of the variance rather than throw away the first data points.

Regarding your question 3. I like the Introduction to Mathematical Statistics by Hogg, McKean and Craig, which usually gives the detail of how to derive these equations.

share|improve this answer

For question 1, the second equation follows from Bayes' rule as you point out, and I don't see how to avoid that.

For question 2, yes, you can do this. Just use a prior of the same form as your second equation.

For question 3, I would look for something about exponential families. Maybe someone will recommend a good resource.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.