11
$\begingroup$

I am reading up on prior distributions and I calculated Jeffreys prior for a sample of normally distributed random variables with unknown mean and unknown variance. According to my calculations, the following holds for Jeffreys prior: $$ p(\mu,\sigma^2)=\sqrt{det(I)}=\sqrt{det\begin{pmatrix}1/\sigma^2 & 0 \\ 0 & 1/(2\sigma^4)\end{pmatrix}}=\sqrt{\frac{1}{2\sigma^6}}\propto\frac{1}{\sigma^3}.$$ Here, $I$ is Fisher's information matrix.

However, I have also read publications and documents which state

as Jeffreys prior for the case of a normal distribution with unkown mean and variance. What is the 'actual' Jeffreys prior?

$\endgroup$
6
$\begingroup$

I think the discrepancy is explained by whether the authors consider the density over $\sigma$ or the density over $\sigma^2$. Supporting this interpretation, the exact thing that Kass and Wassermann write is $$ \pi(\mu, \sigma) = 1 / \sigma^2, $$ while Yang and Berger write $$ \pi(\mu, \sigma^2) = 1 / \sigma^4. $$

$\endgroup$
  • 2
    $\begingroup$ Thanks, I overlooked this. However, this still does not explain the discrepancy between $1/\sigma^3$ and $1/\sigma^4$. $\endgroup$ – Nussig Jun 11 '15 at 1:02
  • 3
    $\begingroup$ Actually, having a prior of $\pi(\mu, \sigma)=1/\sigma^2$ is the same as having a prior $\pi(\mu, \sigma^2)=1/\sigma^3$, due the reparametrization property of Jeffreys prior: $$ \pi(\mu, \sigma)=\pi(\mu, \sigma^2)det(J_f)\propto \frac{1}{\sigma^3}2\sigma \propto \frac{1}{\sigma^2}$$ with $J_f$ the Jacobian matrix of $f: (\mu, \sigma)\to (\mu, \sigma^2)$, i.e. $$J_f=\begin{pmatrix}1&0\\0&2\sigma\end{pmatrix}$$. $\endgroup$ – Nussig Jun 11 '15 at 2:00
  • 3
    $\begingroup$ @Nussig, I checked the calculation, and I think you are right arriving at $1/\sigma^3$. You are also right that the reparametrization amounts only to a factor $1/\sigma$. Considering this, your calculation is in accordance with Kass and Wassermann, and I can only guess that Yang and Berger made a mistake. This makes sense also since the former is a regular reviewed journal paper and the latter is a draft of a kind of formula collection. $\endgroup$ – A. Donda Jun 11 '15 at 2:20
  • 3
    $\begingroup$ Kass and Wassermann also note that Jeffreys introduced a modified rule, according to which location and scale parameters should be treated separately. This leads to $\pi(\mu, \sigma) = 1 / \sigma$ and therefore $\pi(\mu, \sigma^2) = 1 / \sigma^2$, but still not to $\pi(\mu, \sigma^2) = 1 / \sigma^4$. $\endgroup$ – A. Donda Jun 11 '15 at 2:22
  • 2
    $\begingroup$ Jim Berger is still an active scientist, so to be sure you might check directly with him: stat.duke.edu/~berger $\endgroup$ – A. Donda Jun 11 '15 at 2:28
4
$\begingroup$

The existing answers already well answer the original question. As a physicist, I would just like to add to this discussion a dimensionality argument. If you consider $\mu$ and $\sigma^2$ to describe a distribution of a random variable in a real 1D space and measured in meters, they have the dimensions $[\mu] \sim m$ and $[\sigma^2] \sim m^2$. To have a physically correct prior, you need it to have the right dimensions, i.e. the only powers of $\sigma$ physically possible in a non-parametric prior are: $$ \pi(\mu, \sigma) \sim 1/\sigma^{2} $$ and $$ \pi(\mu, \sigma^2) \sim 1/\sigma^{3} $$.

$\endgroup$
  • $\begingroup$ Why is there $\sigma^{3}$ in the second expression? $\endgroup$ – cerebrou Feb 25 at 15:59
3
$\begingroup$

$\frac{1}{\sigma^3}$ is the Jeffreys prior. However in practice $\frac{1}{\sigma^2}$ is quite often used cause it leads to a relatively simple posterior, the "intuition" of this prior is that it corresponds with a flat prior on $\log(\sigma)$.

$\endgroup$
  • 1
    $\begingroup$ Thanks, @Noshgul. I get the point about the flat prior on $\log(\sigma)$. However, could you elaborate on 'relatively simple posterior'? If I am not mistaken, Jeffrey's prior results in a normal-inverse-$\chi^2$ posterior, i.e. $$ (\mu,\sigma^2)|D \sim \mathcal{N}\chi^{-1}\left(\overline{X}, n,n, \frac{1}{n}\sum(X_i-\overline{X})^2\right). $$ The prior $1/\sigma^2$ should result in a normal-inverse-$\chi^2$ posterior, too, just with different parameters. $\endgroup$ – Nussig Jun 10 '15 at 19:57
  • 1
    $\begingroup$ Ooh, yes it leads to a normal-inverse-$\chi^2(\bar{X},n,n-1,s^2)$. I just find it more natural that the marginal of $\sigma^2$ is an inverse $\chi^2$ with n-1 instead of n degrees of freedom. Anyhow, I certainly did not want to imply that the other priors would lead to annoying distributions. To be honest I didn't know the posterior of the Jeffry's prior by heart nor did I really think to much about it when I wrote the post. $\endgroup$ – Jorne Biccler Jun 11 '15 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.