2
$\begingroup$

Text: Bayesian Data Analysis 3E by Gelman, section 3.6

Let $y | \mu, \Sigma \sim \text{MVN}(\mu, \Sigma),$ where

  1. $\mu$ is a column vector of length $d$
  2. $\Sigma$ is a $d \times d$ symmetric, positive definite, variance matrix
  3. both unknown

The conjugate prior for $(\mu, \Sigma)$ is the normal-inverse-Wishart distribution, where $$\begin{align} \Sigma &\sim \text{Inv-Wishart}_{\nu_0} \left( \Lambda_0^{-1} \right) \\ \mu | \Sigma &\sim \text{MVN} \left( \mu_0, \Sigma /\kappa_0 \right) \end{align},$$ which gives the conjugate prior density to be $$p(\mu, \Sigma) \propto |\Sigma|^{- \left( \frac{\nu_0 + d}{2} + 1\right)} \exp \left( -\frac12 \text{tr} \left( \Lambda_0 \Sigma^{-1} \right) - \frac{\kappa_0}{2} (\mu - \mu_0)^T \Sigma^{-1}(\mu - \mu_0) \right)$$

In another case where $\Sigma$ follows the inverse-Wishart with $d-1$ degrees of freedom (the other parameter is not specified, but I assuming is it $\Lambda_0^{-1}$), the author suggests using the multivariate Jeffreys noninformative prior for $(\mu, \Sigma)$, i.e. $$p(\mu, \Sigma) \propto |\Sigma|^{- \frac{d+1}{2}}.$$

The author says that this is the limit of the conjugate prior density as $\kappa_0 \rightarrow 0$, $\nu_0 \rightarrow -1$, and $|\Lambda_0| \rightarrow 0$. The first limit seems to zero out the second term in the exponential above. The second limit seems to give $|\Sigma|^{{- \frac{d+1}{2}}}$. The third limit seems like it should lead to zeroing out the first term in the exponential, but I cannot see how.

I was hoping someone had some insight on how $|\Lambda_0| \rightarrow 0$ helps give the Jeffreys prior.

$\endgroup$
1
$\begingroup$

Perhaps what was meant is that one takes the limit as $\|\Lambda_0\|$ goes to $0$, where $\|\Lambda_0\|$ is a norm on the space of $d\times d$ matricies, rather than just requiring the determinant $|\Lambda_0| \to 0$. Indeed, one might have $\Lambda_0$ converge to some singular, non-zero, matrix and we wouldn't obtain Jeffreys prior; just consider $\Lambda_0 \to \mathbf{1}\mathbf{1}^\top$.

On the other hand, recall that Frobenius norm $\|\Lambda_0\| = \sqrt{\operatorname{tr}(\Lambda_0^\top \Lambda_0)}$ is induced by the Frobenius inner product $\langle X, Y \rangle = \operatorname{tr}(X^\top Y)$. Then Cauchy-Schwartz implies $\operatorname{tr}(\Lambda_0 \Sigma^{-1}) = \langle \Lambda_0, \Sigma^{-1}\rangle \le \|\Lambda_0\| \cdot \|\Sigma^{-1}\|$ and so as $\|\Lambda_0\| \to 0$ you have the first term of your exponential going to $0$. This implies $|\Lambda_0| \to 0$, but is stronger.

Now, the above argument applies specifically to the Frobenius norm. But we recall from elementary school linear algebra that all norms on finite dimensional vector spaces are equivalent, so in fact this applies to any norm on a space of $d \times d$ matrices.

$\endgroup$
1
$\begingroup$

Some of the normalizing constants have the thing you're taking the limit with respect to, so you have to write it all out.

$$ p(\Sigma) = \frac{|\Lambda_0|^{\nu_0/2}}{2^{\nu_0 d / 2} \Gamma_d(\nu_0/2)}\left|\Sigma\right|^{-(\nu_0+d+1)/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right) $$

$$ p(\mu \mid \Sigma) = \exp\left(- \frac{\kappa_0}{2} (\mu - \mu_0)^T \Sigma^{-1}(\mu - \mu_0) \right) |\Sigma|^{-1/2} (2\pi)^{d/2} $$

First $$ \lim_{\kappa_0 \to 0} p(\mu, \Sigma) = \frac{|\Lambda_0|^{\nu_0/2}}{2^{\nu_0 d / 2} \Gamma_d(\nu_0/2)}\left|\Sigma\right|^{-(\nu_0+d+1)/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right)|\Sigma|^{-1/2} (2\pi)^{d/2}. $$

Then take the limit with respect to $\nu_0$. $\lim_{\nu_0 \to -1} \lim_{\kappa_0 \to 0} p(\mu, \Sigma)$ equals $$ \frac{|\Lambda_0|^{-1/2}2^{d / 2}}{ \Gamma_d(-1/2)}\left|\Sigma\right|^{-d/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right)|\Sigma|^{-1/2} (2\pi)^{d/2} $$ which simplifies to something proportional to $$ |\Lambda_0|^{-1/2}\left|\Sigma\right|^{-(d+1)/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right) $$ (I dropped $2^{d }$,$\Gamma_d(-1/2)$, $(\pi)^{d/2}$).

Then you take the limit as $|\Lambda_0| \to 0$ to get the final result. I must admit, though, that I am not totally clear on how the last step works. For some additional reason that isn't stated in the text, $\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right)|\Lambda_0|^{-1/2}$ goes to some constant that doesn't depend on $\Sigma$. It will have different limits depending on how you make the determinant go to $0$ (see: What is the limit of this expression?).

$\endgroup$
  • $\begingroup$ For these types of arguments it is typical to "ignore" the normalizing constant. If you take the limit honestly then you will get jibberish and it most likely will depend on the rates at which the terms go to zero. You are right that it can make a difference ignoring constants because the inferences from Jeffreys does not necessarily coincide with the inference of the prior you get from taking this type of limit. But it is still usually what is meant when one encounters this argument for motivating Jeffreys; ignore the normalizing constant, even though you are taking the limit. $\endgroup$ – guy Aug 22 '18 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.