Obtaining Jeffreys prior by taking the limit of a particular prior density on $(\mu, \Sigma)$

Question

Text: Bayesian Data Analysis 3E by Gelman, section 3.6

Let $y | \mu, \Sigma \sim \text{MVN}(\mu, \Sigma),$ where

1. $\mu$ is a column vector of length $d$
2. $\Sigma$ is a $d \times d$ symmetric, positive definite, variance matrix
3. both unknown

The conjugate prior for $(\mu, \Sigma)$ is the normal-inverse-Wishart distribution, where $$\begin{align} \Sigma &\sim \text{Inv-Wishart}_{\nu_0} \left( \Lambda_0^{-1} \right) \\ \mu | \Sigma &\sim \text{MVN} \left( \mu_0, \Sigma /\kappa_0 \right) \end{align},$$ which gives the conjugate prior density to be $$p(\mu, \Sigma) \propto |\Sigma|^{- \left( \frac{\nu_0 + d}{2} + 1\right)} \exp \left( -\frac12 \text{tr} \left( \Lambda_0 \Sigma^{-1} \right) - \frac{\kappa_0}{2} (\mu - \mu_0)^T \Sigma^{-1}(\mu - \mu_0) \right)$$

In another case where $\Sigma$ follows the inverse-Wishart with $d-1$ degrees of freedom (the other parameter is not specified, but I assuming is it $\Lambda_0^{-1}$), the author suggests using the multivariate Jeffreys noninformative prior for $(\mu, \Sigma)$, i.e. $$p(\mu, \Sigma) \propto |\Sigma|^{- \frac{d+1}{2}}.$$

The author says that this is the limit of the conjugate prior density as $\kappa_0 \rightarrow 0$, $\nu_0 \rightarrow -1$, and $|\Lambda_0| \rightarrow 0$. The first limit seems to zero out the second term in the exponential above. The second limit seems to give $|\Sigma|^{{- \frac{d+1}{2}}}$. The third limit seems like it should lead to zeroing out the first term in the exponential, but I cannot see how.

I was hoping someone had some insight on how $|\Lambda_0| \rightarrow 0$ helps give the Jeffreys prior.

Answer 1

Perhaps what was meant is that one takes the limit as $\|\Lambda_0\|$ goes to $0$, where $\|\Lambda_0\|$ is a norm on the space of $d\times d$ matricies, rather than just requiring the determinant $|\Lambda_0| \to 0$. Indeed, one might have $\Lambda_0$ converge to some singular, non-zero, matrix and we wouldn't obtain Jeffreys prior; just consider $\Lambda_0 \to \mathbf{1}\mathbf{1}^\top$.

On the other hand, recall that Frobenius norm $\|\Lambda_0\| = \sqrt{\operatorname{tr}(\Lambda_0^\top \Lambda_0)}$ is induced by the Frobenius inner product $\langle X, Y \rangle = \operatorname{tr}(X^\top Y)$. Then Cauchy-Schwartz implies $\operatorname{tr}(\Lambda_0 \Sigma^{-1}) = \langle \Lambda_0, \Sigma^{-1}\rangle \le \|\Lambda_0\| \cdot \|\Sigma^{-1}\|$ and so as $\|\Lambda_0\| \to 0$ you have the first term of your exponential going to $0$. This implies $|\Lambda_0| \to 0$, but is stronger.

Now, the above argument applies specifically to the Frobenius norm. But we recall from elementary school linear algebra that all norms on finite dimensional vector spaces are equivalent, so in fact this applies to any norm on a space of $d \times d$ matrices.

Answer 2

Some of the normalizing constants have the thing you're taking the limit with respect to, so you have to write it all out.

$$ p(\Sigma) = \frac{|\Lambda_0|^{\nu_0/2}}{2^{\nu_0 d / 2} \Gamma_d(\nu_0/2)}\left|\Sigma\right|^{-(\nu_0+d+1)/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right) $$

$$ p(\mu \mid \Sigma) = \exp\left(- \frac{\kappa_0}{2} (\mu - \mu_0)^T \Sigma^{-1}(\mu - \mu_0) \right) |\Sigma|^{-1/2} (2\pi)^{d/2} $$

First $$ \lim_{\kappa_0 \to 0} p(\mu, \Sigma) = \frac{|\Lambda_0|^{\nu_0/2}}{2^{\nu_0 d / 2} \Gamma_d(\nu_0/2)}\left|\Sigma\right|^{-(\nu_0+d+1)/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right)|\Sigma|^{-1/2} (2\pi)^{d/2}. $$

Then take the limit with respect to $\nu_0$. $\lim_{\nu_0 \to -1} \lim_{\kappa_0 \to 0} p(\mu, \Sigma)$ equals $$ \frac{|\Lambda_0|^{-1/2}2^{d / 2}}{ \Gamma_d(-1/2)}\left|\Sigma\right|^{-d/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right)|\Sigma|^{-1/2} (2\pi)^{d/2} $$ which simplifies to something proportional to $$ |\Lambda_0|^{-1/2}\left|\Sigma\right|^{-(d+1)/2}\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right) $$ (I dropped $2^{d }$,$\Gamma_d(-1/2)$, $(\pi)^{d/2}$).

Then you take the limit as $|\Lambda_0| \to 0$ to get the final result. I must admit, though, that I am not totally clear on how the last step works. For some additional reason that isn't stated in the text, $\text{etr}\left(-\frac{1}{2}\Lambda_0 \Sigma^{-1}\right)|\Lambda_0|^{-1/2}$ goes to some constant that doesn't depend on $\Sigma$. It will have different limits depending on how you make the determinant go to $0$ (see: What is the limit of this expression?).