Lower-bound on covariance estimated via Laplace approximation?

Question

I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum, i.e. that $$ \mathbf{\Sigma} = -\mathbf{H}^{-1} $$ where $$ \mathbf{H}_{ij} = \frac{\partial^2}{\partial\theta_i \partial \theta_j} \ln{P}(\hat{\underline{\theta}}| \mathcal{D}). $$ However, in some cases where there are a very large number of model parameters, even if the Hessian is known analytically calculating its inverse might not be tractable.

I'm working on such a case at the minute, and so far have been approximating the uncertainty for the $i$'th parameter $\sigma_i$ by assuming the Hessian is diagonal, such that $$ \sigma_i = \sqrt{\left(-\frac{\partial^2}{\partial\theta^{2}_i} \ln{P}(\hat{\underline{\theta}}| \mathcal{D})\right)^{-1}}. $$ However, intuitively it seems to me that when the Hessian is not diagonal, assuming it is should always underestimate the actual value you would obtain by calculating the inverse, i.e. $\sqrt{\mathbf{\Sigma}_{ii}}$.

Is this actually true? i.e. that $$ \sigma_i \le \sqrt{\mathbf{\Sigma}_{ii}} $$ or equivalently $$ -\frac{1}{\mathbf{H}_{ii}} \le -(\mathbf{H}^{-1})_{ii} $$

Answer 1

This seems to be a general inequality for a positive definite (p.d.) symmetric matrix $\mathbf{A}$ of size $n$, namely: $A_{ii}^{-1} \leq \mathbf{A}^{-1}|_{ii}$.

For a proof, consider first the special case $i=1$, and write $\mathbf{A}$ in block form $$ \mathbf{A} = \begin{bmatrix} a & \mathbf{c}^\top \\ \mathbf{c} & \mathbf{B} \end{bmatrix} $$ where $\mathbf{c}$ is a column vector of length $n-1$ and $\mathbf{B}$ is a symmetric matrix of size $n-1$ which is also p.d. We can find an expression for the first diagonal element of $\mathbf{A}^{-1}$ by using a matrix relation $$ \begin{bmatrix} a & \mathbf{c}^\top \\ \mathbf{c} & \mathbf{B} \end{bmatrix} \begin{bmatrix} x \\ \mathbf{y} \end{bmatrix} = \begin{bmatrix} u \\ \mathbf{v} \end{bmatrix} \qquad \text{or} \qquad \left\{ \begin{array}{ll} a x + \mathbf{c}^\top \mathbf{y} &= u \\ \mathbf{c} x + \mathbf{B} \mathbf{y} &= \mathbf{v} \end{array} \right.. $$ Using the system of two equations, we find from the second equation: $\mathbf{y} = \mathbf{B}^{-1}[\mathbf{v} - \mathbf{c}x]$ and the first equation then gives $$ [a - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{c} ] x = u - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{v}. $$
By identification, the coefficient of $x$ between the brackets is the inverse of the diagonal element $\mathbf{A}^{-1}|_{11}$ of $\mathbf{A}^{-1}$. It is $> 0$ because it writes as $\mathbf{d}^\top \mathbf{A} \mathbf{d}$ for $\mathbf{d} := [1, \,-\mathbf{c}^\top \mathbf{B}^{-1}]^\top$. Similarly $a > 0$ and $\mathbf{c}^\top \mathbf{B}^{-1}\mathbf{c} \geq 0$, so $$ \left. \mathbf{A}^{-1} \right|_{11} = \frac{1}{a - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{c} } \quad \geq \quad \frac{1}{a} = A_{11}^{-1} $$ as wanted.

Now note that the $i$-th diagonal element of $\mathbf{A}$ is the first diagonal element of $\mathbf{A}_{\star} := \mathbf{P} \mathbf{A} \mathbf{P}$ where $\mathbf{P}$ is a symmetric permutation matrix with $\mathbf{P}^{-1} = \mathbf{P}$ and it is easily seen that then the $i$-th diagonal element of $\mathbf{A}^{-1}$ is the first diagonal element of $\mathbf{A}_{\star}^{-1}$ as well. So the result for the general case follows from the special case applied to $\mathbf{A}_{\star}$.

Answer 2

It is also easy to see from the law of total variance. If you have $$x \sim N(\mu, \Sigma)$$ and $x_j$ is the j-the element of $x$ and $\hat{x_j}$ is all but j-th element. Then $$x_j|\hat{x_j}\sim N(\hat{\mu_j}, (\Sigma^{-1}|_{jj})^{-1})$$ Now using the law of total variance. $$\Sigma|_{jj} = Var(x_j) = E[Var(x_j|\hat{x_j})] + Var(E[x_j|\hat{x_j}])\ge E[Var(x_j|\hat{x_j})]= (\Sigma^{-1}|_{jj})^{-1} $$ Therefore $$(\Sigma|_{jj})^{-1}\le (\Sigma^{-1}|_{jj})$$

This also explains when the difference between $(\Sigma|_{jj})^{-1}$ and $(\Sigma^{-1}|_{jj})$ will be large -- when your specific variable is strongly correlated with others.