Lower-bound on covariance estimated via Laplace approximation? I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum, i.e. that
$$
\mathbf{\Sigma} = -\mathbf{H}^{-1}
$$
where
$$
\mathbf{H}_{ij} = \frac{\partial^2}{\partial\theta_i \partial \theta_j} \ln{P}(\hat{\underline{\theta}}| \mathcal{D}).
$$
However, in some cases where there are a very large number of model parameters, even if the Hessian is known analytically calculating its inverse might not be tractable.
I'm working on such a case at the minute, and so far have been approximating the uncertainty for the $i$'th parameter $\sigma_i$ by assuming the Hessian is diagonal, such that
$$
\sigma_i = \sqrt{\left(-\frac{\partial^2}{\partial\theta^{2}_i} \ln{P}(\hat{\underline{\theta}}| \mathcal{D})\right)^{-1}}.
$$
However, intuitively it seems to me that when the Hessian is not diagonal, assuming it is should always underestimate the actual value you would obtain by calculating the inverse, i.e. $\sqrt{\mathbf{\Sigma}_{ii}}$.
Is this actually true? i.e. that
$$
\sigma_i \le \sqrt{\mathbf{\Sigma}_{ii}}
$$
or equivalently
$$
-\frac{1}{\mathbf{H}_{ii}} \le -(\mathbf{H}^{-1})_{ii}
$$
 A: This seems to be a general inequality for a positive definite (p.d.)
symmetric matrix $\mathbf{A}$ of size $n$, namely: $A_{ii}^{-1} \leq
\mathbf{A}^{-1}|_{ii}$.
For a proof, consider first the special case  $i=1$, and write $\mathbf{A}$
in block form
$$
    \mathbf{A} = 
     \begin{bmatrix}
 a & \mathbf{c}^\top \\
 \mathbf{c} & \mathbf{B}
     \end{bmatrix} 
$$
where $\mathbf{c}$ is a column vector of length $n-1$ and
$\mathbf{B}$ is a symmetric matrix of size $n-1$ which is also p.d.
We can find an expression for the first diagonal element of
$\mathbf{A}^{-1}$ by using a matrix relation
$$ 
     \begin{bmatrix}
 a & \mathbf{c}^\top \\
 \mathbf{c} & \mathbf{B}
     \end{bmatrix} \begin{bmatrix} x \\ \mathbf{y} \end{bmatrix}
     =   \begin{bmatrix} u \\ \mathbf{v} \end{bmatrix}
     \qquad \text{or}  \qquad
     \left\{
     \begin{array}{ll}
     a x + \mathbf{c}^\top \mathbf{y} &= u \\
     \mathbf{c} x + \mathbf{B} \mathbf{y} &= \mathbf{v}
     \end{array}
     \right..
$$
Using the system of two equations, we find from the second equation:
$\mathbf{y} = \mathbf{B}^{-1}[\mathbf{v} - \mathbf{c}x]$
and the first equation then gives
$$
   [a - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{c} ] x
   = u - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{v}.
$$
By identification, the coefficient of $x$ between the brackets is the
inverse of the diagonal element $\mathbf{A}^{-1}|_{11}$ of
$\mathbf{A}^{-1}$. It is $> 0$ because it writes as $\mathbf{d}^\top
\mathbf{A} \mathbf{d}$ for $\mathbf{d} := [1, \,-\mathbf{c}^\top
\mathbf{B}^{-1}]^\top$. Similarly $a > 0$ and $\mathbf{c}^\top
\mathbf{B}^{-1}\mathbf{c} \geq 0$, so
$$
   \left. \mathbf{A}^{-1} \right|_{11}
   = \frac{1}{a - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{c} } \quad \geq
   \quad
   \frac{1}{a} = A_{11}^{-1}
$$
as wanted.
Now note that the $i$-th diagonal element of $\mathbf{A}$ is the
first diagonal element of $\mathbf{A}_{\star} :=
\mathbf{P} \mathbf{A} \mathbf{P}$ where $\mathbf{P}$ is a
symmetric permutation matrix with $\mathbf{P}^{-1} = \mathbf{P}$ and
it is easily seen that then the $i$-th diagonal element of
$\mathbf{A}^{-1}$ is the first diagonal element of
$\mathbf{A}_{\star}^{-1}$ as well. So the result for the general
case follows from the special case applied to
$\mathbf{A}_{\star}$.
A: It is also easy to see from the law of total variance.
If you have $$x \sim N(\mu, \Sigma)$$ 
and $x_j$ is the j-the element of $x$ and $\hat{x_j}$ is all but j-th element.
Then $$x_j|\hat{x_j}\sim  N(\hat{\mu_j}, (\Sigma^{-1}|_{jj})^{-1})$$
Now using the law of total variance.
$$\Sigma|_{jj} = Var(x_j) = E[Var(x_j|\hat{x_j})] + Var(E[x_j|\hat{x_j}])\ge E[Var(x_j|\hat{x_j})]= (\Sigma^{-1}|_{jj})^{-1} $$ 
Therefore $$(\Sigma|_{jj})^{-1}\le (\Sigma^{-1}|_{jj})$$
This also explains when the difference between $(\Sigma|_{jj})^{-1}$ and  $(\Sigma^{-1}|_{jj})$ will be large -- when your specific variable is strongly correlated with others. 
