2
$\begingroup$

I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum, i.e. that $$ \mathbf{\Sigma} = -\mathbf{H}^{-1} $$ where $$ \mathbf{H}_{ij} = \frac{\partial^2}{\partial\theta_i \partial \theta_j} \ln{P}(\hat{\underline{\theta}}| \mathcal{D}). $$ However, in some cases where there are a very large number of model parameters, even if the Hessian is known analytically calculating its inverse might not be tractable.

I'm working on such a case at the minute, and so far have been approximating the uncertainty for the $i$'th parameter $\sigma_i$ by assuming the Hessian is diagonal, such that $$ \sigma_i = \sqrt{\left(-\frac{\partial^2}{\partial\theta^{2}_i} \ln{P}(\hat{\underline{\theta}}| \mathcal{D})\right)^{-1}}. $$ However, intuitively it seems to me that when the Hessian is not diagonal, assuming it is should always underestimate the actual value you would obtain by calculating the inverse, i.e. $\sqrt{\mathbf{\Sigma}_{ii}}$.

Is this actually true? i.e. that $$ \sigma_i \le \sqrt{\mathbf{\Sigma}_{ii}} $$ or equivalently $$ -\frac{1}{\mathbf{H}_{ii}} \le -(\mathbf{H}^{-1})_{ii} $$

$\endgroup$
  • $\begingroup$ The tags covariance-matrix and delta-method would be relevant here. On the other hand, I do not see why bayesian, variational bayes are relevant, and probability seems too general. $\endgroup$ – Yves Oct 29 '18 at 18:27
  • $\begingroup$ Note: if computing the MLE is significantly computationally cheaper than inverting the full Hessian, bootstrap and profile likelihood methods can be used to get valid confidence intervals for small subsets of the parameters. $\endgroup$ – Cliff AB Oct 29 '18 at 18:30
4
$\begingroup$

This seems to be a general inequality for a positive definite (p.d.) symmetric matrix $\mathbf{A}$ of size $n$, namely: $A_{ii}^{-1} \leq \mathbf{A}^{-1}|_{ii}$.

For a proof, consider first the special case $i=1$, and write $\mathbf{A}$ in block form $$ \mathbf{A} = \begin{bmatrix} a & \mathbf{c}^\top \\ \mathbf{c} & \mathbf{B} \end{bmatrix} $$ where $\mathbf{c}$ is a column vector of length $n-1$ and $\mathbf{B}$ is a symmetric matrix of size $n-1$ which is also p.d. We can find an expression for the first diagonal element of $\mathbf{A}^{-1}$ by using a matrix relation $$ \begin{bmatrix} a & \mathbf{c}^\top \\ \mathbf{c} & \mathbf{B} \end{bmatrix} \begin{bmatrix} x \\ \mathbf{y} \end{bmatrix} = \begin{bmatrix} u \\ \mathbf{v} \end{bmatrix} \qquad \text{or} \qquad \left\{ \begin{array}{ll} a x + \mathbf{c}^\top \mathbf{y} &= u \\ \mathbf{c} x + \mathbf{B} \mathbf{y} &= \mathbf{v} \end{array} \right.. $$ Using the system of two equations, we find from the second equation: $\mathbf{y} = \mathbf{B}^{-1}[\mathbf{v} - \mathbf{c}x]$ and the first equation then gives $$ [a - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{c} ] x = u - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{v}. $$
By identification, the coefficient of $x$ between the brackets is the inverse of the diagonal element $\mathbf{A}^{-1}|_{11}$ of $\mathbf{A}^{-1}$. It is $> 0$ because it writes as $\mathbf{d}^\top \mathbf{A} \mathbf{d}$ for $\mathbf{d} := [1, \,-\mathbf{c}^\top \mathbf{B}^{-1}]^\top$. Similarly $a > 0$ and $\mathbf{c}^\top \mathbf{B}^{-1}\mathbf{c} \geq 0$, so $$ \left. \mathbf{A}^{-1} \right|_{11} = \frac{1}{a - \mathbf{c}^\top \mathbf{B}^{-1} \mathbf{c} } \quad \geq \quad \frac{1}{a} = A_{11}^{-1} $$ as wanted.

Now note that the $i$-th diagonal element of $\mathbf{A}$ is the first diagonal element of $\mathbf{A}_{\star} := \mathbf{P} \mathbf{A} \mathbf{P}$ where $\mathbf{P}$ is a symmetric permutation matrix with $\mathbf{P}^{-1} = \mathbf{P}$ and it is easily seen that then the $i$-th diagonal element of $\mathbf{A}^{-1}$ is the first diagonal element of $\mathbf{A}_{\star}^{-1}$ as well. So the result for the general case follows from the special case applied to $\mathbf{A}_{\star}$.

$\endgroup$
4
$\begingroup$

It is also easy to see from the law of total variance. If you have $$x \sim N(\mu, \Sigma)$$ and $x_j$ is the j-the element of $x$ and $\hat{x_j}$ is all but j-th element. Then $$x_j|\hat{x_j}\sim N(\hat{\mu_j}, (\Sigma^{-1}|_{jj})^{-1})$$ Now using the law of total variance. $$\Sigma|_{jj} = Var(x_j) = E[Var(x_j|\hat{x_j})] + Var(E[x_j|\hat{x_j}])\ge E[Var(x_j|\hat{x_j})]= (\Sigma^{-1}|_{jj})^{-1} $$ Therefore $$(\Sigma|_{jj})^{-1}\le (\Sigma^{-1}|_{jj})$$

This also explains when the difference between $(\Sigma|_{jj})^{-1}$ and $(\Sigma^{-1}|_{jj})$ will be large -- when your specific variable is strongly correlated with others.

$\endgroup$
  • 1
    $\begingroup$ +1 Nice answer. The assertion on the conditional variance can easily be proved using blocks. It is always fun to derive results on matrices from Gaussian r.vs. $\endgroup$ – Yves Oct 29 '18 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.