I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum, i.e. that $$ \mathbf{\Sigma} = -\mathbf{H}^{-1} $$ where $$ \mathbf{H}_{ij} = \frac{\partial^2}{\partial\theta_i \partial \theta_j} \ln{P}(\hat{\underline{\theta}}| \mathcal{D}). $$ However, in some cases where there are a very large number of model parameters, even if the Hessian is known analytically calculating its inverse might not be tractable.
I'm working on such a case at the minute, and so far have been approximating the uncertainty for the $i$'th parameter $\sigma_i$ by assuming the Hessian is diagonal, such that $$ \sigma_i = \sqrt{\left(-\frac{\partial^2}{\partial\theta^{2}_i} \ln{P}(\hat{\underline{\theta}}| \mathcal{D})\right)^{-1}}. $$ However, intuitively it seems to me that when the Hessian is not diagonal, assuming it is should always underestimate the actual value you would obtain by calculating the inverse, i.e. $\sqrt{\mathbf{\Sigma}_{ii}}$.
Is this actually true? i.e. that $$ \sigma_i \le \sqrt{\mathbf{\Sigma}_{ii}} $$ or equivalently $$ -\frac{1}{\mathbf{H}_{ii}} \le -(\mathbf{H}^{-1})_{ii} $$
covariance-matrix
anddelta-method
would be relevant here. On the other hand, I do not see whybayesian
,variational bayes
are relevant, andprobability
seems too general. $\endgroup$ – Yves Oct 29 '18 at 18:27