I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's.

If I were to try to approximate this covariance matrix with a diagonal one $\widehat{\Sigma} = \textrm{Diag}(s_1, \ldots, s_n)$ using Variational Inference (to estimate the marginal variances), I would get that $$\widehat{\Sigma} = (\textrm{Diag}(\Sigma^{-1}))^{-1} \qquad .$$

I know that, for "thin" (almost singular) Gaussian distributions, this is an extremely poor approximation. For example, for the 2D Gaussian $$\Sigma=\left[\begin{array}{cc} 1 & \rho^{2}\\ \rho^{2} & 1 \end{array}\right] \qquad ,$$ we'll approximate the marginal variance to be $\rho \cdot (1-\rho)$, which is terrible for a high correlation.

Is there any simple way to characterize or describe this effect of VI giving a terrible approximation for the diagonals of "thin" (almost singular) covariance matrices in higher dimensions? For example, "the approximation is roughly as bad as the condition number". Or perhaps, "the approximation heavily favours the smallest eigenvalue".

  • $\begingroup$ It depends on how you measure the quality of approximation. A natural idea is to recognize that the smallest number $\lambda\ge 0$ for which the diagonal of $\Sigma^{-1}-\lambda$ becomes singular--and therefore fails to produce any reasonable value of $\hat\Sigma$ when inverted--is the reciprocal of the largest eigenvalue of $\Sigma.$ I don't know how relevant this might be to whatever application you have in mind, though. $\endgroup$ – whuber May 31 '19 at 20:13
  • 2
    $\begingroup$ I'm trying to approximate the marginals, so the diagonal of the original $\Sigma$. The point is that, if $\Sigma$ corresponds to a highly correlated Gaussian, then $\widehat{\Sigma}$ will be an extremely poor approximation for its diagonal. I've updated the post to reflect this. I don't have an application, per se. It's just, in a paper I'm writing, I wish to succinctly state that variational inference is bad at approximating the marginals of "thin" (almost-singular) gaussians, and I want to relate this to some reasonable quantity, like the conditional number or lowest EV of $\Sigma$. $\endgroup$ – chausies May 31 '19 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.