# Low rank plus diagonal approximation of covariance of model parameters

In this paper regarding Bayesian Deep Learning, in Section 3.4, the authors want to approximate the covariance of a distribution over the model parameters $$\theta$$. They first get an estimate of the diagonal covariance by approximating $$$$\Sigma_{diag} = E[\theta^2] - E[\theta]^2$$$$ where the expected values are estimated from $$T$$ samples. They also compute a low-rank approximation $$\Sigma_{low-rank}$$ of the full covariance matrix using only the last $$K$$ samples.

Then, they approximate the covariance matrix by $$\Sigma = \frac{1}{2} (\Sigma_{diag} - \Sigma_{low-rank})$$

What is the reason for doing this? Why do we need a low rank approximation? And why is the sum of the diagonal and the low rank a good estimate of the covariance?

The reason they use a low rank plus diagonal approximation is to be parameter efficient. In the case of a deep neural network (the paper is about training Bayesian Neural Networks), the number of parameters in the networks (which becomes latent variables in the Bayesian model) is very large.

Considering all the covariances across all these model parameters would require a number of variational parameters roughly proportional to the square of the number of model parameters, which is probably too many for a computer to handle.

By using a diagonal plus low rank approximation, you need only a number of variational parameters roughly proportional to the number of model parameters directly.

The diagonal plus low rank approximation is more efficient than just a diagonal approximation, or a block diagonal approximation.