# What is the geometric relationship between the covariance matrix and the inverse of the covariance matrix?

The covariance matrix represents the dispersion of data points while the inverse of the covariance matrix represents the tightness of data points. How is the dispersion and tightness related geometrically?

For example, the determinant of the covariance matrix represents the volume of the dispersion of data points. What does the determinant of the inverse of the covariance matrix represent? The determinant is related to volume, but I don't understand how to interpret the volume of the inverse of the covariance matrix (or the volume of the information matrix).

Similarly, the trace represents kind of the mean square error of the data points, but what does the trace of the inverse of the covariance matrix represent?

I don't quite understand how to interpret the inverse of the covariance matrix geometrically, or how it is related to the covariance matrix.

• Interesting question. Could anyone relate it also to Mahalanobis distance ? – javierazcoiti May 4 '20 at 7:15
• – Sextus Empiricus May 4 '20 at 21:50
• – Sextus Empiricus May 4 '20 at 22:00

Before I answer your questions, allow me to share how I think about covariance and precision matrices.

Covariance matrices have a special structure: they are positive semi-definite (PSD), which means for a covariance matrix $$\Sigma$$ of size $$m\text{x}m$$, there are vectors $$x$$ of size $$m\text{x}1$$ such that $$x^T\Sigma x\geq0$$.

Such matrices enjoy a very nice property: they can be decomposed as $$\Sigma=R\Lambda R^T$$, where R is a rotation matrix, and $$\Lambda$$ is a diagonal matrix.

Now that we have the definition out of the way, let's take a look at what this means with the help of a $$\Sigma$$ of size 2x2 (i.e. our dataset has two variables). In the image below, we see in figure a, an identity covariance matrix, which implies no correlation between the data variables. This can be drawn as a circle. Below the image, we see an identity covariance matrix decomposed into its $$\Sigma=R\Lambda R^T$$ form.

In figure b, we see what happens to the geometry if we scale the variance of variables by two different factors. The variables are still uncorrelated, but their respective variances are now m, and n, respectively. Now how do we introduce correlation into the mix? We rotate the ellipse with the help of the rotation matrix, which for the figure c is simply:

$$R = \begin{bmatrix} cos(\theta) & sin(\theta)\\ -sin(\theta) & cos(\theta) \end{bmatrix}$$

Rotation matrices have a nice property: they are orthonormal and $$RR^T=1 \therefore R^T=R^{-1}$$

After that digression, lets come back to our covariance matrix. For $$\Sigma$$: $$\Sigma = R\Lambda R^T = \begin{bmatrix} R_{11} & R_{12}\\ R_{21} & R_{22} \end{bmatrix} \begin{bmatrix} \lambda_1 & 0\\ 0 & \lambda_2 \end{bmatrix} \begin{bmatrix} R_{11} & R_{21}\\ R_{12} & R_{22} \end{bmatrix}$$

Now some fun facts: $$det(\Sigma)=\prod_{i}\lambda_i=\lambda_1\lambda_2$$ and $$tr(\Sigma)=\sum_{i}\lambda_i=\lambda_1+\lambda_2$$. Here is the kicker: $$R$$ actually consists of eigenvectors of $$\Sigma$$ and $$\lambda_i$$ are the eigenvalues.

Finally, note that $$\Sigma^{-1}$$ is also PSD with the following decomposition: $$\Sigma^{-1} = (R\Lambda R^T)^{-1} = (\Lambda R^T)^{-1}(R)^{-1}=(R^T)^{-1}\Lambda^{-1}R^{-1}=R\Lambda^{-1}R^T$$, in the last simplification, we made use of $$RR^T=1$$.

Furthermore: $$\Lambda^{-1} = \begin{bmatrix} \frac{1}{\lambda_1} & 0\\ 0 & \frac{1}{\lambda_2} \end{bmatrix}$$, that is, we simply take inverse of elements along diagonals!

How is the dispersion and tightness related geometrically?

Dispersion gives you a sense of the area of the ellipse compared to that of the circle, tightness is the inverse of dispersion. Dispersion tells you how much area change happens to the unit circle (with uncorrelated variables and identity eigenvectors), tightness tells you how much area you have to undo in the ellipse so it ends up being unit variance.

What does the determinant of the inverse of the covariance matrix represent?

Since $$\Lambda^{-1} = \begin{bmatrix} \frac{1}{\lambda_1} & 0\\ 0 & \frac{1}{\lambda_2} \end{bmatrix}$$, the determinant of precision matrix ($$\frac{1}{\lambda_1\lambda_2}$$) tells you how much area change you have to undo on your data variance so you end up with unit variance. Recall that $$det(\Sigma)=\lambda_1\lambda_2$$.

What does the trace of the inverse of the covariance matrix represent?

Its equal to $$\lambda_1^{-1}+\lambda_2^{-1}$$. Geometric interpretation of $$tr(\Sigma^{-1})$$ is less clear.

• Great answer! The OP also mentions the claim "the trace represents kind of the mean square error of the data points" - any thoughts on that? – Cam.Davidson.Pilon May 11 '20 at 23:25
• the geometric interpretation of the trace is the inverse harmonic mean, no? – Gideon Kogan May 12 '20 at 6:51
• @Cam.Davidson.Pilon Yes, MSE can be computed as trace of the error covariance matrix. GideonKogan: That's vaguely true for $tr(\Sigma^{-1})$, but $tr(\Sigma)=\lambda_1+\lambda_2$. Also though I thought of the harmonic mean, I am not too fluent in it's geometrical interpretation. – PAF May 13 '20 at 13:14
• @Cam.Davidson.Pilon also just realized you're THE Cam DP of the Bayesian book! Nice bumping into you here on CV :) – PAF May 13 '20 at 16:11
• Looking forward to your "Covariance Matrices for Hackers" :) – Cam.Davidson.Pilon May 13 '20 at 17:47