How to interpret an inverse covariance or precision matrix?

Question

I was wondering whether anyone could point me to some references that discuss the interpretation of the elements of the inverse covariance matrix, also known as the concentration matrix or the precision matrix.

I have access to Cox and Wermuth's Multivariate Dependencies, but what I'm looking for is an interpretation of each element in the inverse matrix. Wikipedia states: "The elements of the precision matrix have an interpretation in terms of partial correlations and partial variances," which leads me to this page. Is there an interpretation without using linear regression? IE, in terms of covariances or geometry?

Answer 1

There are basically two things to be said. The first is that if you look at the density for the multivariate normal distribution (with mean 0 here) it is proportional to $$\exp\left(-\frac{1}{2}x^T P x\right)$$ where $P = \Sigma^{-1}$ is the inverse of the covariance matrix, also called the precision. This matrix is positive definite and defines via $$(x,y) \mapsto x^T P y$$ an inner product on $\mathbb{R}^p$. The resulting geometry, which gives specific meaning to the concept of orthogonality and defines a norm related to the normal distribution, is important, and to understand, for instance, the geometric content of LDA you need to view things in the light of the geometry given by $P$.

The other thing to be said is that the partial correlations can be read of directly from $P$, see here. The same Wikipedia page gives that the partial correlations, and thus the entries of $P$, have a geometrical interpretation in terms of cosine to an angle. What is, perhaps, more important in the context of partial correlations is that the partial correlation between $X_i$ and $X_j$ is 0 if and only if entry $i,j$ in $P$ is zero. For the normal distribution the variables $X_i$ and $X_j$ are then conditionally independent given all other variables. This is what Steffens book, that I referred to in the comment above, is all about. Conditional independence and Graphical models. It has a fairly complete treatment of the normal distribution, but it may not be that easy to follow.

Answer 2

I like this probabilistic graphical model to illustrate NRH's point that the partial correlation is zero if and only if X is conditionally independent from Y given Z, with the assumption that all involved variables are multivariate Gaussian (the property does not hold in the general case):

(the $y_i$ are Gaussian random variables; ignore T and k)

Source: David MacKay's talk on Gaussian Process Basics, 25th minute.

Answer 3

The interpretation based on partial correlations is probably the most statistically useful, since it applies to all multivariate distributions. In the special case of the multivariate Normal distribution, zero partial correlation corresponds to conditional independence.

You can derive this interpretation by using the Schur complement to get a formula for the entries of the concentration matrix in terms of the entries of the covariance matrix. See http://en.wikipedia.org/wiki/Schur_complement#Applications_to_probability_theory_and_statistics

Answer 4

Covariance matrix can represent relations between all variables while inverse covariance shows the relations of elements with their neighbors (as wikipedia said partial/pair wise relations).

I borrow the following example from here in 24:10. Imagine 5 masses are connected together and vowelling around with 6 springs. The covariance matrix would contain correlation of all masses, if one goes right, others can also goes right, but the inverse covariance matrix shows the relation of those masses that are connected by same springs (neighbors) and it contains many zeros and it is not necessary positive.

Answer 5

Bar-Shalom and Fortmann (1988) make mention of the inverse covariance in the context of Kalman filtering as follows:

...[T]here is a recursion for the inverse covariance (or information matrix)

$\mathbf{P}^{-1}(k+1|k+1) = \mathbf{P}^{-1}(k+1|k) + \mathbf{H}'(k+1) \mathbf{R}^{-1}(k+1)\mathbf{H}(k+1)$

...Indeed, a complete set of prediction and update equations, known as the information filter[8, 29, 142], can be developed for the inverse covariance and a transformed state vector $\mathbf{P}^{-1}\hat{\mathbf{x}}$.

The book is indexed at Google.