66
$\begingroup$

I was wondering whether anyone could point me to some references that discuss the interpretation of the elements of the inverse covariance matrix, also known as the concentration matrix or the precision matrix.

I have access to Cox and Wermuth's Multivariate Dependencies, but what I'm looking for is an interpretation of each element in the inverse matrix. Wikipedia states: "The elements of the precision matrix have an interpretation in terms of partial correlations and partial variances," which leads me to this page. Is there an interpretation without using linear regression? IE, in terms of covariances or geometry?

$\endgroup$
  • 4
    $\begingroup$ did you read the entire Wikipedia page? There is a section on geometry and on conditional independence for the normal distribution. You can find more in this book. $\endgroup$ – NRH May 14 '11 at 7:18
  • $\begingroup$ @NRH The geometry is explained in the partial correlation page, which I'm not even sure how it relates to the concentration matrix yet. Does that graphical models book have an explanation of the elements of the concentration matrix? Thanks! $\endgroup$ – Vinh Nguyen May 16 '11 at 19:09
  • $\begingroup$ see answer below. $\endgroup$ – NRH May 16 '11 at 20:26
  • 2
34
$\begingroup$

There are basically two things to be said. The first is that if you look at the density for the multivariate normal distribution (with mean 0 here) it is proportional to $$\exp\left(-\frac{1}{2}x^T P x\right)$$ where $P = \Sigma^{-1}$ is the inverse of the covariance matrix, also called the precision. This matrix is positive definite and defines via $$(x,y) \mapsto x^T P y$$ an inner product on $\mathbb{R}^p$. The resulting geometry, which gives specific meaning to the concept of orthogonality and defines a norm related to the normal distribution, is important, and to understand, for instance, the geometric content of LDA you need to view things in the light of the geometry given by $P$.

The other thing to be said is that the partial correlations can be read of directly from $P$, see here. The same Wikipedia page gives that the partial correlations, and thus the entries of $P$, have a geometrical interpretation in terms of cosine to an angle. What is, perhaps, more important in the context of partial correlations is that the partial correlation between $X_i$ and $X_j$ is 0 if and only if entry $i,j$ in $P$ is zero. For the normal distribution the variables $X_i$ and $X_j$ are then conditionally independent given all other variables. This is what Steffens book, that I referred to in the comment above, is all about. Conditional independence and Graphical models. It has a fairly complete treatment of the normal distribution, but it may not be that easy to follow.

$\endgroup$
  • 1
    $\begingroup$ Sorry I'm a bit confused wrt to the Wikipedia formula for partial correlation; I've seen several implementations taking ${\bf\color{red} -} \frac{p_{ij}}{ \sqrt{p_{ii} p_{jj}}}$ (with a minus sign). Are you sure that the Wikipedia formula is correct? $\endgroup$ – Sheljohn Jul 1 '15 at 17:07
  • 1
    $\begingroup$ @Sh3ljohn, you are perfectly right. There is a minus missing in the Wikipedia formula. $\endgroup$ – NRH Jul 2 '15 at 12:20
  • $\begingroup$ Isn't the first answer really talking more about the Fisher information than the precision matrix? I mean they coincide in the really special/nice Gaussian case, but they don't coincide generally. Obviously the two concepts are related (Cramer-Rao lower bound, asymptotic distribution of MLE, etc.) but it doesn't seem helpful to conflate them (specifically I came to this question looking for his question about how to distinguish Fisher information and the inverse correlation matrix). $\endgroup$ – Chill2Macht Dec 1 '17 at 4:32
24
$\begingroup$

I like this probabilistic graphical model to illustrate NRH's point that the partial correlation is zero if and only if X is conditionally independent from Y given Z, with the assumption that all involved variables are multivariate Gaussian (the property does not hold in the general case):

enter image description here

(the $y_i$ are Gaussian random variables; ignore T and k)

Source: David MacKay's talk on Gaussian Process Basics, 25th minute.

$\endgroup$
12
$\begingroup$

The interpretation based on partial correlations is probably the most statistically useful, since it applies to all multivariate distributions. In the special case of the multivariate Normal distribution, zero partial correlation corresponds to conditional independence.

You can derive this interpretation by using the Schur complement to get a formula for the entries of the concentration matrix in terms of the entries of the covariance matrix. See http://en.wikipedia.org/wiki/Schur_complement#Applications_to_probability_theory_and_statistics

$\endgroup$
11
$\begingroup$

Covariance matrix can represent relation between all variables while inverse covariance, shoes the relation of element with their neighbors (as wikipedia said partial/pair wise relation).

I borrow the following example from here in 24:10, imagine 5 masses are connected together and vowelling around with 6 springs, covariance matrix would contain correlation of all masses, if one goes right, others can also goes right. but the inverse covariance matrix shoes the relation of those masses that are connected by same spring (neighbors) and it contains many zeros and its not necessary positive.

$\endgroup$
  • 1
    $\begingroup$ Where is this explained in the video? It's an hour long. Thanks! $\endgroup$ – Vinh Nguyen May 16 '11 at 19:07
  • $\begingroup$ you are right, its on 24:10, I think thats the best example to understand the nature of cov matrix and its inverse $\endgroup$ – user4581 May 21 '11 at 11:44
5
$\begingroup$

Bar-Shalom and Fortmann (1988) make mention of the inverse covariance in the context of Kalman filtering as follows:

...[T]here is a recursion for the inverse covariance (or information matrix)

$\mathbf{P}^{-1}(k+1|k+1) = \mathbf{P}^{-1}(k+1|k) + \mathbf{H}'(k+1) \mathbf{R}^{-1}(k+1)\mathbf{H}(k+1)$

...Indeed, a complete set of prediction and update equations, known as the information filter[8, 29, 142], can be developed for the inverse covariance and a transformed state vector $\mathbf{P}^{-1}\hat{\mathbf{x}}$.

The book is indexed at Google.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.