46
$\begingroup$

I'm curious about the nature of $\Sigma^{-1}$. Can anybody tell something intuitive about "What does $\Sigma^{-1}$ say about data?"

Edit:

Thanks for replies

After taking some great courses, I'd like to add some points:

  1. It is measure of information, i.e., $x^T\Sigma^{-1}x$ is amount of info along the direction $x$.
  2. Duality: Since $\Sigma$ is positive definite, so is $\Sigma^{-1}$, so they are dot-product norms, more precisely they are dual norms of each other, so we can derive Fenchel dual for the regularized least squares problem, and do maximization w.r.t dual problem. We can choose either of them, depending on their conditioning.
  3. Hilbert space: Columns (and rows) of $\Sigma^{-1}$ and $\Sigma$ span the same space. So there is not any advantage (other that when one of these matrices is ill-conditioned) between representation with $\Sigma^{-1}$ or $\Sigma$
  4. Bayesian Statistics: norm of $\Sigma^{-1}$ plays an important role in the Bayesian statistics. I.e. it determined how much information we have in prior, e.g., when covariance of the prior density is like $\|\Sigma^{-1}\|\rightarrow 0 $ we have non-informative (or probably Jeffreys prior)
  5. Frequentist Statistics: It is closely related to Fisher information, using the Cramér–Rao bound. In fact, fisher information matrix (outer product of gradient of log-likelihood with itself) is Cramér–Rao bound it, i.e. $\Sigma^{-1}\preceq \mathcal{F}$ (w.r.t positive semi-definite cone, i.e. w.r.t. concentration ellipsoids). So when $\Sigma^{-1}=\mathcal{F}$ the maximum likelihood estimator is efficient, i.e. maximum information exist in the data, so frequentist regime is optimal. In simpler words, for some likelihood functions (note that functional form of the likelihood purely depend on the probablistic model which supposedly generated data, aka generative model), maximum likelihood is efficient and consistent estimator, rules like a boss. (sorry for overkilling it)
$\endgroup$
  • 3
    $\begingroup$ I think PCA picks up eigenvector with large eigenvalues rather than small eigenvalues. $\endgroup$ – wdg Dec 3 '14 at 13:43
  • 2
    $\begingroup$ (3) Is incorrect, because it is tantamount to asserting the columns of $\Sigma^{-1}$ are those of $\Sigma$ (up to a permutation), which is true only for the identity matrix. $\endgroup$ – whuber May 15 '15 at 17:37
15
$\begingroup$

It is a measure of precision just as $\Sigma$ is a measure of dispersion.

More elaborately, $\Sigma$ is a measure of how the variables are dispersed around the mean (the diagonal elements) and how they co-vary with other variables (the off-diagonal) elements. The more the dispersion the farther apart they are from the mean and the more they co-vary (in absolute value) with the other variables the stronger is the tendency for them to 'move together' (in the same or opposite direction depending on the sign of the covariance).

Similarly, $\Sigma^{-1}$ is a measure of how tightly clustered the variables are around the mean (the diagonal elements) and the extent to which they do not co-vary with the other variables (the off-diagonal elements). Thus, the higher the diagonal element, the tighter the variable is clustered around the mean. The interpretation of the off-diagonal elements is more subtle and I refer you to the other answers for that interpretation.

$\endgroup$
  • 3
    $\begingroup$ A strong counter-example to your last statement about off-diagonal elements in $\Sigma^{-1}$ is afforded by the simplest nontrivial example in two dimensions, $\Sigma^{-1}=\left( \begin{array}{cc} \frac{1}{1-\rho ^2} & -\frac{\rho }{1-\rho ^2} \\ -\frac{\rho }{1-\rho ^2} & \frac{1}{1-\rho ^2} \\ \end{array} \right).$ The larger off-diagonal values correspond to more extreme values of the correlation coefficient $\rho,$ which is the opposite of what you appear to be saying. $\endgroup$ – whuber Oct 22 '13 at 14:27
  • $\begingroup$ @whuber Right. I should get rid of the 'absolute' word in the last sentence. Thanks $\endgroup$ – prop Oct 22 '13 at 14:39
  • 3
    $\begingroup$ Thanks, but that still doesn't cure the problem: the relationship you assert between the off-diagonal elements of the inverse and the co-variation does not exist. $\endgroup$ – whuber Oct 22 '13 at 16:43
  • $\begingroup$ @whuber I think it does. In your example, the off-diagonal elements are negative. Therefore, as $\rho$ increases the off-diagonal elements decrease. You can check this by noting the following: at $\rho = 0$ the off-diagonal element is $0$; as $\rho$ approaches $1$ the off-diagonal elements approach $-\infty$ and the derivative of the off-diagonal element with respect to $\rho$ is negative. $\endgroup$ – prop Oct 22 '13 at 17:03
  • 2
    $\begingroup$ My off-diagonal elements are positive when $\rho\lt 0.$ $\endgroup$ – whuber Oct 22 '13 at 19:16
17
$\begingroup$

Using superscripts to denote the elements of the inverse, $1/\sigma^{ii}$ is the variance of the component of variable $i$ that is uncorrelated with the $p-1$ other variables, and $-\sigma^{ij}/\sqrt{\sigma^{ii}\sigma^{jj}}$ is the partial correlation of variables $i$ and $j$, controlling for the $p-2$ other variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.