5
$\begingroup$

What exactly is the problem of having non-invertible covariance matrix? Why is getting the inverse of this matrix so important? This problem is often encountered when doing regressiong works on samples, but even under context of sampling, I do not see how this becomes the problem..

$\endgroup$
1
  • $\begingroup$ Please explain the context of your question. If you were doing a regression analysis, for instance, then it would be immediately obvious why the inverse is important: it is a crucial, unavoidable part of the solution. In other contexts it might or might not be interesting or important to invert the covariance matrix. Your application of the multivariate-analysis tag also suggests you are interested in a covariance matrix of multiple dependent response variables, but this focus is not evident in the post itself. $\endgroup$
    – whuber
    Commented Jun 9, 2015 at 17:21

2 Answers 2

1
$\begingroup$

In short, if $\Sigma$, the covariance matrix in a multivariate normal distribution, is not invertible, then the density is not defined, as the multivariate normal density is

$(2\pi)^{-k/2} |\Sigma| ^{-1/2} e^{(x - \mu)^T \Sigma^{-1}(x-\mu)/2 }$

Of course, this will not be defined if $\Sigma$ is not invertible.

As @whuber points out, in different scenarios there are different heuristic reasons for why this would occur and why, heuristically, this leads to a problem in those scenarios.

In terms of sampling, $sometimes$ this can be easily remedied. In particular $\Sigma$ will not invertible if the determinant = 0. In this case, at least one of the values of $X$ is a merely a linear combination of the other values. For the sake of example, let's suppose that $X_k$ = $\sum_{i = 1}^{k-1} X_i$, and the covariance of $X_{(-k)}$ (i.e. the $X_i$ s.t. $i < k$) is invertible. Then one can sample from $X_{(-k)}$ and then calculate $X_k$ from $X_{(-k)}$. So even though the density function is not properly defined, we can still draw samples from this distribution by being clever.

$\endgroup$
1
$\begingroup$

Since you mark the regression tag, I assume that's the context. People are often interested in information like the following:

i) estimating a coefficient

ii) getting a standard error of an estimated coefficient

iii) testing an estimated coefficient

iv) getting a confidence interval or a prediction interval

these relate in some way to values in the inverse of the X'X matrix or equivalently, (after sweeping out the constant from X), the matrix of sums of squares and cross products which is a multiple of the variance covariance matrix of the predictors. You don't necessarily need to explicitly invert the matrix to get them, but if it's not invertible, they'll either be undefined, not uniquely defined, or infinite.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.