In linear models, in my book, the condition index is defined as $\sqrt{\lambda_{max} \over \lambda_{min}}$ where $\lambda_{max}$ is the maximum eigenvalue of $ZZ^*$, i.e., the correlation matrix of the independent variables, and $\lambda_{min}$ is the minimum eigenvalue.

I could not find out why the condition index helps me to find out multicollinearity. So, I've come up with my own explanation:

In principal components analysis, eigenvalues explain the variance of data in each of the eigenvectors. Thus, having a large ratio of max eigenvalue to min eigenvalue means that the data can be explained by a smaller number of eigenvectors than the number of independent variables, which means that there is collinearity among independent variables.

Given these, here are my questions.

  1. Is my own explanation correct?

  2. What's a better explanation? (Please understand that I do not have strong linear algebra knowledge. So, please explain in very easy words.)

  • 2
    $\begingroup$ For those seeking more information on the Web, "condition index" is better known as condition number (or the square root thereof). $\endgroup$ – whuber Dec 30 '11 at 15:24
  • 2
    $\begingroup$ The explanation of eigen values is correct. Explanation on ratio is not perfect. The eigen values are calculated of scaled version of $Z'Z$.When there is no collinearity at all, the eigenvalues, condition indices and condition number will all equal one. As collinearity increases, eigenvalues will be both greater and smaller than 1, and the condition indices and the condition number will increase. $\endgroup$ – vinux Dec 30 '11 at 15:31

Your thinking is basically correct.

Let $Z$ be an $M \times N$ matrix, i.e., $N$ observations of $M$ random variables (or features).

A condition number that "equals infinity" implies that, for any of the $M$ observations, any one of the $N$ variables can be described as a weighted sum of the other $(N-1)$ variables. That defines exact multicollinearity.

Appendix: $\lambda_{min} = 0$ implies that there exists a nonzero eigenvector $q$ such that

$$ZZ^Tq = \lambda_{min}q = 0$$

$$\Rightarrow q^TZZ^Tq = \lambda_{min}q^Tq = \lambda_{min} 1 = 0.$$

Since $0 = q^T ZZ^T q = (Z^T q)^2 \geq 0 $,

$$Z^T q = 0$$

which implies that the nullspace of $Z^T$ is non-trivial.

I realize this doesn't address the case when $0 < \lambda_{min} \ll \lambda_{max}$, i.e. approximate multicollinearity.


For a very thorough explanation, see David Belsley's books, either (with Kuh and Welsch) Regression Diagnostics or Conditioning Diagnostics. The latter book is more concentrated on condition indices, but it's older and out of print. The former covers more types of problems in regression, but should give you plenty of info on conditioning indices.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.