8
$\begingroup$

I have two correlation matrices, one with a condition number of 9 and the other with a condition number of 70. From what i have read, it will appear that the first matrix is better conditioned than the other based on these figures alone, but i am struggling to really interpret how much better one correlation matrix is relative to the other, or if there are other ways to really interpret the condition number.

Apologies for my English, if my post is not clear, please let me know and i will try to write it again.

$\endgroup$
8
$\begingroup$

The condition number of a correlation matrix is not of great interest in its own right. It comes into its own when that matrix gives the coefficients of a set of linear equations, as happens for multiple linear regression using standardized regressors.

Belsley, Kuh, and Welsh--who were among the first to point out and systematically exploit the relevance of the condition number in this context--have a nice explanation, which I will broadly quote. They begin by giving a definition of

the spectral norm, denoted $||A||$ and defined as $$||A|| \equiv {\sup}_{||z||=1}||Az||.$$

Geometrically, its the maximum amount by which $A$ will rescale the unit sphere: its maximum "stretch," if you will. They point out the obvious relations that $||A||$ therefore is the largest singular value of $A$ and $||A^{-1}||$ is the reciprocal of the smallest singular value of $A$ (when $A$ is invertible). (I like to think of this as the maximum "squeezing" of $A$.) They then assert that $||A||$ actually is a norm, and add the (easily proven) facts

$||Az|| \le ||A|| \cdot ||z|| \tag{4}$

$||AB|| \le ||A||\cdot ||B|| \tag{5} $ for all commensurate $A$ and $B$.

These remarks are then applied:

We shall now see that the spectral norm is directly relevant to an analysis of the conditioning of a linear system of equations $Az = c, A $ $n\times n$ and nonsingular with solution $z=A^{-1}c$. We can ask how much the solution vector $z$ would change $(\delta z)$ if there were small changes or perturbations in the elements of $c$ or $A$, denoted $\delta c$ and $\delta A$. In the event that $A$ is fixed but $c$ changes by $\delta c$, we have $\delta z = A^{-1}\delta c$, or $$||\delta z|| \le ||A ^{-1} || \cdot || \delta c ||.$$ Further, employing property $(4)$ above to the equation system, we have $$||c|| \le ||A|| \cdot ||z||;$$ and from multiplying these last two expressions we obtain $$\frac{||\delta z||}{||z||} \le ||A|| \cdot ||A^{-1}|| \cdot \frac{||\delta c || }{||c||}.$$

That is, the magnitude $||A||\cdot ||A^{-1}||$ provides a bound for the relative change in the length of the solution vector $z$ that can result from a given relative change in the length of $c$. A similar result holds for perturbations in the elements of the matrix $A$. Here it can be shown that $$\frac{||\delta z||}{||z + \delta z||} \le ||A|| \cdot ||A^{-1}|| \cdot \frac{||\delta A||}{||A||}.$$

(The key step in this demonstration, which is left as an exercise, is to observe $\delta z = -A^{-1}(\delta A)(z + \delta z)$ and apply norms to both sides.)

Because of its usefulness in this context, the magnitude $||A||\cdot ||A^{-1}||$ is defined to be the condition number of the nonsingular matrix $A$ ... .

(Based on the earlier characterizations, we may conceive of the condition number as being a kind of "aspect ratio" of $A$: the most it can stretch any vector times the most it can squeeze any vector. It would be directly related to the maximum eccentricity attained by any great circle on the unit sphere after being operated on by $A$.)

The condition number bounds how much the solution $z$ of a system of equations $Az=c$ can change, on a relative basis, when its components $A$ and $c$ are changed.

However, these inequalities are not tight: for any given $A$, the extent to which the bounds are reasonably accurate representations of actual changes depends on $A$ and the changes $\delta A$ and $\delta c$. Condition numbers are assertions about worst cases. Thus, a matrix with condition number $9$ can be considered to be $70/9$ times better than one with condition number $70$, but that does not necessarily mean that it will be precisely that much better (at not propagating errors) than the other.

Reference

Belsley, Kuh, & Welsch, Regression Diagnostics. Wiley, 1980: Section 3.2.

$\endgroup$
1
$\begingroup$

Super high condition number would mean that some variables are highly correlated. 70 is not that big of a condition number to me.

High or low condition number doesn't mean that one correlation matrix is "better" than the other. All it means is that variables are more correlated or less. Whether it's good or not depends on the application.

UPDATE: I'm assuming you don't have a super high dimensional case, because in this case @whuber is right, and you may end up with low correlation but high condition number. Intuitively, it's easy to see why. Consider, a matrix where all elements are equal to $\rho$, except the diagonals that are ones. In this case if you take any two columns, they'll look very similar to each other. In fact, they'll differ in exactly only two rows, where one of them is 1 and the other is $\rho$. If you have a very high dimensional matrix, then from linear algebra point of view these are almost the same columns, i.e. the matrix will look kinda rank-defficient.

$\endgroup$
  • 3
    $\begingroup$ I believe the situation is subtler than that. If you were to name a "super high" condition number--call it $\gamma$--and also to specify a positive correlation coefficient corresponding to not "highly correlated"--call it $r$--then I could find a correlation matrix with condition number $\gamma$, yet containing no correlation exceeding $r$. This can be done with any $n\times n$ matrix whose off-diagonal correlations are all equal to $r$, provided $$n \ge \frac{(\gamma-1)(1-r)}{r}.$$ This shows that you must take the dimension $n$ into account. (This is another "curse of dimensionality.") $\endgroup$ – whuber Aug 21 '15 at 19:47
  • $\begingroup$ @whuber, right, I didn't think of this. Your equation though does not handle $r=0$, i.e. identity matrix whose condition number is 1. $\endgroup$ – Aksakal Aug 21 '15 at 20:04
  • $\begingroup$ That's why I stipulated $r$ had to be positive. $\endgroup$ – whuber Aug 21 '15 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.