# How are the condition numbers of a design matrix and its correlation matrix related?

Given a design matrix $$X$$ for a linear regression model, what is the relationship between the condition number of $$X$$ and its correlation matrix $$R$$?

I would be interested in the case of a centered standardized $$X.$$

• There is no determinate relationship--but if you were to stipulate that all the column means of $X$ are zero, then something can be said; and if you were to standardize the columns of $X$, then the relation $R=X^\prime X/n$ permits very specific conclusions to be drawn. Which case might you be concerned about? – whuber May 12 '19 at 17:14
• @whuber thanks for the speedy reply. I would be interested in the case of a centred standardized $X$, so the latter. – datapipe May 12 '19 at 17:16

When the columns of $$X$$ are standardized, the condition number

### Definitions

By definition, the condition number of $$X$$ is obtained by considering the effect of $$X$$ (qua linear transformation) on all possible nonzero vectors $$e.$$ If we (for the purposes of this thread only) define the "stretch" of $$X$$ at $$e$$ to be the amount by which $$X$$ changes its length,

$$\operatorname{stretch}_X(e) = \frac{|Xe|}{|e|},$$

then the condition number of $$X$$ measures the range of stretching as

$$\kappa(X) = \frac{\sup_{e\ne 0} \operatorname{stretch}_X(e)}{\inf_{e\ne 0} \operatorname{stretch}_X(e)}.$$

### The SVD

$$X$$ can always be written in the form

$$X= U\,D\,V^\prime$$

where $$U^\prime U$$ and $$V^\prime V$$ are identity matrices and $$D$$ is a diagonal matrix with numbers $$\sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_k \ge 0$$ on its diagonal. This is called a Singular Value Decomposition, or SVD, of $$X.$$

These conditions state that neither $$V^\prime$$ nor $$U$$ change the norms: only $$D$$ can do that. This makes it obvious that $$\sigma_1$$ is the largest stretch of $$X$$ and $$\sigma_k$$ its smallest stretch, whence if $$\sigma_k\ne 0,$$

$$\kappa(X) = \frac{\sigma_1}{\sigma_k}.$$

The foregoing geometric interpretation of stretching shows that $$\kappa$$ is well-defined as the "stretching range" of $$X$$ and this in turn demonstrates that the ratio $$\sigma_1/\sigma_k$$ is well-defined (that is, it is independent of any SVD of $$X$$). Thus, we don't have to worry about the details of the SVD, such as whether it is unique.

### Correlation

When the columns of $$X$$ are standardized, its correlation matrix is proportional to

$$R = c X^\prime X = c (U D V^\prime)^\prime\, (U D V^\prime) = c\ V D U^\prime U D V^\prime = V\,(c\,D^2)\, V\,^\prime.\tag{*}$$

(The constant of proportionality $$c$$ usually is taken to be $$1/n$$ or $$1/(n-1),$$ depending on how the correlation matrix is defined, but the only fact we will need is that this constant is positive.)

### Solution

$$(*)$$ is a singular value decomposition of $$R$$, whence its condition number is the ratio of the largest and smallest singular values of $$D^2.$$ Since $$D^2$$ is the diagonal matrix with entries $$c\sigma_i^2,$$ this shows

$$\kappa(R) = \frac{c\,\sigma_1^2}{c\,\sigma_k^2} = \left(\frac{\sigma_1}{\sigma_k}\right)^2 = \kappa(X)^2.$$

Moreover, if $$\sigma_k=0$$ then $$\sigma_k^2=0$$ shows both $$X$$ and $$R^2$$ simultaneously have undefined (or, if you like, "infinite") condition numbers.