Given a design matrix $X$ for a linear regression model, what is the relationship between the condition number of $X$ and its correlation matrix $R$?
I would be interested in the case of a centered standardized $X.$
Given a design matrix $X$ for a linear regression model, what is the relationship between the condition number of $X$ and its correlation matrix $R$?
I would be interested in the case of a centered standardized $X.$
When the columns of $X$ are standardized, the condition number
By definition, the condition number of $X$ is obtained by considering the effect of $X$ (qua linear transformation) on all possible nonzero vectors $e.$ If we (for the purposes of this thread only) define the "stretch" of $X$ at $e$ to be the amount by which $X$ changes its length,
$$\operatorname{stretch}_X(e) = \frac{|Xe|}{|e|},$$
then the condition number of $X$ measures the range of stretching as
$$\kappa(X) = \frac{\sup_{e\ne 0} \operatorname{stretch}_X(e)}{\inf_{e\ne 0} \operatorname{stretch}_X(e)}.$$
$X$ can always be written in the form
$$X= U\,D\,V^\prime$$
where $U^\prime U$ and $V^\prime V$ are identity matrices and $D$ is a diagonal matrix with numbers $\sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_k \ge 0$ on its diagonal. This is called a Singular Value Decomposition, or SVD, of $X.$
These conditions state that neither $V^\prime$ nor $U$ change the norms: only $D$ can do that. This makes it obvious that $\sigma_1$ is the largest stretch of $X$ and $\sigma_k$ its smallest stretch, whence if $\sigma_k\ne 0,$
$$\kappa(X) = \frac{\sigma_1}{\sigma_k}.$$
The foregoing geometric interpretation of stretching shows that $\kappa$ is well-defined as the "stretching range" of $X$ and this in turn demonstrates that the ratio $\sigma_1/\sigma_k$ is well-defined (that is, it is independent of any SVD of $X$). Thus, we don't have to worry about the details of the SVD, such as whether it is unique.
When the columns of $X$ are standardized, its correlation matrix is proportional to
$$R = c X^\prime X = c (U D V^\prime)^\prime\, (U D V^\prime) = c\ V D U^\prime U D V^\prime = V\,(c\,D^2)\, V\,^\prime.\tag{*}$$
(The constant of proportionality $c$ usually is taken to be $1/n$ or $1/(n-1),$ depending on how the correlation matrix is defined, but the only fact we will need is that this constant is positive.)
$(*)$ is a singular value decomposition of $R$, whence its condition number is the ratio of the largest and smallest singular values of $D^2.$ Since $D^2$ is the diagonal matrix with entries $c\sigma_i^2,$ this shows
$$\kappa(R) = \frac{c\,\sigma_1^2}{c\,\sigma_k^2} = \left(\frac{\sigma_1}{\sigma_k}\right)^2 = \kappa(X)^2.$$
Moreover, if $\sigma_k=0$ then $\sigma_k^2=0$ shows both $X$ and $R^2$ simultaneously have undefined (or, if you like, "infinite") condition numbers.