2
$\begingroup$

Given a design matrix $X$ for a linear regression model, what is the relationship between the condition number of $X$ and its correlation matrix $R$?

I would be interested in the case of a centered standardized $X.$

$\endgroup$
  • $\begingroup$ There is no determinate relationship--but if you were to stipulate that all the column means of $X$ are zero, then something can be said; and if you were to standardize the columns of $X$, then the relation $R=X^\prime X/n$ permits very specific conclusions to be drawn. Which case might you be concerned about? $\endgroup$ – whuber May 12 '19 at 17:14
  • 1
    $\begingroup$ @whuber thanks for the speedy reply. I would be interested in the case of a centred standardized $X$, so the latter. $\endgroup$ – datapipe May 12 '19 at 17:16
3
$\begingroup$

When the columns of $X$ are standardized, the condition number

Definitions

By definition, the condition number of $X$ is obtained by considering the effect of $X$ (qua linear transformation) on all possible nonzero vectors $e.$ If we (for the purposes of this thread only) define the "stretch" of $X$ at $e$ to be the amount by which $X$ changes its length,

$$\operatorname{stretch}_X(e) = \frac{|Xe|}{|e|},$$

then the condition number of $X$ measures the range of stretching as

$$\kappa(X) = \frac{\sup_{e\ne 0} \operatorname{stretch}_X(e)}{\inf_{e\ne 0} \operatorname{stretch}_X(e)}.$$

The SVD

$X$ can always be written in the form

$$X= U\,D\,V^\prime$$

where $U^\prime U$ and $V^\prime V$ are identity matrices and $D$ is a diagonal matrix with numbers $\sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_k \ge 0$ on its diagonal. This is called a Singular Value Decomposition, or SVD, of $X.$

These conditions state that neither $V^\prime$ nor $U$ change the norms: only $D$ can do that. This makes it obvious that $\sigma_1$ is the largest stretch of $X$ and $\sigma_k$ its smallest stretch, whence if $\sigma_k\ne 0,$

$$\kappa(X) = \frac{\sigma_1}{\sigma_k}.$$

The foregoing geometric interpretation of stretching shows that $\kappa$ is well-defined as the "stretching range" of $X$ and this in turn demonstrates that the ratio $\sigma_1/\sigma_k$ is well-defined (that is, it is independent of any SVD of $X$). Thus, we don't have to worry about the details of the SVD, such as whether it is unique.

Correlation

When the columns of $X$ are standardized, its correlation matrix is proportional to

$$R = c X^\prime X = c (U D V^\prime)^\prime\, (U D V^\prime) = c\ V D U^\prime U D V^\prime = V\,(c\,D^2)\, V\,^\prime.\tag{*}$$

(The constant of proportionality $c$ usually is taken to be $1/n$ or $1/(n-1),$ depending on how the correlation matrix is defined, but the only fact we will need is that this constant is positive.)

Solution

$(*)$ is a singular value decomposition of $R$, whence its condition number is the ratio of the largest and smallest singular values of $D^2.$ Since $D^2$ is the diagonal matrix with entries $c\sigma_i^2,$ this shows

$$\kappa(R) = \frac{c\,\sigma_1^2}{c\,\sigma_k^2} = \left(\frac{\sigma_1}{\sigma_k}\right)^2 = \kappa(X)^2.$$

Moreover, if $\sigma_k=0$ then $\sigma_k^2=0$ shows both $X$ and $R^2$ simultaneously have undefined (or, if you like, "infinite") condition numbers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.