Why does the condition number of the covariance matrix explode as number of variables increases?

Question

From asset returns of $N$ stocks, the symmetric covariance matrix sized $N\times N$ is constructed, which treats the asset returns as variables.

• When the number of variables $N$ is fairly low like $N=5$ or $N=12$, the condition number is relatively low around cond$=1-5$.
• As I increase the number of variables in the covariance matrix though, such as $N = 30$ or $N=50$, it already explodes to the cond$=500^+$ range.

This discussion explains the worsening of condition number for when the features/variables have different scales, but this obviously doesn't apply to my case because all of the variables are in the same units: returns.

What my case does have in common with theirs though is that the standard deviations of the variables are higher or lower than one another (stocks being more or less risky than one another), but I wouldn't call this a difference in scale.

Why is the covariance matrix condition number so reactive to an increase in the number of variables $N$?

Answer 1

Explaining this in the comments was a little limiting, apologies:

Assuming centered data matrix $X$, then your covariance matrix $M = X^T X$. This will have high condition number if the range of singular values of $M$ is high, because condition number is defined $\kappa(M) = \frac{s_{\text{max}}}{s_{\text{min}}}$ where $s_{\text{max}}$ and $s_{\text{min}}$ are the min and max singular values of $M$.

Let's look at what features of $X$ will produce a high range in the signular values. In general, the singular values of $M$ satisfy: $$ M = \sum_{i=1}^N s_i v_i v_i^T = V \Sigma V^T $$ Where the $v_i$ (the columns of V) are some orthogonal vectors, and $\Sigma$ is a diagonal matrix whose on-diagonal elements are the singular values $s_i$ and everything else is 0. Since $V^{-1} = V^T$ (because orthogonal) we can see that: $$ \Sigma = V^T M V = V^T X^T X V = (XV)^T(XV) $$ Letting $(XV)_i$ denote the $i^{\text{th}}$ column of $XV$, matrix multiplication is set up so that: $$ s_i = (XV)_i^T (XV)_i = | (XV)_i |^2 $$ Thus, if some columns of $XV$ are very big and others are very small, then some $s_i$ will be very big and others will be very small. When this happens, then your condition number will be large (by the definition of condition number).

Recall from linear algebra that, since $V$ is an orthogonal matrix, the columns of $XV$ are just rotations of the columns of $X$. In effect, what multiplication by $V$ is doing is rotating your data matrix so that the directions along which it varies the most are aligned with the cardinal directions of the data space. The large columns of $XV$ correspond to the directions along which the data varies a lot, and the small columns correspond to the directions where the data varies only a little bit. For your data, it sounds like it's the case that only $D << N$ columns of $XV$ have any appreciable magnitude, and that the rest of very very small. This number $D$ doesn't grow much, but $N$ does. As $N$ grows, the data varies less and less along each new dimension, bringing $s_{\text{min}}$ lower and lower, and causing $\kappa(M)$ to explode.