PCA and degrees of freedom

Question

I had thought that a principal components analysis gives us information about the number of degrees of freedom (i.e. independent variables) underlying a system which we may only be able to observe through derived (and possibly correlated) variables. But a simple experiment has made me doubt this:

I generated 1,000 sets of 10 drawings from independent standard normal distributions ($\mathcal{N}(0,1)$) and calculated the PCA for that data matrix. The correlation matrix was very close to diagonal, as I'd hope. The eigenvalues, and corresponding proportion of variances were:

 1.14   12%
 1.11   11%
 1.07   11%
 1.04   10%
 1.02   10%
 0.98   10%
 0.96   10%
 0.93    9%
 0.85    9%
 0.81    8%

This chimed with my expectations: this system has ten independent variables, and the eigenvalues make it clear that all ten principal components carry a significant contribution to the total variance of the system.

Next, for each set of observations, I derived a new set of variables, $\{y_1,...,y_{10}\}$, by taking linear combinations in the form of cumulative averages of the original variables $\{x_1,...,x_{10}\}$:

$$ y_n = \frac{1}{n}\sum_{i=1}^{n}x_i$$

I've lost no information by doing this: all the $\{x\}$ can be exactly recovered from $\{y\}$. However, while the $x$ were independent, the $y$ are clearly correlated (and the correlation matrix confirms this).

I calculated the PCA for $\{y\}$ and obtained this set of eigenvalues:

 1.98   68%
 0.57   19%
 0.18    6%
 0.09    3%
 0.04    1%
 0.03    1%
 0.01    0%
 0.01    0%
 0.01    0%
 0.00    0%

Which suggests that only five or six principal components are necessary to capture virtually all the variance in the system.

Now, to some extent, I see that this makes sense: the $y$ are quite strongly correlated, so e.g. knowing $\{y_1,...,y_9\}$ places tight constraints on $y_{10}$. But the system is still driven by ten underlying independent variables, and all I've done is take linear combinations of them to form my new variables. Shouldn't the PCA be able to see through this?

Clearly my intuition is a little awry. Can anyone help me to see this more clearly?

Answer 1

I think the point that I was missing is that PCA does not tell us about the bare number of degrees of freedom, but rather the variance-weighted number of degrees of freedom.

In my first example all ten variables had equal variance (barring sampling noise), and so the PCA shows each successive principal component adding a roughly equal amount of variance.

In the second example the variables I'm working with have a wide range of variances: $y_1$ has a much larger variance than $y_{10}$. The PCA therefore correctly tells me that fewer principal components are required to capture most of the variance, because the higher indexed $y$'s contribute increasingly little to the total variance of the system.