I had thought that a principal components analysis gives us information about the number of degrees of freedom (i.e. independent variables) underlying a system which we may only be able to observe through derived (and possibly correlated) variables. But a simple experiment has made me doubt this:

I generated 1,000 sets of 10 drawings from independent standard normal distributions ($\mathcal{N}(0,1)$) and calculated the PCA for that data matrix. The correlation matrix was very close to diagonal, as I'd hope. The eigenvalues, and corresponding proportion of variances were:

 1.14   12%
 1.11   11%
 1.07   11%
 1.04   10%
 1.02   10%
 0.98   10%
 0.96   10%
 0.93    9%
 0.85    9%
 0.81    8%

This chimed with my expectations: this system has ten independent variables, and the eigenvalues make it clear that all ten principal components carry a significant contribution to the total variance of the system.

Next, for each set of observations, I derived a new set of variables, $\{y_1,...,y_{10}\}$, by taking linear combinations in the form of cumulative averages of the original variables $\{x_1,...,x_{10}\}$:

$$ y_n = \frac{1}{n}\sum_{i=1}^{n}x_i$$

I've lost no information by doing this: all the $\{x\}$ can be exactly recovered from $\{y\}$. However, while the $x$ were independent, the $y$ are clearly correlated (and the correlation matrix confirms this).

I calculated the PCA for $\{y\}$ and obtained this set of eigenvalues:

 1.98   68%
 0.57   19%
 0.18    6%
 0.09    3%
 0.04    1%
 0.03    1%
 0.01    0%
 0.01    0%
 0.01    0%
 0.00    0%

Which suggests that only five or six principal components are necessary to capture virtually all the variance in the system.

Now, to some extent, I see that this makes sense: the $y$ are quite strongly correlated, so e.g. knowing $\{y_1,...,y_9\}$ places tight constraints on $y_{10}$. But the system is still driven by ten underlying independent variables, and all I've done is take linear combinations of them to form my new variables. Shouldn't the PCA be able to see through this?

Clearly my intuition is a little awry. Can anyone help me to see this more clearly?

  • $\begingroup$ Did you do the PCA on the correlation or the covariance matrix? The higher subscripted y's would have lower variance and this could influence your results differently between the 2 methods. $\endgroup$ – Greg Snow Mar 7 '14 at 20:27
  • $\begingroup$ These results were generated on the covariance matrix. However, I also tried redefining my y's so that they all had the same variance - this made very little difference to the distribution of variance between the principal components. $\endgroup$ – atkins Mar 7 '14 at 20:31
  • $\begingroup$ You should provide more details on what you do, or the code for the simulations. I am quite familiar with PCA but I don’t think I understand your questions. $\endgroup$ – Elvis Mar 7 '14 at 21:39
  • $\begingroup$ I think my question is a bit poorly defined. I'm happy with the mechanism and implementation of PCA, but I have a conceptual difficulty about what it tells us in terms of the number of underlying independent variables (aka "latent variables"). I expected the PCA to make it clear that, even in the second case, there are ten underlying independent variables. $\endgroup$ – atkins Mar 11 '14 at 13:13
  • $\begingroup$ You can do a test to see if the eigenvalues are significantly different than zero. $\endgroup$ – Aaron Jul 10 '14 at 14:37

I think the point that I was missing is that PCA does not tell us about the bare number of degrees of freedom, but rather the variance-weighted number of degrees of freedom.

In my first example all ten variables had equal variance (barring sampling noise), and so the PCA shows each successive principal component adding a roughly equal amount of variance.

In the second example the variables I'm working with have a wide range of variances: $y_1$ has a much larger variance than $y_{10}$. The PCA therefore correctly tells me that fewer principal components are required to capture most of the variance, because the higher indexed $y$'s contribute increasingly little to the total variance of the system.

  • $\begingroup$ This answer raises the question of exactly what you mean by "degree of freedom." What is your definition of it and why is it relevant? $\endgroup$ – whuber Jul 10 '14 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.