Assuming I have a data set with $d$ dimensions (e.g. $d=20$) so that each dimension is i.i.d. $X_i \sim U[0;1]$ (alternatively, each dimension $X_i \sim \mathcal N[0;1]$) and independent of each other.

Now I draw a random object from this dataset and take the $k=3\cdot d$ nearest neighbors and compute PCA on this set. In contrast to what one might expect, the eigenvalues aren't all the same. In 20 dimensions uniform, a typical result looks like this:

0.11952316626613427, 0.1151758808663646, 0.11170020254046743, 0.1019390988585198,
0.0924502502204256, 0.08716272453538032, 0.0782945015348525, 0.06965903935713605, 
0.06346159593226684, 0.054527131148532824, 0.05346303562884964, 0.04348400728546128, 
0.042304834600062985, 0.03229641081461124, 0.031532033468325706, 0.0266801529298156, 
0.020332085835946957, 0.01825531821510237, 0.01483790669963606, 0.0068195084468626625

For normal distributed data, the results appear to be very similar, at least when rescaling them to a total sum of $1$ (the $\mathcal N[0;1]^d$ distribution clearly has a higher variance in the first place).

I wonder if there is any result that predicts this behavior? I'm looking for a test if the series of eigenvalues is somewhat regular, and how many of the eigenvalues are as expected and which ones significantly differ from the expected values.

For a given (small) sample size $k$, is there a result if a correlation coefficient for two variables is significant? Even i.i.d. variables will have a non-0 result occasionally for low $k$.

  • 1
    $\begingroup$ hmmm, could you print those results with fewer sig figs? I cannot parse them easily... $\endgroup$ – shabbychef Jun 28 '12 at 22:19
  • $\begingroup$ Well, as you can see the magnitude is of interest. Naively, one would expect them to all have the same magnitude. $\endgroup$ – Has QUIT--Anony-Mousse Jun 29 '12 at 6:17

There is a large literature on the distribution of eigenvalues for random matrices (you can try googling random matrix theory). In particular, the Marcenko-Pastur distribution predicts the distribution of eigenvalues for the covariance matrix of $i.i.d.$ data with mean of zero and equal variance as the number of variables and observations goes to infinity. Closely related is Wigner's semicircle distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.