How to prove higher eigenvalues correspond to significant principal components?

How to prove higher eigenvalues correspond to significant principal components? Is there any explanation or at least intuition for that, if not a proof?

I would like to convince myself with some intuition that eigenvalues of covariance matrix correspond to principal components and higher eigenvalues correspond to significant principal components etc.

• PCA has nothing to do with "significance" in the statistical sense of p-value. If you PCA-analyse covariance matrix then eigenvalues are the variances of the components (see e.g. stats.stackexchange.com/q/22569/3277). – ttnphns May 17 '14 at 8:19
• Even glossing significance as importance or interest, this is not necessarily true. It's often been pointed that PCs that are nearly constant (have very low eigenvalue) could be interesting as showing (near) invariants. – Nick Cox May 17 '14 at 10:01

Your data is a set of points in an n-space. Let's say that you have two variables, i.e. your data matrix is $N\times 2$. If there is no correlation the scatterplot is bounded by a circle, otherwise by an ellipse, the eigenvectors of the covariance matrix are parallel to the ellipse axes and their lengths are proportional to the square root of their respective eigenvalues. This is why the eigenvalues correspond to (directions and amounts of) variability. Look a this figure (R code below):

dat <- read.table("http://www.ces.clemson.edu/~calvinw/MthSc807/MthSc807/data/T3_7_SONS.DAT")
x <- dat$V1 - mean(dat$V1)
y <- dat$V2 - mean(dat$V2)
plot(x,y, xlim=c(-30,30), ylim=c(-30,30))
# ellipse
library(ellipse)
S <- cov(dat[,1:2])
lines(ellipse(S))
# eigenvectors and their length (square root of eigenvalues)
eig <- eigen(S)
lambda <- eig$values a1 <- eig$vectors[,1]
a2 <- eig$vectors[,2] arrows(0, 0, sqrt(lambda[1])*a1[1], sqrt(lambda[1])*a1[2], length=0.1) arrows(0, 0, sqrt(lambda[2])*a2[1], sqrt(lambda[2])*a2[2], length=0.1)  If your data matrix is$N\times 3\$ you get an ellipsoid, but if it is flat like a cuttlebon, then the two longer eigenvectors correspond to "principal components" of variability.

Parallel analysis may suit your purposes. It does not involve a null hypothesis significance test, and its results should not be confused with statistical significance in the usual sense of the term. However, it gives a fairly intuitive indication of how large eigenvalues should be at a minimum to make sense as indicators of valid dimensional constructs.

Parallel analysis simulates random samples of data from uncorrelated distributions that resemble your sample. It calculates the eigenvalues of these simulated data, and compares these to your sample's eigenvalues, which should be higher if your variables relate at all...at least on as many components as make sense to extract from your data. When your data's eigenvalues drop below the size of the eigenvalues of the simulated data, you can be fairly confident that those components describe too little variance to be meaningful and replicable.

The intuitiveness of this comparison derives from the fact that the simulated data come from distributions that have no common components (because they are uncorrelated). Thus any component extracted from those data only describes covariance due to sampling error. Again, parallel analysis is not a significance test per se; it won't tell you the probability that your eigenvalues are larger than they should be due to sampling error. It does give you a basis for comparison though: it indicates how big eigenvalues can be expected to grow due entirely to sampling error.

BTW, it's worth noting that parallel analysis has been referred to as "a method for determining significant principal components". I think this appeals to a less conventional understanding of "significant" – as I've said, one that is not based on the probability of the data given the null. See this:

Franklin, S. B., Gibson, D. J., Robertson, P. A., Pohlmann, J. T., & Fralish, J. S. (1995). Parallel analysis: A method for determining significant principal components. Journal of Vegetation Science, 6(1), 99–106. Retrieved from http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1004&context=pb_pubs.

• OK, I will see those. I appreciate your effort in answering. – dexterdev May 18 '14 at 6:03
• @Nick: I am not at all an expert on parallel analysis, but as far as I understand it amounts to generating a random data matrix and obtaining its eigenvalue spectrum. Surely one could do it, say, 10000 times and obtain a Monte Carlo distribution of the eigenvalue spectra. Comparing the empirical spectrum to this distribution would seem to be a bona fide significance test. Or is this not called "parallel analysis" anymore? – amoeba says Reinstate Monica May 18 '14 at 8:48
• I'm not much of an expert on it either, but I was wondering about that too... – Nick Stauner May 18 '14 at 8:58
• According to @Alexis (see e.g. here), there is "Monte Carlo parallel analysis", "which uses a high centile of eigenvalues of random data, rather than the mean", exactly as I wrote above. I am just saying that you made a very strong point in your answer that parallel analysis is not a significance test, whereas there is an obvious variant of it which actually is a significance test (how well it works is another question). – amoeba says Reinstate Monica May 19 '14 at 9:22