# PCA basic: Must eigenvalues converge to zero at high dimension?

Recently, I obtained several PCA plots, and because I am unable to produce eigenvalues for higher dimensions, I tried to extrapolate them based on the available data. The reason why I want to do this is to check whether the PC1 explains sufficient proportion of the total variance.

For an analysis up to the 25th dimension (i.e. up to PC25), PC1 is able to explain 40.6% of total variance. I extrapolated eigenvalues beyond the 25th dimension using an exponential line-of-best-fit based on the available 25 eigenvalues. Here is the plot when I tried to extrapolate to the 50th eigenvalue:

Line-of-best-fit equation: y = 0.7415705 + (64097570 - 0.7415705)/(1 + (x/0.000446242)^1.708334)

From the plot, it seems that the line-of-best-fit is a good model. However, the biggest problem is that the extrapolated eigenvalues never converge to zero at extremely high principal component. This means that if I included up to 50th eigenvalue, the PC1 now explains only 37.5% of the total variance. If extrapolating up to the 1000th eigenvalue, the PC1 explains only a mere amount of 12.2%.

Does this procedure seem reasonable to you? Because of the nature of extrapolation (never converging to zero), the higher eigenvalue I go, the significantly less my PC1 explains the total variance.

Basically, my questions are: Must eigenvalues converge to zero at higher dimensions? Or is it possible for eigenvalues to converge at a finite value?

• Under random matrix theory, the PCs of even just random numbers follow a particular distribution. So in this sense, the histogram of PCs would be more relevant. – John Jul 18 '18 at 11:31

Recall two facts:

• Number of eigenvalues is equal to rank of matrix.
• Sum of eigenvalues of matrix is equal to it's trace (sum of entries on diagonal).

In PCA "matrix" mentioned above is correlation or covariance matrix of your data set, so (provided none of your variables is linear combination of others):

• Number of eigenvalues is equal to number of variables
• Sum of eigenvalues is equal to number of variables (if you use correlation matrix) or to sum of their variances (if you use covariance matrix).

This means that extrapolation you've made suffers from two things:

• You extrapolate "too far away" (do you have 1000 variables to extrapolate to 1000th eigenvalue?)
• You ignore the fact that sum of eigenvalues is well known.

To answer your question: technically eigenvalues do not "converge" to anything, because we have finite number of them.

• I'm studying the PCA of human population. So I'm not very sure how many variables (SNPs) are included. But I imagine it is in the order of thousands. Do you have any idea? – Rudy Winono Jul 18 '18 at 13:22
• To be honest I don't understand the need of extrapolation. Why you can't calculate all eigenvalues? – Łukasz Deryło Jul 18 '18 at 13:33
• I think you want to say "Number of non-zero eigenvalues is equal to the rank of the matrix". (Or computationally the number of singular values higher than a particular tolerance threshold.) – usεr11852 Jul 18 '18 at 23:25
• Yes, you're right. – Łukasz Deryło Jul 19 '18 at 5:55

Must eigenvalues converge to zero at higher dimensions? Or is it possible for eigenvalues to converge at a finite value?

For any matrix of finite dimensions, it has a finite number of eigenvalues. Every finite sequence converges to the final value of the sequence. Consider this definition of convergence of a sequence:

A sequence $x_n$ is said to be convergent to a limit $L$ if given any real positive number $\epsilon$ there exists an integer $n$ such that for all $M\gt n$, $|x_M-L|\lt\epsilon$.

Your plot doesn't really demonstrate much of anything. All you've discovered is that you can order the eigenvalues in decreasing order of size. Since the eigenvalues are not constrained to be equal, some can be larger than others.

But there's not particular reason that the eigenvalues must be distinct. For example, the eigenvalues of $I_k$ the identity matrix of rank $k$ is the value $1$ repeated $k$ times.

I don't know what it means to "extrapolate" an eigenvalue; either you've computed all of the eigenvalues for the matrix, or you haven't. The relationship between the trace of the matrix and the sums of its eigenvalues is a little bit relevant here, but only insofar as if you know the sums of $k-m$ eigenvalues for a matrix of dimension $k\times k$, you know the value of the sum of the remaining $m$. (In some special cases, we can bound eigenvalues by knowing something special about the matrix. This is a somewhat larger topic, though.)

Another relevant observation is that a covariance matrix is positive semi-definite by construction. This implies that the smallest possible eigenvalue is 0. So, if your matrix is rank-deficient, it will have at least 1 eigenvalue of zero. So you know for certain that the smallest value of a rank-deficient matrix must be zero.

The reverse is also true. If your covariance matrix is full rank, then you'll necessarily have all positive eigenvalues. This is true of any full-rank matrix, no matter how large. This is another sense for which your conjecture about “convergence” of eigenvalues to 0 is false.

The shape of the eigenvalue plot is entirely determined by your underlying data generating process and will only follow a smooth trend if the underlying processes have a smooth distribution of contributing to the variation.

This would be quite possible in a real world observation study, but would be unlikely in an experiment where some conditions are well controlled to provide variation greater than any background processes.

If the control is more precise than the inherent variation then you would expect to see the first few pcs explain the controlled variation before a sharp drop for pcs explaining uncontrolled background processes.

There is an upper bound to the number of non zero eigenvalues you can obtain, the rank of the matrix.

If your matrix has fewer samples (n) than variables (v) you expect to see the remaining variance to be zero by the nth PC as you can describe the full variance of the data set by using n transformations of the original variables.

If v is your rank then you can simply use the original variables to describe the full variance of the dataset, so the eigenvalue remaining is zero. Remember that PCA is just a series of simultaneous equations.