Choosing number of principal components to retain

Question

One method that was suggested to me is to look at a scree plot and check for "elbow" to determine the correct number of PCs to use. But if the plot is not clear, does R have a calculation to determine the number?

fit <- princomp(mydata, cor=TRUE)

Answer 1

The following article : Component retention in principal component analysis with application to cDNA microarray data by Cangelosi and Goriely gives a rather nice overview of the standard rule of thumbs to detect the number of components in a study. (Scree plot, Proportion of total variance explained, Average eigenvalue rule, Log-eigenvalue diagram, etc.) Most of them are quite straightforward to implement in R.

In general if your scree plot is very inconclusive then you just need to "pick your poison". There is no absolute right or wrong for any data as in reality the number of PCs to use actually depends on your understanding of the problem. The only data-set you can "really" know the dimensionality of is the one you constructed yourself. :-) Principal Components in the end of the day provide the optimal decomposition of the data under an RSS metric (where as a by-product you get each component to represent a principal mode of variation) and including or excluding a given number of components dictates your perception about the dimensionality of your problem.

As matter of personal preference, I like Minka's approach on this Automatic choice of dimensionality for PCA which based on a probabilistic interpretation of PCA but then again, you get into the game of trying to model the likelihood of your data for a given dimensionality. (Link provides Matlab code if you wish to follow this rationale.)

Try to understand your data more. eg. Do you really believe that 99.99% of your data-set's variation is due to your model's covariates? If not probably you probably don't need to include dimensions that exhibit such a small proportion of total variance. Do you think that in reality a component reflects variation below a threshold of just noticeable differences? That again probably means that there is little relevance in including that component to your analysis.

In any case, good luck and check your data carefully. (Plotting them makes wonders also.)

Answer 2

There has been very nice subsequent work on this problem in the past few years since this question was originally asked and answered. I highly recommend the following paper by Gavish and Donoho: The Optimal Hard Threshold for Singular Values is 4/sqrt(3)

Their result is based on asymptotic analysis (i.e. there is a well-defined optimal solution as your data matrix becomes infinitely large), but they show impressive numerical results that show the asymptotically optimal procedure works for small and realistically sized datasets, even under different noise models.

Essentially, the optimal procedure boils down to estimating the noise, $\sigma$, added to each element of the matrix. Based on this you calculate a threshold and remove principal components whose singular value falls below the threshold. For a square $n \times n$ matrix, the proportionality constant 4/sqrt(3) shows up as suggested in the title:

$$\lambda = \frac{4\sigma\sqrt{n}}{\sqrt{3}}$$

They also explain the non-square case in the paper. They have a nice code supplement (in MATLAB) here, but the algorithms would be easy to implement in R or anywhere else: https://purl.stanford.edu/vg705qn9070

Caveats:

• If you have missing data, I'm not sure this will work
• If each feature in your dataset has different noise magnitudes, I'm not sure this will work (though whitening could probably get around this under certain assumptions)
• Would be interesting to see if similar results hold for other low-rank matrix factorizations (e.g. non-negative matrix factorization).

Answer 3

The problem with Kaiser's criterion (all eigenvalues greater than one) is that the number of factors extracted is usually about one third the number of items or scales in the battery, regardless of whether many of the additional factors are noise. Parallel analysis and the scree criterion are generally more accurate procedures for determining the number of factors to extract (according to classic texts by Harmon and Ledyard Tucker as well as more recent work by Wayne Velicer.