# Low variance components in PCA, are they really just noise? Is there any way to test for it?

I'm trying to decide if a component of a PCA shall be retained, or not. There are a gazillion of criteria based on the magnitude of the eigenvalue, described and compared e.g. here or here.

However, in my application I know that the small(est) eigenvalue will be small compared to the large(st) eigenvalue and the criteria based on magnitude would all reject the small(est) one. This is not what I want. What I am interested in: is there any method known that takes the actual corresponding component of the small eigenvalue into account, in the sense: is it really "just" noise as implied in all the textbooks, or is there "something" of potential interest left? If it is really noise, remove it, otherwise keep it, regardless of the magnitude of the eigenvalue.

Is there some kind of established randomness or distribution test for components in PCA that I am unable to find? Or does anyone know of a reason that this would be a silly idea?

## Update

Histograms (green) and normal approximations (blue) of components in two use cases: once probably really noise, once probably not "just" noise (yes, the values are small, but probably not random). The largest singular value is ~160 in both cases, the smallest, i.e. this singular value, is 0.0xx - way too small for any of the cut-off methods.

What I'm looking for is a way to formalize this ...

• Many of the tests you refer to have exactly the property you ask for: they attempt to distinguish "noise" from "signal." – whuber Feb 19 '14 at 18:53
• I have recently been interested in a similar question, but in a specific situation when you have multiple measurements for each data point. See Choosing number of PCA components when multiple samples for each data point are available. Maybe it applies to your case as well? – amoeba Feb 19 '14 at 21:09
• Using distributional tests on PCs to decide on their randomness sounds as a very interesting idea (that I have never seen applied); something similar is done in ICA, that specifically looks for maximally non-Gaussian components. Doing PCA and then discarding components that are "too Gaussian" has ICA flavour to it and might actually work! – amoeba Dec 22 '14 at 0:36
• They are not necessarily just noise, look at some of the images in my answer here – kjetil b halvorsen May 16 '20 at 2:40

One way of testing the randomness of a small principal component (PC) is to treat it like a signal instead of noise: i.e., try to predict another variable of interest with it. This is essentially principal components regression (PCR).

In the predictive context of PCR, Lott (1973) recommends selecting PCs in a way that maximizes $R^2$; Gunst and Mason (1977) focus on $MSE$. PCs with small eigenvalues (even the smallest!) can improve predictions (Hotelling, 1957; Massy, 1965; Hawkins, 1973; Hadi & Ling, 1998; Jackson, 1991), and have proven very interesting in some published, predictive applications (Jolliffe, 1982, 2010). These include:

• A chemical engineering model using PCs 1, 3, 4, 6, 7, and 8 of 9 total (Smith & Campbell, 1980)
• A monsoon model using PCs 8, 2, and 10 (in order of importance) out of 10 (Kung & Sharif, 1980)
• An economic model using PCs 4 and 5 out of 6 (Hill, Fomby, & Johnson, 1977)

The PCs in the examples listed above are numbered according to their eigenvalues' ranked sizes. Jolliffe (1982) describes a cloud model in which the last component contributes most. He concludes:

The above examples have shown that it is not necessary to find obscure or bizarre data in order for the last few principal components to be important in principal component regression. Rather it seems that such examples may be rather common in practice. Hill et al. (1977) give a thorough and useful discussion of strategies for selecting principal components which should have buried forever the idea of selection based solely on size of variance. Unfortunately this does not seem to have happened, and the idea is perhaps more widespread now than 20 years ago.

Furthermore, excluding small-eigenvalue PCs can introduce bias (Mason & Gunst, 1985). Hadi and Ling (1998) recommend considering regression $SS$ as well; they summarize their article thus:

The basic conclusion of this article is that, in general, the PCs may fail to account for the regression fit. As stated in Theorem 1, it is theoretically possible that the first $(p-1)$ PCs, which can have almost 100% of the variance, contribute nothing to the fit, while the response variable $\text{Y}$ may fit perfectly the last PC which is always ignored by the PCR methodology.

The reason for the failure of the PCR in accounting for the variation of the response variable is that the PCs are chosen based on the PCD [principal components decomposition] which depends only on $\text{X}$. Thus, if PCR is to be used, it should be used with caution and the selection of the PCs to keep should be guided not only by the variance decomposition but also by the contribution of each principal component to the regression sum of squares.

I owe this answer to @Scortchi, who corrected my own misconceptions about PC selection in PCR with some very helpful comments, including: "Jolliffe (2010) reviews other ways of selecting PCs." This reference may be a good place to look for further ideas.

References

- Gunst, R. F., & Mason, R. L. (1977). Biased estimation in regression: an evaluation using mean squared error. Journal of the American Statistical Association, 72(359), 616–628.
- Hadi, A. S., & Ling, R. F. (1998). Some cautionary notes on the use of principal components regression. The American Statistician, 52(1), 15–19. Retrieved from http://www.uvm.edu/~rsingle/stat380/F04/possible/Hadi+Ling-AmStat-1998_PCRegression.pdf.
- Hawkins, D. M. (1973). On the investigation of alternative regressions by principal component analysis. Applied Statistics, 22(3), 275–286.
- Hill, R. C., Fomby, T. B., & Johnson, S. R. (1977). Component selection norms for principal components regression. Communications in Statistics – Theory and Methods, 6(4), 309–334.
- Hotelling, H. (1957). The relations of the newer multivariate statistical methods to factor analysis. British Journal of Statistical Psychology, 10(2), 69–79.
- Jackson, E. (1991). A user's guide to principal components. New York: Wiley.
- Jolliffe, I. T. (1982). Note on the use of principal components in regression. Applied Statistics, 31(3), 300–303. Retrieved from http://automatica.dei.unipd.it/public/Schenato/PSC/2010_2011/gruppo4-Building_termo_identification/IdentificazioneTermodinamica20072008/Biblio/Articoli/PCR%20vecchio%2082.pdf.
- Jolliffe, I. T. (2010). Principal components analysis (2nd ed.). Springer.
- Kung, E. C., & Sharif, T. A. (1980). Regression forecasting of the onset of the Indian summer monsoon with antecedent upper air conditions. Journal of Applied Meteorology, 19(4), 370–380. Retrieved from http://iri.columbia.edu/~ousmane/print/Onset/ErnestSharif80_JAS.pdf.
- Lott, W. F. (1973). The optimal set of principal component restrictions on a least-squares regression. Communications in Statistics – Theory and Methods, 2(5), 449–464.
- Mason, R. L., & Gunst, R. F. (1985). Selecting principal components in regression. Statistics & Probability Letters, 3(6), 299–301.
- Massy, W. F. (1965). Principal components regression in exploratory statistical research. Journal of the American Statistical Association, 60(309), 234–256. Retrieved from http://automatica.dei.unipd.it/public/Schenato/PSC/2010_2011/gruppo4-Building_termo_identification/IdentificazioneTermodinamica20072008/Biblio/Articoli/PCR%20vecchio%2065.pdf.
- Smith, G., & Campbell, F. (1980). A critique of some ridge regression methods. Journal of the American Statistical Association, 75(369), 74–81. Retrieved from https://cowles.econ.yale.edu/P/cp/p04b/p0496.pdf.

• ... and there is no whatsoever guarantee that the effect you need to solve your problem is larger than other effects which are just noise wrt. the problem at hand. I've seen data where 95% of the variance was noise due to some physical effects... – cbeleites unhappy with SX Feb 19 '14 at 23:59
• Very nice review, but (sorry to quibble again) pace Hadi & Ling, selecting the PCs to retain in a regression based on their strong relationship to the response, is as dangerous as selecting the original predictors based on their strong relationship to the response. Cross-validation is essential & shrinkage preferable. Personally I'd prefer a judicious use of PCA together with subject-matter knowledge to guide data reduction on predictors, blind to the response, e.g. using the first PC of groups of predictors measuring much the same thing, or determined by variable clustering. – Scortchi - Reinstate Monica Feb 19 '14 at 23:59
• +1 (long time ago) to this answer, but after reviewing this thread now, I must say that this answer does not answer the original question almost at all: OP was asking about whether one can use any distributional tests on components to judge on their randomness. See also my last comment to the OP. – amoeba Dec 22 '14 at 0:33

Adding to @Nick Stauner's answer, when you're dealing with subspace clustering, PCA is often a poor solution.

When using PCA, one is mostly concerned about the eigenvectors with the highest eigenvalues, which represent the directions towards which the data is 'stretched' the most. If your data is comprised of small subspaces, PCA will solemnly ignore them since they don't contribute much to the overall data variance.

So, small eigenvectors are not always pure noise.