I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no.

If we perform PCA then we calculate linear combinations of the features to build principal components that explain most of the variance of the dataset. However, we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Am I correct?. Should we always reduce dimensions with PCA if needed or there are considerations that need to be taken (as the one above)?

  • 12
    $\begingroup$ Where exactly did you hear that you should always apply PCA? I don't recall anyone doing it even "commonly", not to say "always". $\endgroup$
    – Tim
    Feb 6 '20 at 14:28
  • 2
    $\begingroup$ As an anecdote, in my line of work we're very limited in the amount of data we can collect due to practical limitations. Reducing dimensions after the fact doesn't help us. We need some feature selection method to determine what data we should be collecting. Edit: This comment was for an earlier version which asked whether PCA should always be applied. $\endgroup$
    – TPM
    Feb 6 '20 at 14:34
  • $\begingroup$ @Tim As I said in the original post, I was wondering if it can be applied. Haven't heard it anywhere. $\endgroup$
    – Brandon
    Feb 6 '20 at 14:36
  • 1
    $\begingroup$ I'd ask the opposite question: is PCA ever recommended? :) $\endgroup$
    – gented
    Feb 7 '20 at 13:28
  • 1
    $\begingroup$ Suppose you're doing classification of images. Then if you do PCA, you are throwing away information about which pixels are near each other. This will be awful. $\endgroup$
    – CrabMan
    Feb 7 '20 at 17:31

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize this intuitive idea with the No Free Lunch theorem.)

It's easy enough to construct an example where the eigenvectors to the smallest eigenvalues are the most informative. If you discard this data, you're discarding the most helpful information for your classification or regression problem, and your model would be improved if you had retained them.

More concretely, suppose $A$ is our design matrix, and each column is mean-centered. Then we can use SVD to compute the PCA of $A$. (see: Relationship between SVD and PCA. How to use SVD to perform PCA?)

For an example in the case of a linear model, this gives us a factorization $$ AV = US $$

and we wish to predict some outcome $y$ as a linear combination of the PCs: $AV\beta = y+\epsilon$ where $\epsilon$ is some noise. Further, let's assume that this linear model is the correct model.

In general, the vector $\beta$ can be anything, just as in an ordinary OLS regression setting; but in any particular problem, it's possible that the only nonzero elements of $\beta$ are the ones corresponding to the smallest positive singular values. Whenever this is the case, using PCA to reduce the dimension of $AV$ by discarding the smallest singular values will also discard the only relevant predictors of $y$. In other words, even though we started out with the correct model, the truncated model is not correct because it omits the key variables.

In other words, PCA has a weakness in a supervised learning scenario because it is not "$y$-aware." Of course, in the cases where PCA is a helpful step, then $\beta$ will have nonzero entries corresponding to the larger singular values.

I think this example is instructive because it shows that even in the special case that the model is linear, truncating $AV$ risks discarding information.

Other common objections include:

Some more examples can be found in this closely-related thread (thanks, @gung!): Examples of PCA where PCs with low variance are "useful"

  • $\begingroup$ That makes sense and that's why I was thinking of. Would other techniques that take the class into account, such as LDA, work better in that case? $\endgroup$
    – Brandon
    Feb 6 '20 at 14:52
  • $\begingroup$ @Brandon Seems like "Should we always use LDA?" is a different question. $\endgroup$
    – Sycorax
    Feb 6 '20 at 15:52
  • 3
    $\begingroup$ @SycoraxsaysReinstateMonica: but I suspect the answer is the same for all "Should we always use $MODEL?" questions... $\endgroup$ Feb 6 '20 at 15:54
  • 1
    $\begingroup$ @cbeleitessupportsMonica Sure, but also that's another example of a different question. We have lots of threads about the No Free Lunch theorem and even more threads aimed at debunking various examples of "First Year Students' Dreams." I wouldn't expect to find an answer to those questions in the comments of a thread about PCA. $\endgroup$
    – Sycorax
    Feb 6 '20 at 16:01
  • $\begingroup$ @Tim there is a proof of exactly this situation in a paper I published with my supervisor (pdf) in 2014. We showed that in the case of change detection, if you assume all features are equally likely to be subject to change (which is true in certain situations like EEG seizure monitoring), then the smallest eigenvalue features are better to retain. $\endgroup$ Feb 7 '20 at 7:51

First of all, blindly throwing a model on some data cannot be possibly recommended (you may be able to relax that no-no if you have an infinite amount of independent cases at hand...).

There is a formulation of the no-free lunch theorem that is related to the question: it states that over all possible data sets, no model is better than any other. The usual conclusion from that is that models are superior, iff they are better suited for the particular task at hand (including both what the purpose of the analysis is and particular characteristics of the data).

So, the more sensible question you should ask youself is whether your data has characteristics that make it suitable for PCA.

For example, I work mostly with spectroscopic data. This kind of data has properties that align very well with bilinear models such as PCA or PLS, and much less well with a feature selection picking particular measurement channels (wavelengths, features). In particular, I know for physical and chemical reasons that the information I'm seeking is usually spread out quite "thin" over large regions of the spectrum. Because of that, I routinely use PCA as exploratory tool, e.g. to check whether there is large variance that is not correlated with the outcome I want to predict/study. And possibly even to have a look whether I can find out what the source of such variance is and then decide how to deal with that. I then decide whether to use PCA as feature reduction - whereas I know from the beginning that feature selection picking particular wavelength is hardly ever appropriate.

Contrast that, say, with gene microarray data where I know beforehand that the information is probably concentrated in a few genes with all other genes carrying noise only. Here, feature selection is needed.

we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Of course, and in my field (chemometrics) for regression this observation is the textbook trigger to move on from Principal Component Regression to Partial Least Squares Regression.


Of course not, I don't recall reading/hearing any scientific method's name with the word always, let alone PCA. And, there are many other methods that can be used for dimensionality reduction, e.g. ICA, LDA, variuous feature selection methods, matrix/tensor factorization techniques, autoencoders ...

  • $\begingroup$ Same here. I was just wondering if it can always be applied without losing valuable information in the dimensionality reduction process $\endgroup$
    – Brandon
    Feb 6 '20 at 14:37
  • 5
    $\begingroup$ Careful with tSNE as dimensionality reduction, particularly if you plan to use it downstream for further tasks beyond visualization. There's no global mapping from your input features to tSNE space, so given only the tSNE output, there's no way to go back to the original features. It's great for visualization, but I don't recommending using tSNE-reduced data for classification/regression. $\endgroup$ Feb 7 '20 at 14:36
  • $\begingroup$ It seems I couldn’t stop while giving examples $\endgroup$
    – gunes
    Feb 7 '20 at 14:38

The two major limitations of PCA:

1) It assumes linear relationship between variables.

2) The components are much harder to interpret than the original data.

If the limitations outweigh the benefit, one should not use it; hence, pca should not always be used. IMO, it is better to not use PCA, unless there is a good reason to.

  • $\begingroup$ You can have a linear relationship between variables and still not have a very meaningful compression by maximizing variance retained. $\endgroup$ Feb 9 '20 at 3:49
  • 1
    $\begingroup$ Unlike a probabilistic model, PCA is an algebraic procedure that does not assume anything about the underlying variables, except perhaps for technical conditions to ensure that PCA is feasible. As @amoeba says in another thread, PCA, as a data transformation, dimensionality reduction, exploration, and visualization tool, does not make any assumptions. You can run it on any data whatsoever. The components are just linear combinations, and that is not too difficult to interpret, comparing to basically any other form of combinations. $\endgroup$ Feb 9 '20 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.