22
$\begingroup$

I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no.

If we perform PCA then we calculate linear combinations of the features to build principal components that explain most of the variance of the dataset. However, we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Am I correct?. Should we always reduce dimensions with PCA if needed or there are considerations that need to be taken (as the one above)?

$\endgroup$
  • 11
    $\begingroup$ Where exactly did you hear that you should always apply PCA? I don't recall anyone doing it even "commonly", not to say "always". $\endgroup$ – Tim Feb 6 at 14:28
  • 2
    $\begingroup$ As an anecdote, in my line of work we're very limited in the amount of data we can collect due to practical limitations. Reducing dimensions after the fact doesn't help us. We need some feature selection method to determine what data we should be collecting. Edit: This comment was for an earlier version which asked whether PCA should always be applied. $\endgroup$ – TPM Feb 6 at 14:34
  • $\begingroup$ @Tim As I said in the original post, I was wondering if it can be applied. Haven't heard it anywhere. $\endgroup$ – Brandon Feb 6 at 14:36
  • $\begingroup$ @Brandon If you have a covariance matrix, you can diagonalize it. Whether or not that is useful is another story. $\endgroup$ – Dave Feb 6 at 14:37
  • $\begingroup$ I'd ask the opposite question: is PCA ever recommended? :) $\endgroup$ – gented Feb 7 at 13:28
25
$\begingroup$

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize this intuitive idea with the No Free Lunch theorem.)

It's easy enough to construct an example where the eigenvectors to the smallest eigenvalues are the most informative. If you discard this data, you're discarding the most helpful information for your classification or regression problem, and your model would be improved if you had retained them.

More concretely, suppose $A$ is our design matrix, and each column is mean-centered. Then we can use SVD to compute the PCA of $A$. (see: Relationship between SVD and PCA. How to use SVD to perform PCA?)

For an example in the case of a linear model, this gives us a factorization $$ AV = US $$

and we wish to predict some outcome $y$ as a linear combination of the PCs: $AV\beta = y+\epsilon$ where $\epsilon$ is some noise. Further, let's assume that this linear model is the correct model.

In general, the vector $\beta$ can be anything, just as in an ordinary OLS regression setting; but in any particular problem, it's possible that the only nonzero elements of $\beta$ are the ones corresponding to the smallest positive singular values. Whenever this is the case, using PCA to reduce the dimension of $AV$ by discarding the smallest singular values will also discard the only relevant predictors of $y$. In other words, even though we started out with the correct model, the truncated model is not correct because it omits the key variables.

In other words, PCA has a weakness in a supervised learning scenario because it is not "$y$-aware." Of course, in the cases where PCA is a helpful step, then $\beta$ will have nonzero entries corresponding to the larger singular values.

I think this example is instructive because it shows that even in the special case that the model is linear, truncating $AV$ risks discarding information.

Other common objections include:

  • PCA is a linear model, but the relationships among features may not have the form of a linear factorization. This implies that PCA will be a distortion.

  • PCA can be hard to interpret, because it tends to yield "dense" factorizations, where all features in $A$ have nonzero effect on each PC.

Some more examples can be found in this closely-related thread (thanks, @gung!): Examples of PCA where PCs with low variance are "useful"

$\endgroup$
  • 5
    $\begingroup$ An example, or formal proof, would improve this answer a lot. $\endgroup$ – Tim Feb 6 at 14:46
  • $\begingroup$ That makes sense and that's why I was thinking of. Would other techniques that take the class into account, such as LDA, work better in that case? $\endgroup$ – Brandon Feb 6 at 14:52
  • 3
    $\begingroup$ @SycoraxsaysReinstateMonica: but I suspect the answer is the same for all "Should we always use $MODEL?" questions... $\endgroup$ – cbeleites unhappy with SX Feb 6 at 15:54
  • 1
    $\begingroup$ @cbeleitessupportsMonica Sure, but also that's another example of a different question. We have lots of threads about the No Free Lunch theorem and even more threads aimed at debunking various examples of "First Year Students' Dreams." I wouldn't expect to find an answer to those questions in the comments of a thread about PCA. $\endgroup$ – Sycorax says Reinstate Monica Feb 6 at 16:01
  • 1
    $\begingroup$ @Tim I'll be able to write a lengthier answer once I'm done at the office. $\endgroup$ – Sycorax says Reinstate Monica Feb 6 at 16:19
14
$\begingroup$

First of all, blindly throwing a model on some data cannot be possibly recommended (you may be able to relax that no-no if you have an infinite amount of independent cases at hand...).

There is a formulation of the no-free lunch theorem that is related to the question: it states that over all possible data sets, no model is better than any other. The usual conclusion from that is that models are superior, iff they are better suited for the particular task at hand (including both what the purpose of the analysis is and particular characteristics of the data).

So, the more sensible question you should ask youself is whether your data has characteristics that make it suitable for PCA.


For example, I work mostly with spectroscopic data. This kind of data has properties that align very well with bilinear models such as PCA or PLS, and much less well with a feature selection picking particular measurement channels (wavelengths, features). In particular, I know for physical and chemical reasons that the information I'm seeking is usually spread out quite "thin" over large regions of the spectrum. Because of that, I routinely use PCA as exploratory tool, e.g. to check whether there is large variance that is not correlated with the outcome I want to predict/study. And possibly even to have a look whether I can find out what the source of such variance is and then decide how to deal with that. I then decide whether to use PCA as feature reduction - whereas I know from the beginning that feature selection picking particular wavelength is hardly ever appropriate.

Contrast that, say, with gene microarray data where I know beforehand that the information is probably concentrated in a few genes with all other genes carrying noise only. Here, feature selection is needed.


we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Of course, and in my field (chemometrics) for regression this observation is the textbook trigger to move on from Principal Component Regression to Partial Least Squares Regression.

$\endgroup$
4
$\begingroup$

Of course not, I don't recall reading/hearing any scientific method's name with the word always, let alone PCA. And, there are many other methods that can be used for dimensionality reduction, e.g. ICA, LDA, variuous feature selection methods, matrix/tensor factorization techniques, autoencoders ...

$\endgroup$
  • $\begingroup$ Same here. I was just wondering if it can always be applied without losing valuable information in the dimensionality reduction process $\endgroup$ – Brandon Feb 6 at 14:37
  • 4
    $\begingroup$ Careful with tSNE as dimensionality reduction, particularly if you plan to use it downstream for further tasks beyond visualization. There's no global mapping from your input features to tSNE space, so given only the tSNE output, there's no way to go back to the original features. It's great for visualization, but I don't recommending using tSNE-reduced data for classification/regression. $\endgroup$ – Nuclear Wang Feb 7 at 14:36
  • $\begingroup$ It seems I couldn’t stop while giving examples $\endgroup$ – gunes Feb 7 at 14:38
3
$\begingroup$

The two major limitations of PCA:

1) It assumes linear relationship between variables.

2) The components are much harder to interpret than the original data.

If the limitations outweigh the benefit, one should not use it; hence, pca should not always be used. IMO, it is better to not use PCA, unless there is a good reason to.

$\endgroup$
  • $\begingroup$ You can have a linear relationship between variables and still not have a very meaningful compression by maximizing variance retained. $\endgroup$ – Frans Rodenburg Feb 9 at 3:49
  • 1
    $\begingroup$ Unlike a probabilistic model, PCA is an algebraic procedure that does not assume anything about the underlying variables, except perhaps for technical conditions to ensure that PCA is feasible. As @amoeba says in another thread, PCA, as a data transformation, dimensionality reduction, exploration, and visualization tool, does not make any assumptions. You can run it on any data whatsoever. The components are just linear combinations, and that is not too difficult to interpret, comparing to basically any other form of combinations. $\endgroup$ – Richard Hardy Feb 9 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.