33
$\begingroup$

I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no.

If we perform PCA then we calculate linear combinations of the features to build principal components that explain most of the variance of the dataset. However, we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Am I correct?. Should we always reduce dimensions with PCA if needed or there are considerations that need to be taken (as the one above)?

$\endgroup$
7
  • 14
    $\begingroup$ Where exactly did you hear that you should always apply PCA? I don't recall anyone doing it even "commonly", not to say "always". $\endgroup$
    – Tim
    Feb 6, 2020 at 14:28
  • 3
    $\begingroup$ As an anecdote, in my line of work we're very limited in the amount of data we can collect due to practical limitations. Reducing dimensions after the fact doesn't help us. We need some feature selection method to determine what data we should be collecting. Edit: This comment was for an earlier version which asked whether PCA should always be applied. $\endgroup$
    – TPM
    Feb 6, 2020 at 14:34
  • $\begingroup$ @Tim As I said in the original post, I was wondering if it can be applied. Haven't heard it anywhere. $\endgroup$
    – Brandon
    Feb 6, 2020 at 14:36
  • 1
    $\begingroup$ I'd ask the opposite question: is PCA ever recommended? :) $\endgroup$
    – gented
    Feb 7, 2020 at 13:28
  • 1
    $\begingroup$ Suppose you're doing classification of images. Then if you do PCA, you are throwing away information about which pixels are near each other. This will be awful. $\endgroup$
    – CrabMan
    Feb 7, 2020 at 17:31

4 Answers 4

39
$\begingroup$

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize this intuitive idea with the No Free Lunch theorem.)

It's easy enough to construct an example where the eigenvectors to the smallest eigenvalues are the most informative. If you discard this data, you're discarding the most helpful information for your classification or regression problem, and your model would be improved if you had retained them.

More concretely, suppose $A$ is our $n \times p$ design matrix with $n$ observations of $p$ features, and each column is mean-centered. Then we can use SVD to compute the PCA of $A$. (see: Relationship between SVD and PCA. How to use SVD to perform PCA?)

For an example in the case of a linear model, this gives us a factorization $$ AV = US $$

and we wish to predict some outcome $y$ as a linear combination of the PCs: $AV\beta = y+\epsilon$ where $\epsilon$ is some noise. Further, let's assume that this linear model is the correct model.

In general, the estimated vector $\hat \beta$ can be anything. In the PCA setting where only the top $k$ components are kept, you are implicitly fixing the $\hat \beta$ coefficients of the $p-k$ discarded components to 0. In other words, even though we started out with the correct model, the truncated model is not correct because it omits the key variables.

In other words, PCA has a weakness in a supervised learning scenario because it is not "$y$-aware." Of course, in the cases where PCA is a helpful step, then $\beta$ will have nonzero entries corresponding to the larger singular values.

I think this example is instructive because it shows that even in the special case that the model is linear, truncating $AV$ risks discarding information.

You can even generate data where the discarded components are essential. Create 2 independent features, one that's completely random, and one that perfectly predicts the outcome, but has a smaller variance. Using PCA & keeping $k=1$ components will fail. Moreover, the smaller the variance of the informative feature, the more pronounced this effect will be.

Class separation can occur in the axis with the least variation.

This illustration comes from this answer https://stats.stackexchange.com/a/80450/22311 with my thanks to Flounderer.

This class implements a simple demonstration. It randomly generates data according to my scheme, and then applies PCA, retaining the desired number of features. Then it tunes an SVM classifier and reports the AUC.

class PcaSvm(object):
    def __init__(self, seed):
        self.seed = seed
        self.rng = np.random.default_rng(seed)

    def __call__(self, a, k, sample_size=1000):
        # x1 is uninformative & has standard deviation = 1
        x1 = self.rng.standard_normal(sample_size).reshape((-1, 1))
        # x1 is very informative & has standard deviation = a
        x2 = a * self.rng.standard_normal(sample_size).reshape((-1, 1))
        # y strongly depends on x2; some samples will be perfectly separable or nearly so
        y = self.rng.binomial(n=1, p=expit(1e6 * np.sign(x2))).reshape(-1)
        svc_params = {
            "svc__C": stats.loguniform(1e0, 1e3),
            "svc__gamma": stats.loguniform(1e-4, 1e-2),
        }
        clf = sklearn.pipeline.make_pipeline(
            PCA(n_components=k), StandardScaler(), SVC()
        )
        random_search = RandomizedSearchCV(
            clf,
            param_distributions=svc_params,
            n_iter=60,
            scoring="roc_auc",
            random_state=self.seed,
        )
        random_search.fit(np.hstack([x1, x2]), y)
        best_test_auc = random_search.cv_results_["mean_test_score"].max()
        print(
            f"Using a={a}, the best model with k={k} PCA components has an average AUC (on the test set) of {best_test_auc:.4f}"
        )

When PCA only retains 1 feature, the model is somewhere between worthless and mediocre. When retaining 2 features, the model is literally perfect.

standard deviation of informative feature number of components retained AUC
0.001 1 0.5024
0.1 1 0.5075
0.9 1 0.5197
1.0 1 0.7277
0.001 2 1.0
0.1 2 1.0
0.9 2 1.0
1.0 2 1.0

Other common objections to "always" using PCA include:

  • PCA is a linear model, but the relationships among features may not have the form of a linear factorization. This implies that PCA will be a distortion.

  • PCA can be hard to interpret, because it tends to yield "dense" factorizations, where all features in $A$ have nonzero effect on each PC.

We also have a few related threads (thanks, @gung!):

$\endgroup$
5
  • $\begingroup$ That makes sense and that's why I was thinking of. Would other techniques that take the class into account, such as LDA, work better in that case? $\endgroup$
    – Brandon
    Feb 6, 2020 at 14:52
  • $\begingroup$ @Brandon Seems like "Should we always use LDA?" is a different question. $\endgroup$
    – Sycorax
    Feb 6, 2020 at 15:52
  • 3
    $\begingroup$ @SycoraxsaysReinstateMonica: but I suspect the answer is the same for all "Should we always use $MODEL?" questions... $\endgroup$ Feb 6, 2020 at 15:54
  • 1
    $\begingroup$ @cbeleitessupportsMonica Sure, but also that's another example of a different question. We have lots of threads about the No Free Lunch theorem and even more threads aimed at debunking various examples of "First Year Students' Dreams." I wouldn't expect to find an answer to those questions in the comments of a thread about PCA. $\endgroup$
    – Sycorax
    Feb 6, 2020 at 16:01
  • $\begingroup$ @Tim there is a proof of exactly this situation in a paper I published with my supervisor (pdf) in 2014. We showed that in the case of change detection, if you assume all features are equally likely to be subject to change (which is true in certain situations like EEG seizure monitoring), then the smallest eigenvalue features are better to retain. $\endgroup$ Feb 7, 2020 at 7:51
17
$\begingroup$

First of all, blindly throwing a model on some data cannot be possibly recommended (you may be able to relax that no-no if you have an infinite amount of independent cases at hand...).

There is a formulation of the no-free lunch theorem that is related to the question: it states that over all possible data sets, no model is better than any other. The usual conclusion from that is that models are superior, iff they are better suited for the particular task at hand (including both what the purpose of the analysis is and particular characteristics of the data).

So, the more sensible question you should ask youself is whether your data has characteristics that make it suitable for PCA.


For example, I work mostly with spectroscopic data. This kind of data has properties that align very well with bilinear models such as PCA or PLS, and much less well with a feature selection picking particular measurement channels (wavelengths, features). In particular, I know for physical and chemical reasons that the information I'm seeking is usually spread out quite "thin" over large regions of the spectrum. Because of that, I routinely use PCA as exploratory tool, e.g. to check whether there is large variance that is not correlated with the outcome I want to predict/study. And possibly even to have a look whether I can find out what the source of such variance is and then decide how to deal with that. I then decide whether to use PCA as feature reduction - whereas I know from the beginning that feature selection picking particular wavelength is hardly ever appropriate.

Contrast that, say, with gene microarray data where I know beforehand that the information is probably concentrated in a few genes with all other genes carrying noise only. Here, feature selection is needed.


we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Of course, and in my field (chemometrics) for regression this observation is the textbook trigger to move on from Principal Component Regression to Partial Least Squares Regression.

$\endgroup$
4
$\begingroup$

Of course not, I don't recall reading/hearing any scientific method's name with the word always, let alone PCA. And, there are many other methods that can be used for dimensionality reduction, e.g. ICA, LDA, variuous feature selection methods, matrix/tensor factorization techniques, autoencoders ...

$\endgroup$
3
  • $\begingroup$ Same here. I was just wondering if it can always be applied without losing valuable information in the dimensionality reduction process $\endgroup$
    – Brandon
    Feb 6, 2020 at 14:37
  • 5
    $\begingroup$ Careful with tSNE as dimensionality reduction, particularly if you plan to use it downstream for further tasks beyond visualization. There's no global mapping from your input features to tSNE space, so given only the tSNE output, there's no way to go back to the original features. It's great for visualization, but I don't recommending using tSNE-reduced data for classification/regression. $\endgroup$ Feb 7, 2020 at 14:36
  • $\begingroup$ It seems I couldn’t stop while giving examples $\endgroup$
    – gunes
    Feb 7, 2020 at 14:38
3
$\begingroup$

The two major limitations of PCA:

1) It assumes linear relationship between variables.

2) The components are much harder to interpret than the original data.

If the limitations outweigh the benefit, one should not use it; hence, pca should not always be used. IMO, it is better to not use PCA, unless there is a good reason to.

$\endgroup$
2
  • $\begingroup$ You can have a linear relationship between variables and still not have a very meaningful compression by maximizing variance retained. $\endgroup$ Feb 9, 2020 at 3:49
  • 2
    $\begingroup$ Unlike a probabilistic model, PCA is an algebraic procedure that does not assume anything about the underlying variables, except perhaps for technical conditions to ensure that PCA is feasible. As @amoeba says in another thread, PCA, as a data transformation, dimensionality reduction, exploration, and visualization tool, does not make any assumptions. You can run it on any data whatsoever. The components are just linear combinations, and that is not too difficult to interpret, comparing to basically any other form of combinations. $\endgroup$ Feb 9, 2020 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.