Is PCA always recommended?

Question

I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no.

If we perform PCA then we calculate linear combinations of the features to build principal components that explain most of the variance of the dataset. However, we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Am I correct?. Should we always reduce dimensions with PCA if needed or there are considerations that need to be taken (as the one above)?

Answer 1

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize this intuitive idea with the No Free Lunch theorem.)

It's easy enough to construct an example where the eigenvectors to the smallest eigenvalues are the most informative. If you discard this data, you're discarding the most helpful information for your classification or regression problem, and your model would be improved if you had retained them.

More concretely, suppose $A$ is our $n \times p$ design matrix with $n$ observations of $p$ features, and each column is mean-centered. Then we can use SVD to compute the PCA of $A$. (see: Relationship between SVD and PCA. How to use SVD to perform PCA?)

For an example in the case of a linear model, this gives us a factorization $$ AV = US $$

and we wish to predict some outcome $y$ as a linear combination of the PCs: $AV\beta = y+\epsilon$ where $\epsilon$ is some noise. Further, let's assume that this linear model is the correct model.

In general, the estimated vector $\hat \beta$ can be anything. In the PCA setting where only the top $k$ components are kept, you are implicitly fixing the $\hat \beta$ coefficients of the $p-k$ discarded components to 0. In other words, even though we started out with the correct model, the truncated model is not correct because it omits the key variables.

In other words, PCA has a weakness in a supervised learning scenario because it is not "$y$-aware." Of course, in the cases where PCA is a helpful step, then $\beta$ will have nonzero entries corresponding to the larger singular values.

I think this example is instructive because it shows that even in the special case that the model is linear, truncating $AV$ risks discarding information.

You can even generate data where the discarded components are essential. Create 2 independent features, one that's completely random, and one that perfectly predicts the outcome, but has a smaller variance. Using PCA & keeping $k=1$ components will fail. Moreover, the smaller the variance of the informative feature, the more pronounced this effect will be.

This illustration comes from this answer https://stats.stackexchange.com/a/80450/22311 with my thanks to Flounderer.

This class implements a simple demonstration. It randomly generates data according to my scheme, and then applies PCA, retaining the desired number of features. Then it tunes an SVM classifier and reports the AUC.

class PcaSvm(object):
    def __init__(self, seed):
        self.seed = seed
        self.rng = np.random.default_rng(seed)

    def __call__(self, a, k, sample_size=1000):
        # x1 is uninformative & has standard deviation = 1
        x1 = self.rng.standard_normal(sample_size).reshape((-1, 1))
        # x1 is very informative & has standard deviation = a
        x2 = a * self.rng.standard_normal(sample_size).reshape((-1, 1))
        # y strongly depends on x2; some samples will be perfectly separable or nearly so
        y = self.rng.binomial(n=1, p=expit(1e6 * np.sign(x2))).reshape(-1)
        svc_params = {
            "svc__C": stats.loguniform(1e0, 1e3),
            "svc__gamma": stats.loguniform(1e-4, 1e-2),
        }
        clf = sklearn.pipeline.make_pipeline(
            PCA(n_components=k), StandardScaler(), SVC()
        )
        random_search = RandomizedSearchCV(
            clf,
            param_distributions=svc_params,
            n_iter=60,
            scoring="roc_auc",
            random_state=self.seed,
        )
        random_search.fit(np.hstack([x1, x2]), y)
        best_test_auc = random_search.cv_results_["mean_test_score"].max()
        print(
            f"Using a={a}, the best model with k={k} PCA components has an average AUC (on the test set) of {best_test_auc:.4f}"
        )

When PCA only retains 1 feature, the model is somewhere between worthless and mediocre. When retaining 2 features, the model is literally perfect.

standard deviation of informative feature	number of components retained	AUC
0.001	1	0.5024
0.1	1	0.5075
0.9	1	0.5197
1.0	1	0.7277
0.001	2	1.0
0.1	2	1.0
0.9	2	1.0
1.0	2	1.0

Other common objections to "always" using PCA include:

• PCA is a linear model, but the relationships among features may not have the form of a linear factorization. This implies that PCA will be a distortion.

• PCA can be hard to interpret, because it tends to yield "dense" factorizations, where all features in $A$ have nonzero effect on each PC.

We also have a few related threads (thanks, @gung!):

• Low variance components in PCA, are they really just noise? Is there any way to test for it?

• The first principal component does not separate classes, but other PCs do; how is that possible?

• Examples of PCA where PCs with low variance are "useful"

• How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?

Answer 2

First of all, blindly throwing a model on some data cannot be possibly recommended (you may be able to relax that no-no if you have an infinite amount of independent cases at hand...).

There is a formulation of the no-free lunch theorem that is related to the question: it states that over all possible data sets, no model is better than any other. The usual conclusion from that is that models are superior, iff they are better suited for the particular task at hand (including both what the purpose of the analysis is and particular characteristics of the data).

So, the more sensible question you should ask youself is whether your data has characteristics that make it suitable for PCA.

---

For example, I work mostly with spectroscopic data. This kind of data has properties that align very well with bilinear models such as PCA or PLS, and much less well with a feature selection picking particular measurement channels (wavelengths, features). In particular, I know for physical and chemical reasons that the information I'm seeking is usually spread out quite "thin" over large regions of the spectrum. Because of that, I routinely use PCA as exploratory tool, e.g. to check whether there is large variance that is not correlated with the outcome I want to predict/study. And possibly even to have a look whether I can find out what the source of such variance is and then decide how to deal with that. I then decide whether to use PCA as feature reduction - whereas I know from the beginning that feature selection picking particular wavelength is hardly ever appropriate.

Contrast that, say, with gene microarray data where I know beforehand that the information is probably concentrated in a few genes with all other genes carrying noise only. Here, feature selection is needed.

---

we might be leaving out features that do not explain much of the variance of the dataset but do explain what characterizes one class against another.

Of course, and in my field (chemometrics) for regression this observation is the textbook trigger to move on from Principal Component Regression to Partial Least Squares Regression.

Answer 3

Of course not, I don't recall reading/hearing any scientific method's name with the word always, let alone PCA. And, there are many other methods that can be used for dimensionality reduction, e.g. ICA, LDA, variuous feature selection methods, matrix/tensor factorization techniques, autoencoders ...

Answer 4

The two major limitations of PCA:

1) It assumes linear relationship between variables.

2) The components are much harder to interpret than the original data.

If the limitations outweigh the benefit, one should not use it; hence, pca should not always be used. IMO, it is better to not use PCA, unless there is a good reason to.