What can cause PCA to worsen results of a classifier?

Question

I have a classifier that I'm doing cross-validation on, along with a hundred or so features that I'm doing forward selection on to find optimal combinations of features. I also compare this against running the same experiments with PCA, where I take the potential features, apply SVD, transform the original signals onto the new coordinate space, and use the top $k$ features in my forward selection process.

My intuition was that PCA would improve the results, as the signals would be more "informative" than the original features. Is my naive understanding of PCA leading me into trouble? Can anyone suggest some of the common reasons why PCA may improve results in some situations, but worsen them in others?

Answer 1

Consider a simple case, lifted from a terrific and undervalued article "A Note on the Use of Principal Components in Regression ".

Suppose you only have two (scaled and de-meaned) features, denote them $x_1$ and $x_2$ with positive correlation equal to 0.5, aligned in $X$, and a third response variable $Y$ you wish to classify. Suppose that the classification of $Y$ is fully determined by the sign of $x_1 - x_2$.

Performing PCA on $X$ results in the new (ordered by variance) features $[x_1 + x_2, x_1 - x_2]$, since $\operatorname{Var}( x_1 + x_2 ) = 1 + 1 + 2\rho > \operatorname{Var}(x_1 - x_2 ) = 2 - 2\rho$. Therefore, if you reduce your dimension to 1 i.e. the first principal component, you are throwing away the exact solution to your classification!

The problem occurs because PCA is agnostic to $Y$. Unfortunately, one cannot include $Y$ in the PCA either as this will result in data leakage.

---

Data leakage is when your matrix $X$ is constructed using the target predictors in question, hence any predictions out-of-sample will be impossible.

For example: in financial time series, trying to predict the European end-of-day close, which occurs at 11:00am EST, using American end-of-day closes, at 4:00pm EST, is data leakage since the American closes, which occur hours later, have incorporated the prices of European closes.

Answer 2

There is a simple geometric explanation. Try the following example in R and recall that the first principal component maximizes variance.

library(ggplot2)

n <- 400
z <- matrix(rnorm(n * 2), nrow = n, ncol = 2)
y <- sample(c(-1,1), size = n, replace = TRUE)

# PCA helps
df.good <- data.frame(
    y = as.factor(y), 
    x = z + tcrossprod(y, c(10, 0))
)
qplot(x.1, x.2, data = df.good, color = y) + coord_equal()

# PCA hurts
df.bad <- data.frame(
    y = as.factor(y), 
    x = z %*% diag(c(10, 1), 2, 2) + tcrossprod(y, c(0, 8))
)
qplot(x.1, x.2, data = df.bad, color = y) + coord_equal()

PCA Helps

The direction of maximal variance is horizontal, and the classes are separated horizontally.

PCA Hurts

The direction of maximal variance is horizontal, but the classes are separated vertically

Answer 3

PCA is linear, It hurts when you want to see non linear dependencies.

PCA on images as vectors:

A non linear algorithm (NLDR) wich reduced images to 2 dimensions, rotation and scale:

More informations: http://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction

Answer 4

Suppose a simple case with 3 independent variables $x_1,x_2,x_3$ and the output $y$ and suppose now that $x_3=y$ and so you should be able to get a 0 error model.

Suppose now that in the training set the variation of $y$ is very small and so also the variation of $x_3$.

Now if you run PCA and you decide to select only 2 variables you will obtain a combination of $x_1$ and $x_2$. So the information of $x_3$ that was the only variable able to explain $y$ is lost.

Answer 5

I see the question already has an accepted answer but wanted to share this paper that talks about using PCA for feature transformation before classification.

The take-home message (which is visualised beautifully in @vqv's answer) is:

Principal Component Analysis (PCA) is based on extracting the axes on which data shows the highest variability. Although PCA “spreads out” data in the new basis, and can be of great help in unsupervised learning, there is no guarantee that the new axes are consistent with the discriminatory features in a (supervised) classification problem.

For those interested, if you look at Section 4. Experimental results, they compare the classification accuracies with 1) the original features, 2) PCA transformed features, and 3) combination of both, which was something that was new to me.

My conclusion:

PCA-based feature transformations allow to summarize the information from a large number of features into a limited number of components, i.e. linear combinations of the original features. However the principal components are often difficult to interpret (not intuitive), and as the empirical results in this paper indicate they usually do not improve the classification performance.

P.S: I note that one of the limitations of the paper that sould have been listed was the fact that the authors limited performance assessment of the classifiers to 'accuracy' only, which can be a very biased performance indicator.