Is dimensionality reduction almost always useful for classification?

Question

Is singular value decomposition almost always useful in practice for enhancing the predicative power of a trained classification model?

E.x. A dataset for classification has 20,000 features. Run SVD to convert them to top principal components and transform them to 300 features, and trained a classification model. When predict the class of a test instance, convert it to a 300d principal component feature vector, and use the trained model to predict its class.

Are there some notable real datasets of numerous features (variables) in which dimension reduction by SVD would hurt the predictive power of trained classification models?

Answer 1

I think there are two ways to look at the question whether SVD/PCA helps in general.

Is it better to use PCA reduced data instead of the raw data?

Often yes, but there are situations where PCA is not needed.

I'd in addition consider how well the bilinear concept behind PCA fits with the data generation process. I work with linear spectroscopy, which is governed by physical laws that mean that my observed spectra $\mathbf X$ are linear combinations of the spectra $\mathbf S$ of the chemical species I have, weighted by their respective concentrations $c$: $\mathbf X = \mathbf C \mathbf S$.This fits very well with the PCA model of scores $\mathbf T$ and loadings $\mathbf P$: $\mathbf X = \mathbf T \mathbf P$
I don't know of any example where PCA has hurt a model (except gross errors in setting up a combined PCA-whaterver model)

Even if the underlying relationship in your data doesn't suit that well to the bilinear approach of PCA, PCA in the first place is only a rotation of your data which would usually not hurt. Discarding higher PCs leads to the dimension reduction, but due to set up of the PCA, they carry only small amounts of variance - so again, chances are that even if it is not all that suitable, it won't hurt that much, neither.

This is also part of the bias-variance trade-off in the context of PCA as regularization technique (see @usεr11852's anwer).

Is it better to use PCA instead of some other dimension reduction technique?

The answer on this will be application specific. But if your application suggests some other way of feature generation, these features may be far more powerful than some PCs, so this is worth considering.

Again, my data and applications happen to be of a nature where PCA is a rather natural fit, so I use it and I cannot contribute a counter-example.

But: having a PCA hammer does not imply that all problems are nails... Looking for counterexamples, I'd start maybe image analyses where objects in question can appear anywhere in the picture. The people I know who deal with such tasks usually develop specialized features.

The only task I routinely have that comes close to this is detecting cosmic ray spikes in my camera signals (sharp peaks somewhere caused by cosmic rays hitting the CCD). I also use specialized filters to detect them, although they are easy to find after PCA as well. However, we describe that rather as PCA not being robust against spikes and find it disturbing.

Answer 2

Your intuition is correct. Performing a singular value decomposition in order to use the derived scores in a classifier has a positive influence in a classifier's overall performance in most cases. That is because through SVD one will effectively regularise and/or filter out modes of irrelevant variation (aka. noise). Nevertheless there is no theoretical guarantee that a mode of variation that almost perfectly classifies the data you examine, is not excluded if you decide to choose a particular number of $k$ eigenvectors.

For your example in particular taking around 300 modes of orthogonal variation is most probably enough. Note though that especially if you work in situations where the number of features $p$ is massively larger than the number of available samples $n$ taking an arbitrary number $k$ might give you a false sense of security. There are particular methods (eg. LASSO, SCAD, etc.) that deal with data in that regime. As @ttnphns mentioned @amoeba's answer on How can top k principal components retain the predictive power on a dependent variable? is very good. Do not be driven away by the fact that it focuses on regression. Regression is ultimately an infinite dimensional classification (or in bins of machine precision width if you like :) ).

As for real datasets: I haven't not seen any but I suspect that semantic corpora (Latent Semantic Analysis $\approx$ SVD for texts) might exhibit this behaviour. There might be a relatively invariant small word that in TF-IDF terms scores pretty low - like the word US. Such a term could be probably filtered out in favour of other more dominant terms.

Finally, as mentioned if you want to draw some further conclusions from your classifier using principal modes of variations scores might be detrimental to your model's interpretability. You will be dealing with axes that correspond to normalized linear combinations of your original variables. It might not be trivial to associate them with something tangible in your original sample space.