# How to know when to stop reducing dimensions with PCA?

I'm using PCA to reduce dimensionality before I feed the data into a classifier. My bootstrap/cross-validation has shown a significant reduction in test error as a result of applying PCA and keeping the PCs whose standard deviation is a fraction (say, 0.05) of the standard deviation of the first PC. My features are actually histograms (i.e. vector-valued), so instead of applying PCA once globally to the whole dataset, I applied it locally to some features, which I preselected manually based on the number of features (picking the ones with the most columns). I've tried adjusting the aforementioned tolerance, and tried applying PCA to higher and lower numbers of these histogram features.

My question is, can someone please describe a more precise way of finding the optimal amount of dimensionality reduction via PCA as applied above which leads to the highest test accuracy of my classifier? Does it come down to running a loop with a sequence of tolerances and different PCA-treated features and computing the test error for each setting? This would be very computationally expensive.

Note that ridge penalisation/regularisation is basically doing model selection using pca. Although it does it smoothly by shrinking along each principal component axis rather than discretely by dropping small.variance pcs. Note that because you are doing pca on different subsets of variables this would roughly correspond to having different regularising parameters for each group rather than one for all the betas. The elements of statistical learning explains ridging quite well and provides some comparisons.

• Thanks for that. Would lasso relate to PCA even more strongly? – marcos Mar 3 '12 at 17:11
• Lasso is unrelated to pca, as it is not a rotationally invariant algorithm. Ridge regression is rotationally invariate as the penalty depends on the length of the beta vector (which is unchanged under rotations). However if you use the lasso with squared error loss, and the predictors are uncorrelated, then lasso will drop the variables with smallest correlation and shrink the coefficients of the non-zero variables (compared to least squares). – probabilityislogic Mar 4 '12 at 1:18

I know of a few common ways to select PCs.

With regards to feeding it to downstream analysis it is commonly done with a cut-off value $\epsilon$ like so.

So you have $n$ eigenvalues from the decomposition sorted decreasingly. One can construct a pareto curve where each successive entry i is given by the normalized cumulative sum, e.g. $P(i) = (\sum \lambda_1,..,\lambda_i)/(\sum \lambda_1,...,\lambda_n$). Each entry tells you the fraction of the variance explained by considering up to the $ith$ eigenvector. You pick some value $\epsilon \in (0,1)$ to cut that curve and use all the eigenvectors that are required to explain say, 99% of the variance. Common values are 90%, 95%, 99% and 99.9%. Intuitively this is just a orthogonal component denoising step. This is a rather small loop to try out values on. It requires only one eigendecomposition and 4 runs of your downstream analysis. Furthermore this method naturally extends to kernel PCA methods.

I know of another way to select PCs, that to my knowledge isn't popular for feature extraction, but is popular for other analysis. Your pareto curve is monotonically increasing and contains a elbow defined by $max(P(i) - P(i-1))$. The idea being that everything before the elbow is the signal and everything after is the noise. You can try it, but it always gave me bad results for downstream analysis.

If you want to consider your labels you can get pretty fancy with factor analysis.

There's also the option of testing the dependency of your PC's against your labels, where you keep the ones that are above a threshold for shuffled data. I'm not a big fan of feature selection in this context though because it ignores interaction between features, as in the case of say a XOR gate.

Also it sounds like the decomposition was a fruitful effort, you might pursue this further actually using the (weighted?) labels to find the decomposition, i.e. studying cross-covariance matrix decompositions upstream to your classifier.

• One potential issue with apply normal PCA to variable selection is that you implicitly assume that the low variance PCs are poor predictors of the response. There is no intrinsic reason why this should be the case, apart from the fact that low variance PCs are more prone to extrapolation issues (predicting outside the space of the sample) compared to high variance PCs. – probabilityislogic Mar 4 '12 at 4:20
• Right you're very correct. It is highly dependent on that assumption. Discarding any of the nonsingular eigenvectors requires making assumptions about the manifold that the data lies in. If there isn't a extreme class imbalance then it is reasonable to assume that a low-dimensional manifold will describe the relevant signal. – Jessica Mick Mar 4 '12 at 23:23

This is an open problem in psychology/psychometrics. There are a number of methods commonly used. Two that I have found useful are parallel analysis and the minimum avaerage partial criterion. Both of these are implemented in the psych package for R.

Typically, the MAP criterion suggests less factors (say $l$) than does parallel analysis $m$. In that case, I typically use all the solutions between $l$ and $m$ factors. If you were willing to carry out a cross validation process, you could also investigate structural equation modelling, which is implemented in the packages lavaan, sem and OpenMx.

I don't think it is possible to find the "optimal" amount of dimensionality reduction without testing for accuracy at every level. The complexity will increase as more and more dimensions are included which not neccessarly means that the classifier will perform better.

But without prior knowledge of your data, you could try to look at the eigenvalues of the covariance matrix and include, let's say, 95% of your data by taking the dimensions there describes your data with 95% accuracy.

• Should also point out that bias is present and potentially severe when too few dimensions are retained: e.g. trying to represent $n$ observations in three dimensions with $n$ observations in a single dimension. – Alexis Apr 24 '14 at 14:42