Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

I am familiar with PCA as a linear transformation in order to align the axis of the IV space with the directions of maximal variance in order to possibly be able to reduce the dimensionality of the problem. Are there similar techniques (or can one use PCA itself) when facing a binary classification problem? Like a logistic regression? I.E. Suppose we have a dependent variable we want to predict which only takes values 0 and 1 and we have a lot of features from which to predict, but maybe there are "principal components" in the high dimensional feature space which carry the most explanatory power.

share|improve this question
It is unclear in your question how "dimensionality reduction" task relates to "classification" task, and whether "binary" applies to the nature of variables (features) or the number of classes (i.e. there's 2 classes). Please clarify. – ttnphns Mar 26 '12 at 6:10
Because you have only 2 classes, only 1 predictive "principal component" can exist, and that being a linear combination of the features, the regressional model. You could prefer logistic regression or linear regression. You could then drop features with small coefficients from the model if you want to have a concise model. There is no need for preliminary PCA or PLS at all, for me (unless there are collinearity/singularity problems). – ttnphns Mar 26 '12 at 17:32
You mean in a multi-class problem PCA/PLS could be useful but not so in the case of two classes? If i visualize a 3D axis with y-values of 0 or 1, cant there still be two orthogonal u_1, u_2 vectors spanning a "PCA" like frame? (i.e. say two principal components with one capturing more variance?) – Palace Chan Mar 26 '12 at 17:49
Oh also, what about PLS-DA? Wikipedia says: "Partial least squares Discriminant Analysis (PLS-DA) is a variant used when the Y is binary." – Palace Chan Mar 26 '12 at 18:54
You mean in a multi-class problem PCA/PLS could be useful but not so in the case of two classes? No I don't mean this. I mean preliminary PCA is blind to the existance of classes, so if you retain just few first principal components after it, they are not necessarily the best or even good discriminators between the classes. – ttnphns Mar 27 '12 at 10:30

1 Answer

up vote 0 down vote accepted

Yes, you can use PCA retaining only certain components as pre-processing step for classification. It is actually pretty common in chemometric classification. However, I prefer PLS as it also uses your class labels and tries to find good separation (which PCA doesn't).

In both cases, keep in mind that this is a data-driven pre-processing and validation of the final model needs data independent also of these pre-processing steps.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.