Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am familiar with PCA as a linear transformation in order to align the axis of the IV space with the directions of maximal variance in order to possibly be able to reduce the dimensionality of the problem. Are there similar techniques (or can one use PCA itself) when facing a binary classification problem? Like a logistic regression? I.E. Suppose we have a dependent variable we want to predict which only takes values 0 and 1 and we have a lot of features from which to predict, but maybe there are "principal components" in the high dimensional feature space which carry the most explanatory power.

share|improve this question
It is unclear in your question how "dimensionality reduction" task relates to "classification" task, and whether "binary" applies to the nature of variables (features) or the number of classes (i.e. there's 2 classes). Please clarify. – ttnphns Mar 26 '12 at 6:10
Because you have only 2 classes, only 1 predictive "principal component" can exist, and that being a linear combination of the features, the regressional model. You could prefer logistic regression or linear regression. You could then drop features with small coefficients from the model if you want to have a concise model. There is no need for preliminary PCA or PLS at all, for me (unless there are collinearity/singularity problems). – ttnphns Mar 26 '12 at 17:32
You mean in a multi-class problem PCA/PLS could be useful but not so in the case of two classes? If i visualize a 3D axis with y-values of 0 or 1, cant there still be two orthogonal u_1, u_2 vectors spanning a "PCA" like frame? (i.e. say two principal components with one capturing more variance?) – Palace Chan Mar 26 '12 at 17:49
Oh also, what about PLS-DA? Wikipedia says: "Partial least squares Discriminant Analysis (PLS-DA) is a variant used when the Y is binary." – Palace Chan Mar 26 '12 at 18:54
You mean in a multi-class problem PCA/PLS could be useful but not so in the case of two classes? No I don't mean this. I mean preliminary PCA is blind to the existance of classes, so if you retain just few first principal components after it, they are not necessarily the best or even good discriminators between the classes. – ttnphns Mar 27 '12 at 10:30
up vote 1 down vote accepted

Yes, you can use PCA retaining only certain components as pre-processing step for classification. It is actually pretty common in chemometric classification. However, I prefer PLS as it also uses your class labels and tries to find good separation (which PCA doesn't).

In both cases, keep in mind that this is a data-driven pre-processing and validation of the final model needs data independent also of these pre-processing steps.

share|improve this answer
+1. I know this is a very old thread but it was bumped to the front page again. It seems that linear discriminant analysis (LDA) is another natural option to consider in this case. – amoeba Aug 11 '15 at 13:01
Yes, but keep in mind that LDA (at least with the usual algorithms) needs to have reasonably low dimensionality before it can be applied in order to obtain a stable inversion of the covariance matrix. At least in my field, dimensionality reduction is often done in order to become able to do LDA... – cbeleites Aug 11 '15 at 14:40
That's right, but one can use regularized LDA (rLDA), which is pretty similar in spirit to PCA+LDA or PLS+LDA... – amoeba Aug 11 '15 at 16:22

I like Information Gain, but there are others.

I found this paper (Fabrizio Sebastiani, Machine Learning in Automated Text Categorization, ACM Computing Surveys, Vol. 34, No.1, pp.1-47, 2002) to be a good theoretical treatment of text classification, including feature reduction by a variety of methods from the simple (Term Frequency) to the complex (Information-Theoretic).

These functions try to capture the intuition that the best terms for ci are the ones distributed most differently in the sets of positive and negative examples of ci. However, interpretations of this principle vary across different functions. For instance, in the experimental sciences χ2 is used to measure how the results of an observation differ (i.e., are independent) from the results expected according to an initial hypothesis (lower values indicate lower dependence). In DR we measure how independent tk and ci are. The terms tk with the lowest value for χ2(tk, ci) are thus the most independent from ci; since we are interested in the terms which are not, we select the terms for which χ2(tk, ci) is highest.

These techniques help you choose terms that are most useful in separating the training documents into the given classes; the terms with the highest predictive value for your problem.

I've been successful using Information Gain for feature reduction and found this paper (Entropy based feature selection for text categorization Largeron, Christine and Moulin, Christophe and Géry, Mathias - SAC - Pages 924-928 2011) to be a very good practical guide.

Here the authors present a simple formulation of entropy-based feature selection that's useful for implementation in code:

Given a term tj and a category ck, ECCD(tj , ck) can be computed from a contingency table. Let A be the number of documents in the category containing tj ; B, the number of documents in the other categories containing tj ; C, the number of documents of ck which do not contain tj and D, the number of documents in the other categories which do not contain tj (with N = A + B + C + D):

enter image description here

Using this contingency table, Information Gain can be estimated by:

enter image description here

This approach is easy to implement and provides very good Information-Theoretic feature reduction.

You needn't use a single technique either; you can combine them. Ter-Frequency is simple, but can also be effective. I've combined the Information Gain approach with Term Frequency to do feature selection successfully. You should experiment with your data to see which technique or techniques work most effectively.

share|improve this answer
Welcome to the site, @BobDillon. We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. Thus, we're wary of link-only answers, eg due to linkrot. Can you post a full citation & a summary of the information at the link, in case it goes dead? – gung Aug 9 '15 at 17:13
I would have liked to added links to the papers but I do not have enough points yet to have those links as well as images of the equations/tables. – Bob Dillon Aug 11 '15 at 12:52

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.