I am facing a sentiment analysis task where I am using Naive Bayes to classify documents as Positive, Negative or Neutral. I have thought of using Information Gain as my filter for feature selection. Taking into account that I need to classify into 3 classes, would it be ok to use IG without further modifications (since I think IGain works best with 2 classes)?

  • $\begingroup$ Are you sure that it works even for 2 classes? I would expect a heavy overfit. $\endgroup$
    – user88
    May 15, 2012 at 12:06
  • $\begingroup$ It works quite well actually. Why would you expect such a heavy overfit? $\endgroup$ May 15, 2012 at 12:41
  • $\begingroup$ You basically cherry-pick features which can yield trivial and accurate classifiers on a set used for feature selection -- if such can occur at random with a reasonable chance (as in most large-p sets) and the whole problem is not trivial on its own, you are likely in trouble. Yet I don't claim it is certainly your case -- if you have done a proper validation and it works, I won't argue. $\endgroup$
    – user88
    May 15, 2012 at 23:42

1 Answer 1


Information gain is a reasonable objective to use for selecting features (even when there are multiple classes). Note that information gain is a traditional metric for selecting decision attributes for building decision trees. Note that a classic problem with decision tress is when to stop adding decision nodes---too many nodes usually leads to poor generalization. IG will help you determine an ordering of features from most useful to least useful. You will need another method (such as evaluation on a hold-out set) to determine a cut-off point.

You may be interested in reading A Comparative Study on Feature Selection in Text Categorization (1997), which evaluates IG against other methods.

Note that your problem sounds more like ordinal regression (which encodes an ordering in the labels) than regular classification.

  • $\begingroup$ Thanks for the reply. I basically get the information gain score for every feature, and then I get the top K ones. Such K is tested empirically (I select the K at which F1 measure of all classes is max) ... Is this a good way to go? $\endgroup$ May 15, 2012 at 12:41
  • $\begingroup$ Yes, that approach is reasonable and common. See reference I just added to my answer. More sophisticated (but less practical) would be something like Toward Optimal Feature Selection. Ideally, feature selection is integrated with the classifier, such as Lasso $\endgroup$
    – jrennie
    May 15, 2012 at 14:28
  • $\begingroup$ Thanks again! Regarding integrating feature selection inside the classifier (you mean wrappers?), my understanding is that filter(IG, Chisq...) methods are the simplest to implement and the most scalable, thus they are appropriate to treat very large feature spaces like we do in text classification...However, sometimes using pre-processing step with a filter to reduce dimensionality of the feature space, and then a wrapper yields good results. Anyway, I guess I will stick to filters by now... $\endgroup$ May 15, 2012 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.