I have a simple document classification problem where i need to classify some documents to a definite set of classes.

I need to perform a feature selection (where I will select the most important words from each class).

Currently I am calculating the tf-idf scores. Is there a better way to do that ? I heard about chi-square statistic being used in this context, is that true? And if that is the case, could you give me (preferably simple) links to resources that I can read for further information about this?



1 Answer 1


Introduction to Information Retrieval book contains some relevant material.

If python is your cup of tea (and if you have a moderate amount of data) then this deck might be helpful. Basically, one can train nltk's naive bayes classifier that, among other things, allows choosing top N most informative features (so one could then restrict the feature set to, say, top 1000 or top 10000 features - ideally this threshold should be tuned on a holdout sample or using cross validation):

>>> help(nltk.classify.NaiveBayesClassifier.most_informative_features) Help on method most_informative_features in module nltk.classify.naivebayes:

most_informative_features(self, n=100) unbound nltk.classify.naivebayes.NaiveBayesClassifier method
    Return a list of the 'most informative' features used by this
    classifier.  For the purpose of this function, the
    informativeness of a feature C{(fname,fval)} is equal to the
    highest value of P(fname=fval|label), for any label, divided by
    the lowest value of P(fname=fval|label), for any label::

      max[ P(fname=fval|label1) / P(fname=fval|label2) ]

In addition to unigram/bag-of-words based features, one could try adding significant bigrams to the feature list (the deck has some examples). nltk provides multiple ways to calculate significance for collocations (including chi-squared)

Another popular approach is to apply tf-idf to all features first (without any feature selection), and use the regularization (L1 and/or L2) to deal with irrelevant features (the SVM example from the deck corresponds to L2 regularization). The drawback is that the regularization coefficient has to be tuned on a holdout data set or using cross validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.