Feature selection methods for document classtification

Question

I have a simple document classification problem where i need to classify some documents to a definite set of classes.

I need to perform a feature selection (where I will select the most important words from each class).

Currently I am calculating the tf-idf scores. Is there a better way to do that ? I heard about chi-square statistic being used in this context, is that true? And if that is the case, could you give me (preferably simple) links to resources that I can read for further information about this?

thanks

Answer 1

Introduction to Information Retrieval book contains some relevant material.

If python is your cup of tea (and if you have a moderate amount of data) then this deck might be helpful. Basically, one can train nltk's naive bayes classifier that, among other things, allows choosing top N most informative features (so one could then restrict the feature set to, say, top 1000 or top 10000 features - ideally this threshold should be tuned on a holdout sample or using cross validation):

>>> help(nltk.classify.NaiveBayesClassifier.most_informative_features) Help on method most_informative_features in module nltk.classify.naivebayes:

most_informative_features(self, n=100) unbound nltk.classify.naivebayes.NaiveBayesClassifier method
    Return a list of the 'most informative' features used by this
    classifier.  For the purpose of this function, the
    informativeness of a feature C{(fname,fval)} is equal to the
    highest value of P(fname=fval|label), for any label, divided by
    the lowest value of P(fname=fval|label), for any label::

      max[ P(fname=fval|label1) / P(fname=fval|label2) ]

In addition to unigram/bag-of-words based features, one could try adding significant bigrams to the feature list (the deck has some examples). nltk provides multiple ways to calculate significance for collocations (including chi-squared)

Another popular approach is to apply tf-idf to all features first (without any feature selection), and use the regularization (L1 and/or L2) to deal with irrelevant features (the SVM example from the deck corresponds to L2 regularization). The drawback is that the regularization coefficient has to be tuned on a holdout data set or using cross validation.