A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At present, I am trying to using bi-gram language model or (N-gram) for building feature vector, but do not quite know how to do that? Can we just follow the approach of bag-of-words, i.e., computing the frequency count in terms of bi-gram instead of words, and enhancing it using tf-idf weighting scheme?


3 Answers 3


Yes. That will generate many more features though: it might be important to apply some cut-off (for instance discard features such bi-grams or words that occur less than 5 times in your dataset) so as to not drown your classifier with too many noisy features.

  • $\begingroup$ Thanks. Do you mean that my general idea of computing each feature value in terms of bigram (N-gram) is correct? In other words, there is no big difference in computing the feature values between bag-of-words and N-gram model. Thanks for the clarification. $\endgroup$
    – user3125
    Commented Apr 2, 2012 at 14:44
  • $\begingroup$ Yes, you can use both all bigrams + unigrams (words) in a big bag of features (as long as you trim the least frequent with some cut-off level). $\endgroup$
    – ogrisel
    Commented Apr 2, 2012 at 17:05

The number of bigrams can be reduced by selecting only those with positive mutual information.

We did this for generating a bag of bigrams representation at the INEX XML Mining track, http://www.inex.otago.ac.nz/tracks/wiki-mine/wiki-mine.asp.

What we did not try is using the mutual information between the terms in weighting the bi-grams. See https://en.wikipedia.org/wiki/Pointwise_mutual_information , https://www.eecis.udel.edu/~trnka/CISC889-11S/lectures/philip-pmi.pdf and http://www.nltk.org/howto/collocations.html for a better explanation of pointwise mutual information for bigrams.

See https://stackoverflow.com/questions/20018730/computing-pointwise-mutual-information-of-a-text-document-using-python and https://stackoverflow.com/questions/22118350/python-sentiment-analysis-using-pointwise-mutual-information for other questions related to this.

  • $\begingroup$ dead link :-( ... $\endgroup$
    – Renaud
    Commented Nov 25, 2016 at 21:00
  • 1
    $\begingroup$ @Renaud Links have been updated :-) $\endgroup$ Commented Nov 29, 2016 at 1:53

Using random projections to reduce the dimensionality of the data may prove useful to reduce the the space required to store the features, https://en.wikipedia.org/wiki/Random_projection. It scales very well and every example can be projected to a lower dimensional space independently and without any direct optimization methods such as PCA, SVD, Sammon Maps, NMF, etc.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.