Ok so I am building a text classifier with approx. 20000 entries and 20 categories. I have heard several times that using a bigram representation of my input instead of the classic "Bag of Words" representation could potentially boost my classifier's accuracy.

Unfortunately it was not the case for me, so now I wonder : When is it a good idea to make that change? Any insight on what kind of classification tasks actually benefits from this?



Bag of word, whether used with unigrams or bigrams, breaks the structure of the text. Features whose sole existence is informative regarding the concept are valuable in this context.

If you use bigrams, then the feature "White house" is a good indication to politics. When you use unigrams then both "white" and "house" are less informative. Note that you lose the context when using bag of word. With respect to "white" and "house" you will get a similar representation for:

  • The white house announced a new welfare policy.
  • The white car parked near the house.

On the other hand, the more token you aggregate, the higher the number on ngrams you will have. The number of occurrences of each ngram will be lower so your computation will become more sensitive to noise.

Luckily, you don't have to choose either unigrams or bigrams. A straight forward way is to use both. That will lead to many features, which is a problem on its own. A better approaches is to use only the ngrams that are not accidental. You can identify ngrams whose occurrences is significantly higher than the expected if they were indented, lead to more mutual information or whatever measure that fits you needs. That will remove most bigrams, keep bigrams like "white house" and usually a good source for insight in manual observation.

  • $\begingroup$ Ok thank you for your answer, I see more clearly the situation now. So what you suggest is that I feed to my classifier features composed both of unigrams and bigrams (but still in a bag of word format) and delete thos with the lowest frequency of appearances? How do you suggest I deal with the "more ngrams=more noise" problem? Thanks again! $\endgroup$ – LoulouChameau Aug 25 '16 at 7:40
  • $\begingroup$ Bag of words tend to generate many features even if you use unigrams. From the linguistic side, removing stop words is usually beneficial en.wikipedia.org/wiki/Stop_words. From a general ML perspective, doing feature selection is also helpful. $\endgroup$ – DaL Aug 25 '16 at 7:50
  • $\begingroup$ Yes indeed I agree that removing stop words boosted my classifier performance. I will look deeper into features selection, thanks for your help. $\endgroup$ – LoulouChameau Aug 25 '16 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.