# Multinomial Naive Bayes performance with and without Bigrams/Stopwords

I have a dataset of 300 text documents that I have manually classified into three classes. The first two classes are relatively equal in size; the third class is about 40% larger than each of the individual first two classes.

The classes are very similar, i.e. they share similar features. After pre-processing and removing words that only appear in 2% of the texts, Multinomial Naïve Bayes predicts about 48-58% of the test set (20% of the texts) correctly. I notice that the larger third class is predicted more accurately than the smaller first and second classes.

I would like to refine pre-processing to achieve 65-75% accuracy. The first thing I did was to tokenize the features into bigrams. After doing so, the model only predicts between 40-50%, instead of 48-58% with single words. Intuitively, I expected the model to perform better, not worse, with bigrams. Shouldn't bigram tokenization reduce the amount of shared tokens across the classes and increase the predictive power of the non-shared tokens as a result (at least marginally)?

I am a bit concerned because my next refinement is to create a stopword list of words that frequently appear across all classes but have no predictive content. My reasoning was that by reducing the amount of high-frequency shared words, I would increase the predictive power of the remaining words. Isn't this the point of stopword lists?

• Velcome to our site! – kjetil b halvorsen Jun 10 '14 at 13:54

By using bigrams instead of single words, you are also reducing your amount of data, in that $n_{bigrams} < n_{words}$ most of the time. While you are focused on reducing the amount of shared tokens across the classes, bear in mind that you may also be reducing the amount of distinctive tokens per class, which may have a negative effect on predictive power.