I have a dataset of 300 text documents that I have manually classified into three classes. The first two classes are relatively equal in size; the third class is about 40% larger than each of the individual first two classes.
The classes are very similar, i.e. they share similar features. After pre-processing and removing words that only appear in 2% of the texts, Multinomial Naïve Bayes predicts about 48-58% of the test set (20% of the texts) correctly. I notice that the larger third class is predicted more accurately than the smaller first and second classes.
I would like to refine pre-processing to achieve 65-75% accuracy. The first thing I did was to tokenize the features into bigrams. After doing so, the model only predicts between 40-50%, instead of 48-58% with single words. Intuitively, I expected the model to perform better, not worse, with bigrams. Shouldn't bigram tokenization reduce the amount of shared tokens across the classes and increase the predictive power of the non-shared tokens as a result (at least marginally)?
I am a bit concerned because my next refinement is to create a stopword list of words that frequently appear across all classes but have no predictive content. My reasoning was that by reducing the amount of high-frequency shared words, I would increase the predictive power of the remaining words. Isn't this the point of stopword lists?