1
$\begingroup$

I have a dataset of 300 text documents that I have manually classified into three classes. The first two classes are relatively equal in size; the third class is about 40% larger than each of the individual first two classes.

The classes are very similar, i.e. they share similar features. After pre-processing and removing words that only appear in 2% of the texts, Multinomial Naïve Bayes predicts about 48-58% of the test set (20% of the texts) correctly. I notice that the larger third class is predicted more accurately than the smaller first and second classes.

I would like to refine pre-processing to achieve 65-75% accuracy. The first thing I did was to tokenize the features into bigrams. After doing so, the model only predicts between 40-50%, instead of 48-58% with single words. Intuitively, I expected the model to perform better, not worse, with bigrams. Shouldn't bigram tokenization reduce the amount of shared tokens across the classes and increase the predictive power of the non-shared tokens as a result (at least marginally)?

I am a bit concerned because my next refinement is to create a stopword list of words that frequently appear across all classes but have no predictive content. My reasoning was that by reducing the amount of high-frequency shared words, I would increase the predictive power of the remaining words. Isn't this the point of stopword lists?

$\endgroup$
  • $\begingroup$ Velcome to our site! $\endgroup$ – kjetil b halvorsen Jun 10 '14 at 13:54
1
$\begingroup$

Shouldn't bigram tokenization reduce the amount of shared tokens across the classes and increase the predictive power of the non-shared tokens as a result (at least marginally)?

By using bigrams instead of single words, you are also reducing your amount of data, in that $n_{bigrams} < n_{words}$ most of the time. While you are focused on reducing the amount of shared tokens across the classes, bear in mind that you may also be reducing the amount of distinctive tokens per class, which may have a negative effect on predictive power.

I am not an expert in NLP, but I think the same logic can be applied to stop words.

I would test how these are affected by increasing your data set size if labelling is not very time consuming.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.