I have written a Naive Bayes classifier (with Laplace smoothing) and am using it to classify text into a few simple classes. However, I found that the classes are not of the same vocab size -- to be clear this is not the same as a problem with some classes being represented more often in the training data, since this would be addressed by the a priori probabilities or by resampling to equalize classes but neither of these helps on the different vocab sizes.

In short, my catch-all class "none" has a much wider vocabulary so P(w|c) is smaller on average for a given common word than in one of the other classes. Another way to see this is that the probability mass (even though Laplace smoothing gives some mass to every word) is much more concentrated in common words for my actual classes and much more spread out for my catch-all "none" class. This means that the more words there are, the more likely the class chosen will not be "none" even when it should be. How can I address this problem?

  • $\begingroup$ Just to understand it better: you're classifying words, not texts, is is right? Because if you are just classifying words, it is not a problem to have a small $P(w|none)$, you just need it to be bigger than $P(w|c)$ for other classes $c$. $\endgroup$
    – Jundiaius
    Feb 4, 2015 at 13:55
  • $\begingroup$ Sorry if I wasn't clear in my question, I am classifying texts and so having smaller probabilities on average for common words when conditioning on the none class means that any other class becomes more likely to be the chosen one as the text gets longer. Since asking the question, to address this I used a stop words list and did some tf-idf weighing, which seems to do some good work towards this but I was wondering what the correct and more general way to solve this would be. Thanks! $\endgroup$
    – hackartist
    Feb 4, 2015 at 22:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.