I'm building a text classifier using the naive Bayes formula. I'm still early in the development, but I already see a problem with my technique and I was wondering if you guys would have idea that would help me solve this problem.

What I want to do is score texts to order them from the more likely to be in class A to the less likely to be. I only have one class and I want to find the likelyhood that a text is in it.

The problem is I only get prediction really near zero (1,068E-12 for exemple). the reason is most words have a probability of being in class A inferior to 0.5. Even if I have words with probabilities > 0.5, theses probabilities are farther from 1 then the probabilities <0.5 are farther from 0.

So when I choose the N words with the probabilities the farthest from .05 I usually get only (or at least more) probabilities <0.5. And so the more words I use (N) the more the probability is near 0.

Is there some optimization I could implement that would help with this problem (For now I don't even remove stop words, but I plan to)?

Or is a Bayes classifier a bad choice for my problem?


Naive Bayes generally uses a decision rule like $$ \text{argmax}_{C_i} P(C_i)P(D|C_i), $$ which comes from the fact we can write $$ P(C_i|D) = \frac{P(C_i)P(D|C_i)}{P(D)}. $$ and drop the denominator $P(D)$ since it does not depend on the class. However, since $P(D) << 1$ (i.e. there are many possible documents), neglecting it will cause the output of your algorithm to be quite small, so this isn't necessarily indication your implementation is incorrect.

A practical tip: One thing you can and should do is work with sums of log probabilities rather than products of probabilities to avoid underflow errors. Rather than doing $$ P(D|C_i) = \prod_{w_j \in D} P(w_j|C_i), $$ do $$ \log P(D|C_i) = \sum_{w_j \in D} \log P(w_j|C_i). $$ You'll also need to deal with unseen words as well since zero probabilities will give you problems.

  • $\begingroup$ Hi alto. Thanks for answering. I'm a programmer, not a mathematician, so tell me if I understand your last equation correctly. The probability of document D being of class Ci is the sum of the logarithms of the probabilities of a document containing the word wj being of class Ci. Is that right? $\endgroup$ – Mathieu Pagé Jul 9 '12 at 11:35
  • $\begingroup$ Also, does this give me probabilities or just relatives scores? $\endgroup$ – Mathieu Pagé Jul 9 '12 at 11:35
  • $\begingroup$ Not that I currently used the formula presented here : paulgraham.com/naivebayes.html which I'm not sure is equivalent to your first equation. Maybe I started with the wrong algorithm. I'll go read more on naive Bayes. $\endgroup$ – Mathieu Pagé Jul 9 '12 at 11:41
  • $\begingroup$ I think you are reading the conditional backwards. $P(D|C)$ should be read the probability of a document given the class, not the other way around. The wiki page has a good example in the context of document classification here. It might be easier to read through this and then allow us to clear anything up which might be giving you problems afterward. $\endgroup$ – alto Jul 9 '12 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.