1
$\begingroup$

I am trying to implement Naive Bayes, but I am encountering a problem. I have 5000 word features. Hence, every sample is a binary vector of length 5000. The true labels are 1 or 0. The value of P(feature=1 | label=1) and P(feature=0 | label=1) are very small (~0.03) as the feature vector is very sparse. When I calculate the numerator i.e.

P(features | label=1) * P(label=1)

since, the probability values are very small and because of the conditional independence assumption of Naive Bayes, when I multiply 2000 such small terms, I get 0 and hence, a wrong result. What should be done?

$\endgroup$
4
$\begingroup$

The two most commonly used techniques to prevent underflows with a naive Bayes classifier are:

  1. Working in the log space
  2. Using the log-sum-exp trick

More details: Example of how the log-sum-exp trick works in Naive Bayes


FYI:

$\endgroup$
  • $\begingroup$ I did the summation of log probabilities. But, the sum of log probabilities comes out ot be -100000. Still, the answer comes out to be wrong as log P = -100000 and hence P would be 0. What should I do now? $\endgroup$ – Hellboy Feb 11 '17 at 19:42
  • $\begingroup$ @Hellboy What do you want to do with P? $\endgroup$ – Franck Dernoncourt Feb 11 '17 at 19:47
  • $\begingroup$ Since, it is a binary classifier, I'll compare P with 0.5, if it is greater than 0.5, I'll assign positive label to the test sample. $\endgroup$ – Hellboy Feb 11 '17 at 19:51
  • $\begingroup$ @Hellboy How about comparing log P with log 0.5? $\endgroup$ – Franck Dernoncourt Feb 11 '17 at 19:53
  • $\begingroup$ Yes, but every sample is being classified as a negative sample as every sum is less than log 0.5 $\endgroup$ – Hellboy Feb 11 '17 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.