In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes?

The paper states that binary Multinomial NB performs best.

I coded up the NB equations listed in the Naive Bayes spam filtering article on Wikipedia Computing the probability that a message containing a given word is spam:

Biased equation

P(Spam given Word)

and combine the probabilities using the log form of:

combining probabilities

But I cannot tell which NB the Wikipedia article uses... Is it a binary multinomial equation or a Bernoulli form or some other?

Another question:

The article states that many implementations use the non-biased P(S|W) equation. Thinking about the biased equation above compared to the one below, I wonder if the biased equation involves the ratio of spam to non-spam perhaps twice if the training set has the same ratio is the same imbalanced ratio as the live data.

Non-biased equation

not biased probability equation

The concern is from how P(W|S) is the (spam occurrences for this word)/(total spam occurrences). But thinking more, I suspect the equations do not double count... Mmm...

Is the non-biased equation always better when using a real-word imbalanced training set? On my data, it sure is:

Biased Equation: Overall acc= 85% Specificity (TNR)= 6% Sensitivity (TPR)= 99%

Non-biased equation: Overall acc= 80% Specificity (TNR)= 63% Sensitivity (TPR)= 83%

(Note: I applied the equation to avoid over fitting and did a hold out as test...)

Thoughts? Much thanks for the help.

P.S. I coded this in Python from scratch to learn Python. But which library is best to use for this in Python for NB?

P.P.S I also do a lot of R coding, which library is best to use for NB in R?


The "naive" part treats each word (text piece) as independent of other words. It exactly solves the problem by wrongly assuming independence, rather than badly solving the correct problem with correlations between words. Naive Bayes works very well with spam, since the spam must have some goal. It works very slowly without intelligent reasons for the "spam"; eg, U.S. Department of Transportation uses internal manufacturing reports as Naive Bayes uses email, but rather than 200 emails getting good results, DOT needs 30,000 reports for rare events like deaths suggesting a recall. DOT actually uses the Bayes technique CRM114, which goes through not words but all combinations of 5 adjacent words, getting better results. CRM114 works better than standard Naive Bayes, though CRM114 merely modifies Naive Bayes. It takes much longer to get CRM114 working against email spam -- I use standard Naive Bayes, which works very well on email spam.

If your goal is to reduce email spam, rather than doing this for an exercise, consider using the package spamassassin (in Linux) which includes Naive Bayes. When trained with about 200 records of each type, this gets rid of all spam with virtually no false positives, lasting for 6 months before needing a little more email training.

Somehow, Paul Graham got much recognition as the founder of Naive Bayes in email, so he gives a formula http://www.paulgraham.com/naivebayes.html

  • $\begingroup$ Thank you for the tip about CRM114. I am actually going to use the formulas for a text classification problem that doesn't involve email nor spam. The question involves which type of NB algorithm the wiki post refers to. I think it's actually a multinomial Naive Bayes based on my reading of a published paper. While at first blush, people may think I should just use skilearn packages, but I do need the raw formulas for a special project. On my test concept, I got good results after fixing errors in the wiki article. $\endgroup$ – Chris Sep 11 '15 at 6:26

Again, reading this paper Spam Filtering with Naive Bayes – Which Naive Bayes? by Metsis et al., I noticed this sentence:

The multinomial nb with Boolean attributes... It differs from the multi-variate Bernoulli nb in that it does not take into account directly the absence (xi = 0) of tokens from the message (there is no (1 − p(ti | c))^(1−xi) factor), and it estimates the p(t | c) with a different Laplacean prior. (Metsis et al. )

The wikipedia article in question, does not take into account the absence of tokens nor is there (1 − p(ti | c))^(1−xi) expression. Therefore it appears to me to be more closely aligned with "multinomial nb with Boolean attributes".

This also backs up my discovery that the "biased" equation is essentially rubbish. In fact, locking this formula enter image description here with Pr(S) as 0.5 performs better than the actual value -- which simplifies it to nearly to Laplacean smoothing prior.

I do think the Wikipedia article needs editing to clarify this, but I will leave that up to the Wikipedia editors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.