Which Naive Bayes in the Wikipedia article "Naive Bayes spam filtering"?

Question

In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes?

The paper states that binary Multinomial NB performs best.

I coded up the NB equations listed in the Naive Bayes spam filtering article on Wikipedia Computing the probability that a message containing a given word is spam:

Biased equation

and combine the probabilities using the log form of:

But I cannot tell which NB the Wikipedia article uses... Is it a binary multinomial equation or a Bernoulli form or some other?

Another question:

The article states that many implementations use the non-biased P(S|W) equation. Thinking about the biased equation above compared to the one below, I wonder if the biased equation involves the ratio of spam to non-spam perhaps twice if the training set has the same ratio is the same imbalanced ratio as the live data.

Non-biased equation

The concern is from how is the (spam occurrences for this word)/(total spam occurrences). But thinking more, I suspect the equations do not double count... Mmm...

Is the non-biased equation always better when using a real-word imbalanced training set? On my data, it sure is:

Biased Equation: Overall acc= 85% Specificity (TNR)= 6% Sensitivity (TPR)= 99%

Non-biased equation: Overall acc= 80% Specificity (TNR)= 63% Sensitivity (TPR)= 83%

(Note: I applied the to avoid over fitting and did a hold out as test...)

Thoughts? Much thanks for the help.

P.S. I coded this in Python from scratch to learn Python. But which library is best to use for this in Python for NB?

P.P.S I also do a lot of R coding, which library is best to use for NB in R?

Answer 1

The "naive" part treats each word (text piece) as independent of other words. It exactly solves the problem by wrongly assuming independence, rather than badly solving the correct problem with correlations between words. Naive Bayes works very well with spam, since the spam must have some goal. It works very slowly without intelligent reasons for the "spam"; eg, U.S. Department of Transportation uses internal manufacturing reports as Naive Bayes uses email, but rather than 200 emails getting good results, DOT needs 30,000 reports for rare events like deaths suggesting a recall. DOT actually uses the Bayes technique CRM114, which goes through not words but all combinations of 5 adjacent words, getting better results. CRM114 works better than standard Naive Bayes, though CRM114 merely modifies Naive Bayes. It takes much longer to get CRM114 working against email spam -- I use standard Naive Bayes, which works very well on email spam.

If your goal is to reduce email spam, rather than doing this for an exercise, consider using the package spamassassin (in Linux) which includes Naive Bayes. When trained with about 200 records of each type, this gets rid of all spam with virtually no false positives, lasting for 6 months before needing a little more email training.

Somehow, Paul Graham got much recognition as the founder of Naive Bayes in email, so he gives a formula http://www.paulgraham.com/naivebayes.html

Answer 2

Again, reading this paper Spam Filtering with Naive Bayes – Which Naive Bayes? by Metsis et al., I noticed this sentence:

The multinomial nb with Boolean attributes... It differs from the multi-variate Bernoulli nb in that it does not take into account directly the absence (xi = 0) of tokens from the message (there is no (1 − p(ti | c))^(1−xi) factor), and it estimates the p(t | c) with a different Laplacean prior. (Metsis et al. )

The wikipedia article in question, does not take into account the absence of tokens nor is there (1 − p(ti | c))^(1−xi) expression. Therefore it appears to me to be more closely aligned with "multinomial nb with Boolean attributes".

This also backs up my discovery that the "biased" equation is essentially rubbish. In fact, locking this formula with Pr(S) as 0.5 performs better than the actual value -- which simplifies it to nearly to Laplacean smoothing prior.

I do think the Wikipedia article needs editing to clarify this, but I will leave that up to the Wikipedia editors.