I'm prototyping my own Naive Bayes bag o' words model, and I had a question about calculating the feature probabilities.

Let's say I've got two classes, I'll just use spam and not-spam since that's what everyone uses. And let's take the word "viagra" as an example. I have 10 emails in my training set, 5 spam and 5 non-spam. "viagra" appears in all 5 spam documents. In one of the training documents it appears 3 times (this is what my question is about), so that's 7 appearances in spam total. In the non-spam training set, it appears 1 time.

If I want to estimate p(viagra | spam) is it simply:

p(viagra | spam) = 5 spam documents contain viagra / 5 spam documents total = 1

In other words, does the fact that one document mentioned viagra 3 times instead of once really not matter?

Edit: Here's a blog post where the author uses the approach I just laid out: http://ebiquity.umbc.edu/blogger/2010/12/07/naive-bayes-classifier-in-50-lines/

And here's a blog post where the author says: p(viagra | spam) = 7 viagra spam mentions / 8 total mentions http://www.nils-haldenwang.de/computer-science/machine-learning/how-to-apply-naive-bayes-classifiers-to-document-classification-problems

And then one of the answers below says it should be: p(viagra | spam) = 7 viagra mentions in spam / total term count in spam

Can anyone link to a source that gives an opinion on this?


3 Answers 3


In other words, does the fact that one document mentioned viagra 3 times instead of once really not matter?

It does matter. The Multinomial Naive Bayes model takes into account each occurrence of a token, whereas the Bernoulli Naive Bayes model does not (i.e. for the latter model, 3 occurrences of "viagra" is the same as 1 occurrence of "viagra").

Here are two illustrations as well as a comparison table from {1}:

enter image description here

enter image description here

enter image description here

{1} neatly introduces Naive Bayes for text classification, as well as the Multinomial Naive Bayes model and the Bernoulli Naive Bayes model.



It depends on the specific naive model you apply. In general, for Text Classification, you do want to consider repetition of terms, so the answer is yes.

The other point is that you are considering the probability based on the document event space. You can also do it based on the term space:

p(viagra | spam) = 5 times spam term in class spam / 50 terms in the class

You have a lot of information in this [paper] (http://echo.edres.org:8080/betsy/mccallum1.pdf)


I think it depends on what exactly you mean by p(viagra|spam) and how you're modelling the data.

As written, I would interpret your meaning as 'the probability the word viagra is mentioned at least once in a message, given this message is spam'. In that case, yes, the fact that one document mentioned viagra three times has no effect. You've defined a model which does not pay attention to such facts.

Of course, you could have a different model. For example, instead of viagra being representing by a binary variable (present/absent), it could represent the count of the number of times the word appears in the message. In that case, from your raw data you'd estimate an empirical frequency of something like

p(viagra=0|spam) = 0

p(viagra=1|spam) = 4/5

p(viagra=2|spam) = 0

p(viagra=3|spam) = 1/5


I'm not saying that's a better way to do it. I'm just illustrating an alternate situation where your intuition that seeing viagra mentioned three times is relevant holds.

A more practical example might be 'Term Frequency–Inverse Document Frequency', which is a method that pays a lot of attention to the frequency of a word in a document.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.