Naive Bayes feature probabilities: should I double count words?

Question

I'm prototyping my own Naive Bayes bag o' words model, and I had a question about calculating the feature probabilities.

Let's say I've got two classes, I'll just use spam and not-spam since that's what everyone uses. And let's take the word "viagra" as an example. I have 10 emails in my training set, 5 spam and 5 non-spam. "viagra" appears in all 5 spam documents. In one of the training documents it appears 3 times (this is what my question is about), so that's 7 appearances in spam total. In the non-spam training set, it appears 1 time.

If I want to estimate p(viagra | spam) is it simply:

p(viagra | spam) = 5 spam documents contain viagra / 5 spam documents total = 1

In other words, does the fact that one document mentioned viagra 3 times instead of once really not matter?

---

Edit: Here's a blog post where the author uses the approach I just laid out: http://ebiquity.umbc.edu/blogger/2010/12/07/naive-bayes-classifier-in-50-lines/

And here's a blog post where the author says: p(viagra | spam) = 7 viagra spam mentions / 8 total mentions http://www.nils-haldenwang.de/computer-science/machine-learning/how-to-apply-naive-bayes-classifiers-to-document-classification-problems

And then one of the answers below says it should be: p(viagra | spam) = 7 viagra mentions in spam / total term count in spam

Can anyone link to a source that gives an opinion on this?

Answer 1

In other words, does the fact that one document mentioned viagra 3 times instead of once really not matter?

It does matter. The Multinomial Naive Bayes model takes into account each occurrence of a token, whereas the Bernoulli Naive Bayes model does not (i.e. for the latter model, 3 occurrences of "viagra" is the same as 1 occurrence of "viagra").

Here are two illustrations as well as a comparison table from {1}:

{1} neatly introduces Naive Bayes for text classification, as well as the Multinomial Naive Bayes model and the Bernoulli Naive Bayes model.

---

References:

• {1} Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. "Introduction to Information Retrieval." 2009, chapter 13 Text classification and Naive Bayes.

Answer 2

It depends on the specific naive model you apply. In general, for Text Classification, you do want to consider repetition of terms, so the answer is yes.

The other point is that you are considering the probability based on the document event space. You can also do it based on the term space:

p(viagra | spam) = 5 times spam term in class spam / 50 terms in the class

You have a lot of information in this [paper] (http://echo.edres.org:8080/betsy/mccallum1.pdf)

Answer 3

I think it depends on what exactly you mean by p(viagra|spam) and how you're modelling the data.

As written, I would interpret your meaning as 'the probability the word viagra is mentioned at least once in a message, given this message is spam'. In that case, yes, the fact that one document mentioned viagra three times has no effect. You've defined a model which does not pay attention to such facts.

Of course, you could have a different model. For example, instead of viagra being representing by a binary variable (present/absent), it could represent the count of the number of times the word appears in the message. In that case, from your raw data you'd estimate an empirical frequency of something like

p(viagra=0|spam) = 0

p(viagra=1|spam) = 4/5

p(viagra=2|spam) = 0

p(viagra=3|spam) = 1/5

etc.

I'm not saying that's a better way to do it. I'm just illustrating an alternate situation where your intuition that seeing viagra mentioned three times is relevant holds.

A more practical example might be 'Term Frequency–Inverse Document Frequency', which is a method that pays a lot of attention to the frequency of a word in a document.