I'm trying to work through a toy example of Naive Bayes with text classification (spam/ham) to make sure I understand the intuition, but not understanding why my posterior probabilities are not summing up to 1.

To exercise my own intuition, I've created 4 documents:

  1. I would love to go to dinner. (HAM)
  2. Free money now. (SPAM)
  3. They will go now. (HAM)
  4. I love to be free and to have money. (HAM)

From my understanding

$$ P(y=ham|X) = \frac{p(X|y=ham)|p(y=ham)}{p(X)} $$

$p(y=ham)$ is the prior probability, and is the percentage of documents that are ham.

$p(X)$ is the evidence, or intuitively the likelihood of seeing that particular combination of words. It is calculated assuming independence as $$ p(X) = \prod_{i=0}^{N-1}{p(x_i)} $$ where $p(x_i)$ is the probability of finding a certain word in a document.

Finally, $p(X|y=ham)$ is the likelihood of observing that particular combination of words, given that a document is ham.

Now, I want to test a new document that reads

Free money

If I do my calculations,

  • $P(y=ham|X) = \frac{3}{4} = 0.75$
  • $P(X) = P(x = free) \times P(x = money) = \frac{2}{4} \times \frac{2}{4} = \frac{1}{4} = 0.25$
  • $P(X|y = ham) = P(x=free|y=ham) \times P(x=money|y=ham) = \frac{1}{3} \times \frac{1}{3} = 0.11$
  • $P(X|y=spam) = P(x=free|y=spam) \times P(x=money|y=spam) = \frac{1}{1} \times \frac{1}{1} = 1$ Note: I calculated $P(X|y=spam)$ based on the understanding there's only 1 spam document, so the denominator should be 1.

Now, if I calculate my posteriors, I probabilities that sum to greater than 1:

  • $P(y=spam|X) = \frac{1 \times 0.25}{0.25} = 1$
  • $P(y=ham|X) = \frac{0.11 \times 0.75}{0.25} = 0.33$

From what I've always understood, if the events in are mutually exclusive (and in this case, I believe they are- you cannot have both a ham and spam document), then you should be able to sum the posterior probabilities and arrive at 1. This is also supported by answers in the SO post.

However, that's clearly not the case. I get a probability over 1. Why is this?

I've noticed that in tutorials for Naive Bayes, the logic for classification is to check if $P(y = ham|X) > P(y=spam|X)$, not if $P(y=ham|X) > 0.50$. This seems to imply that it is expected that the posteriors may not add up to 1, or may add up beyond 1.


I will go ahead and answer my own question, after doing some more research.

The evidence $P(X)$ should not be calculated simply as $P(X) = P(x_0 = free) \times P(x_1 = money)$, but rather as $$ P(X) = P(X|y=spam)\times P(y=spam) + P(X|y=ham) \times P(y=ham) $$

We assume conditional independence of the observations - that is, we can assume that free and money are independent only after we are given that the document is ham or space. We cannot make this assumption beforehand.

We then arrive at the evidence $P(X) = 0.33$, ultimately providing the posterior probabilities of

$$ P(y=ham|X) = 0.25 $$


$$ P(y=spam|X) = 0.75 $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.