When applying a naive Bayes classifier to document classification, the classifier (at least in the applications I've seen) always iterates only over the words actually present in the document.

That is, when computing (say)

$P(SPAM|d) \propto P(d|SPAM) * P(SPAM) = [P(w_1 | SPAM) \cdots P(w_n | SPAM)] P(SPAM)$

the $w_i$ only include words actually present in the document $d$. So $P(w_1|SPAM)$ is always $P(w_1 \text{ present}|SPAM)$, and never $P(w_1 \text{ not present} | SPAM).$

Why is this? I can't think of any a priori reason why including the missing features would hurt the classifier's performance.


A primary concern is efficiency. If you include all missing words, you have to incorporate every word in the whole corpus.

Beside, adding this words may improve the performance in rare cases, put I predict that in general it will decrease performance on unseen data.

In general, including a missing word will improve the performance if it has not been used in the text on purpose or because the spammer will never consider it.

Three examples

  • Let's say the word eudaimonia has appeared in the corpus several times because the mail inbox does belong to a man who lives in a barrel. No mail containing this word was spam and this a word a spammer will certainly not consider to use to make you open the mail.
  • A word which may have not used on purpose is viagra, just because everyone knows that emails which such words are already filtered using blacklists.
  • The majority of the missing words are, however, a word the spammer may be using anytime (just because it is belonging to the base vocabulary), even it does not appear in the training data set (if it appears, it should be rendered useless by the algorithm just because it is a common word).

So, in summary, you may improve the performance in the validation set, but will hurt the future performance on unseen data due to the mass of common words. You may overcome this problem by throwing a ton of data on the classifer, but at this point it is just getting impractical.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.