Naive Bayes Classifier Unclear

Question

I read the following sentence regarding the Naive Bayes Classifier:

If large number of features have relatively minor effects, taken together, their combined impact could be quite large.

Could you explain that connection, as I don't get why their combined impact should be large when they hardly affect anything individually.

Answer 1

A standard text-book example for Naive Bayes classifier with large number of features is the spam classification problem, where you have the mails in text format and decide whether the mail is spam or not. Typically, we have a large dictionary, say 10K words, and we estimate the class conditional probabilities of each word, e.g. probability of word $w$ appearing in spam or non-spam mails.

For a given mail, we decide based on the log posterior $\log P(S|D)$, which means deciding based on $\log P(D|S) P(S)$, where $S$ represents being spam, and $D$ represents the data (here given mail). Here, $\log P(D|S)=\sum \log P(D_i|S)$, i.e. summation of each word's log-probability of appearance in spam mails (which is calculated from your training set). Each $P(D_i|S)$ is a small probability like $1/10000$ (log10 of it is $-5$), but when we add all (logs) them up, we observe their combined effect. For example, $P(S|D)$ can be around $-5000$, $P(S'|D)$ can be around $-3000$. This difference is created by the small differences in $P(D_i|S)$. Sometimes, words like 1000000$ can create huge differences in class conditional probabilities, because they almost never appear in non-spam mails, however it doesn't hurt the idea of Bayes Classifier, that is analyzing the overall text in a probabilistic framework.

I believe above is the computational perspective; linguistically, consider this answer or any paragraph you read. Individual words in general make no sense. But, they're small contributors of the idea inside.