Naive question about Naive Bayes modeling

Question

In Naive Bayes classifiers, one calculates a frequency table to determine a prediction. A classic example, one calculates the frequency table of words given the context of spam or ham. E.g. P( viagra | spam ) Which is the probability that given a training set with spam messages identified, how often does viagra appear in those spam messages.

Why not calculate simply P(spam | viagra) directly from the training set? Look at the word viagra, what percent of the time does it appear in spam messages?

What benefit is gained from doing the frequency counting in the first manner?

The next step of the combination of the individual probabilities using the "naive" assumption would still take place using the products of the individual P(W|S) probabilities (usually summed in the log form). This question only pertains to the individual probabilities. I do remember building systems in both manners (frequency counting directly and indirectly) with both having adequate performance. Refreshing my memory on this, I am wondering about the need for the first method.

After writing this question, two reasons occurred to me.

Reason #1: For imbalanced classification problems. By determining P(W|S) as well as P(S), then one is able to tune P(S) as a parameter instead of clumsy over/under sampling techniques. Which is more idiomatic Naive Bayes since it is generative in nature.

Reason #2: For classification using numeric predictor variables. For example, using this parameterized Gaussian equation:

equation for a normal distribution

The above gives P(x|S) in the context of spam.

---

Thank you Tim for a good response. I am aware of the step for combining the probabilities to alleviate the need to calculate the joint probabilities as you indicate.

But one aspect it brings to the forefront is that there are several methods for combining the individual probabilities.

The conventional method:

(P(S) * Product( P(Wi | S),... ))
-----------------------------------------------------------
(P(S) * Product( P(Wi | S),... )) + (P(H) * Product( P(Wi | H),... ))

However in the https://en.m.wikipedia.org/wiki/Naive_Bayes_spam_filtering article, it implies that this is suitable:

DO NOT USE THIS FORMULA:

Product( P(S | Wi),... )
-----------------------------------------
Product( P(S | Wi),... ) + Product( P(H | Wi),... )

DO NOT USE THE ABOVE FORMULA. I did notice that there is some controversy concerning this particular formula with the Wikipedia editors... The reason being is that by using this incorrect formulation, the prior P(S) is incorporated too many times and for an imbalanced problem (P(S) < P(H)), the prediction will be biased towards zero.

Note the product formulas should not be used directly but instead via the log() formulation. Remember x*y is log(x)+log(y) See for more details: https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering#Other_expression_of_the_formula_for_combining_individual_probabilities

Answer 1

I agree with you that for calculating conditional probablity given single binary variable, this does not matter, and you can calculate it directly as well. But consider case where you have multiple features, say $p(Y|A,B,C,D)$. To calculate such conditional probability directly from the data, you would need sufficient samples for all the combinations of the different levels of all the variables. This would mean, that you need pretty huge dataset. For English language spam, this would mean samples of all the possible combinations of all the possible English words that can form an e-mail, so an infinitely large dataset.

Naive Bayes algorithm solves this by replacing the need of observing all the possible combinations, by using only the pairs of each of the variables with the predicted variable

$$ p(Y|A,B,C,D) \propto p(A|Y) \,p(B|Y)\, p(C|Y)\, p(D|Y) \,p(Y) $$