# In Naïve Bayes, why do we estimate Pr(W|H)*Pr(H) instead of Pr(W)

This wikipedia article describes spam filtering using Naïve Bayes: https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering

It says P(S|W) is given as Pr(W|S)*Pr(S) / (Pr(W|S)*Pr(S)) + Pr(W|H)*Pr(H)).

However, one could also get P(S|W) by estimating P(W) instead.

Most textbooks simply say it's unnecessary to estimate P(W) which I get, but one could also say it's unnecessary to estimate Pr(W|H)*Pr(H). Why is it that estimating Pr(W|H)*Pr(H)) is preferred?

Example: If we use this example,

The "correct" estimation for P(yes|rain, good) is 0.143 because

P(rain|yes) * P(good|yes) * P(yes) = 1/5 * 1/5 * 5/10 = 0.02
P(rain|no) * P(good|no) * P(no) = 2/5 * 3/5 * 5/10 = 0.12
P(yes|rain, good) = 0.02 / (0.02 + 0.12) = 0.143


Whereas one could also estimate it as follows which gives 0.042 instead:

P(rain|yes) * P(good|yes) * P(yes) = 1/5 * 1/5 * 5/10 = 0.02
P(rain, good) = P(rain) * P(good) = 3/5 * 4/5 = 0.48
P(yes|rain, good) = 0.02 / 0.48 = 0.042


My question is, why is the former preferred, even though they seem to make similar approximations

• Pr(W) = (Pr(W|S)*Pr(S)) + Pr(W|H)*Pr(H)) Apr 10 at 10:47
• Approximating Pr( W1, W2 | S) = Pr( W1 | S) * Pr( W2 | S ) vs. Pr( W1, W2 ) = Pr( W1 ) * Pr( W2 ) is not the same though, no? @J.Delaney Apr 10 at 11:25
• What you wrote is an assumption of conditional independence. Not sure how it is related to the question Apr 10 at 11:32
• What I mean is, by assuming conditional independence, one could estimate Pr(W1,W2|H) or Pr(W1, W2), both of which can be used to estimate Pr(S|W1, W2). But the two ways can result in a different estimate thanks to the conditional independence usually not being true. It seems that estimating Pr(W1,W2|H) is always preferred, and my question is, why is that - why is that better than estimating Pr(W1, W2)? @J.Delaney Apr 10 at 11:42
• I can't see which "two ways" you are referring to. Try clarifying your question Apr 10 at 11:52

You can immediately see that the second calculation is incorrect because probabilities should add to 1, namely you should always have P(yes|rain, good) + P(no|rain, good) = 1. But this is not the case in your second calculation, so clearly it makes no sense.
The reason for this is that conditional independence does not imply independence. Naïve Bayes assumes that the features (e.g. road condition & weather condition) are conditionally independent given the target variable, so for example P(rain,good|yes)=P(rain|yes)*P(good|yes) but this does not imply that P(rain, good) = P(rain) * P(good), hence the inconsistency in your calculation.
• I can satisfy P(yes|rain, good) + P(no|rain, good) = 1 by simply estimating P(no|rain, good) = 1 - P(yes|rain, good) though. And to your second point, you are saying one assumption does not imply a different assumption, which is true, but it doesn't answer why the assumption NB makes is better than assuming raining and road condition being independent. Both are wrong assumptions, so shouldn't they be equally correct? Apr 11 at 8:02