1
$\begingroup$

This wikipedia article describes spam filtering using Naïve Bayes: https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering

It says P(S|W) is given as Pr(W|S)*Pr(S) / (Pr(W|S)*Pr(S)) + Pr(W|H)*Pr(H)).

However, one could also get P(S|W) by estimating P(W) instead.

Most textbooks simply say it's unnecessary to estimate P(W) which I get, but one could also say it's unnecessary to estimate Pr(W|H)*Pr(H). Why is it that estimating Pr(W|H)*Pr(H)) is preferred?

Example: If we use this example,

The "correct" estimation for P(yes|rain, good) is 0.143 because

P(rain|yes) * P(good|yes) * P(yes) = 1/5 * 1/5 * 5/10 = 0.02
P(rain|no) * P(good|no) * P(no) = 2/5 * 3/5 * 5/10 = 0.12
P(yes|rain, good) = 0.02 / (0.02 + 0.12) = 0.143

Whereas one could also estimate it as follows which gives 0.042 instead:

P(rain|yes) * P(good|yes) * P(yes) = 1/5 * 1/5 * 5/10 = 0.02
P(rain, good) = P(rain) * P(good) = 3/5 * 4/5 = 0.48
P(yes|rain, good) = 0.02 / 0.48 = 0.042

My question is, why is the former preferred, even though they seem to make similar approximations

Improve this question
$\endgroup$
6
  • $\begingroup$ Pr(W) = (Pr(W|S)*Pr(S)) + Pr(W|H)*Pr(H)) $\endgroup$
    – J. Delaney
    Apr 10 at 10:47
  • $\begingroup$ Approximating Pr( W1, W2 | S) = Pr( W1 | S) * Pr( W2 | S ) vs. Pr( W1, W2 ) = Pr( W1 ) * Pr( W2 ) is not the same though, no? @J.Delaney $\endgroup$
    – BasilTomato
    Apr 10 at 11:25
  • $\begingroup$ What you wrote is an assumption of conditional independence. Not sure how it is related to the question $\endgroup$
    – J. Delaney
    Apr 10 at 11:32
  • $\begingroup$ What I mean is, by assuming conditional independence, one could estimate Pr(W1,W2|H) or Pr(W1, W2), both of which can be used to estimate Pr(S|W1, W2). But the two ways can result in a different estimate thanks to the conditional independence usually not being true. It seems that estimating Pr(W1,W2|H) is always preferred, and my question is, why is that - why is that better than estimating Pr(W1, W2)? @J.Delaney $\endgroup$
    – BasilTomato
    Apr 10 at 11:42
  • $\begingroup$ I can't see which "two ways" you are referring to. Try clarifying your question $\endgroup$
    – J. Delaney
    Apr 10 at 11:52

1 Answer 1

Reset to default
2
$\begingroup$

You can immediately see that the second calculation is incorrect because probabilities should add to 1, namely you should always have P(yes|rain, good) + P(no|rain, good) = 1. But this is not the case in your second calculation, so clearly it makes no sense.

The reason for this is that conditional independence does not imply independence. Naïve Bayes assumes that the features (e.g. road condition & weather condition) are conditionally independent given the target variable, so for example P(rain,good|yes)=P(rain|yes)*P(good|yes) but this does not imply that P(rain, good) = P(rain) * P(good), hence the inconsistency in your calculation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I can satisfy P(yes|rain, good) + P(no|rain, good) = 1 by simply estimating P(no|rain, good) = 1 - P(yes|rain, good) though. And to your second point, you are saying one assumption does not imply a different assumption, which is true, but it doesn't answer why the assumption NB makes is better than assuming raining and road condition being independent. Both are wrong assumptions, so shouldn't they be equally correct? $\endgroup$
    – BasilTomato
    Apr 11 at 8:02
  • 1
    $\begingroup$ The problem is that in the second case you are assuming both conditional and total independence. The two together in fact imply that the features are independent of the target variable (notice that total independence is a much stronger assumption). This is clearly not what you want, and it makes the calculation inconsistent since you are using estimates that violate your own assumptions (as evident by the fact that you can get different results by performing the same calculation in different ways). $\endgroup$
    – J. Delaney
    Apr 11 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.