3
$\begingroup$

I was following along with an example given here in which we are trying to classify emails as spam ($S \in \{0, 1\}$) based on the occurrence of the words "buy" ($B \in \{0, 1\}$) and "cheap" ($C \in \{0, 1\}$). From the example, we know the following: $$ \begin{align} P(S=1) &= 0.25\\ P(B=1) &= 0.25\\ P(C=1) &= 0.25\\ P(B=1|S=1) &= 0.8\\ P(B=1|S=0) &= 0.067\\ P(C=1|S=1) &= 0.6\\ P(C=1|S=0) &= 0.133 \end{align} $$ Furthermore, from Bayes rule, we can obtain: $$ \begin{align} P(S=1|B=1) = \frac{P(B=1|S=1)P(S=1)}{P(B=1)} = \frac{(0.8)(0.25)}{0.25} = 0.8\\ P(S=1|C=1) = \frac{P(C=1|S=1)P(S=1)}{P(C=1)} = \frac{(0.6)(0.25)}{0.25} = 0.6 \end{align} $$

Using a naive Bayes classifier, we have that $P(B, C | S) = P(B|S)P(C|S)$. Thus, we find that: $$ \begin{align} P(S | B,C) &= \frac{P(B, C| S) P(S)}{P(B, C)}\\ &= \frac{P(B|S)P(C|S)P(S)}{P(B)P(C)} \mbox{ (by assumption)} \tag{1} \end{align} $$

We can rewrite this last line as $\frac{P(S|B)P(S|C)}{P(S)}$ by using Bayes rule: $$ \begin{align} \frac{P(B|S)P(C|S)P(S)}{P(B)P(C)} &= \frac{P(B|S)P(S)}{P(B)}\frac{P(C|S)P(S)}{P(C)}\frac{1}{P(S)} \tag{2a}\\ &= \frac{P(S|B)P(S|C)}{P(S)} \tag{2b} \end{align} $$

Upon substituting, we get: $$ \begin{align} P(S=1|B=1,C=1) &= \frac{P(S=1|B=1)P(S=1|C=1)}{P(S=1)}\\ &= \frac{(0.8)(0.6)}{0.25}\\ &= 1.92 \end{align} $$ which clearly doesn't make sense since probabilities must be between 0 and 1. On the other hand, we can also write: $$ \begin{align} P(S=1|B=1,C=1) &= \frac{P(B=1|S=1)P(C=1|S=1)P(S=1)}{P(B=1)P(C=1)} \tag{3}\\ &= \frac{P(B=1|S=1)P(C=1|S=1)P(S=1)}{\sum_{S \in \{0,1\}}P(B=1|S)P(C=1|S)P(S)}\\ &= \frac{(0.8)(0.6)(0.25)}{(0.067)(0.133)(0.75) + (0.8)(0.6)(0.25)}\\ &= 0.947 \end{align} $$

Why does the second formula work out while the first does not?

$\endgroup$
3
  • $\begingroup$ Another issue is $P(B)P(C)=\dfrac{1}{16}\not=\dfrac{19}{150}=P(B,C)$. Conditional independence does not imply unconditional independence $\endgroup$
    – Henry
    Jan 16, 2020 at 6:15
  • $\begingroup$ How exactly did you "rewrite the last line"? $\endgroup$
    – Tim
    Jan 16, 2020 at 6:21
  • $\begingroup$ Tim, I was looking at the @Henry's deleted comment late last night and thought the mistake had been caught, but it had not. I have edited the question and expanded out how I "rewrote the last line." I now realize the mistake is, as Henry pointed out in his comment above, in my assumption of independence. I was going off the equation given here at 18:45, which seems to state that the independence among features is unconditional, when, in fact, it is class-conditional. I will answer my own question to share with other readers. $\endgroup$ Jan 16, 2020 at 19:02

2 Answers 2

1
$\begingroup$

You can check each of your calculations with this table

S   B   C   Probability
1   1   1   0.12
1   1   0   0.08
1   0   1   0.03
1   0   0   0.02
0   1   1   0.006666667
0   1   0   0.043333333
0   0   1   0.093333333
0   0   0   0.606666667

This will give you $P(S=1 \mid B=1,C=1) =\dfrac{P(S=1,B=1,C=1)}{P(B=1,C=1)}= \dfrac{0.12}{0.12+0.006666667}\approx 0.9473684$ which is indeed what you found with your second method

This illustrates that $P(B=1,C=1)= 0.12+0.006666667 = 0.126666667$ while $P(B=1)\,P(C=1) = 0.25 \times 0.25= 0.0625$, demonstrating the lack of marginal independence

$\endgroup$
1
  • $\begingroup$ Thanks, @Henry. I posted a follow-up question to this one here and was wondering if you could take a look at it when you get a chance. $\endgroup$ Jan 16, 2020 at 23:55
0
$\begingroup$

The assumption made in equation (1) is incorrect. Naive Bayes makes the assumption that the features are conditionally independent, given the class. This is different from marginal independence, which is what I assumed in the denominator of equation (1). Hence, equation (1) should read: $$ \begin{align} P(S | B,C) &= \frac{P(B, C| S) P(S)}{P(B, C)}\\ &= \frac{P(B|S)P(C|S)P(S)}{P(B, C)} \mbox{ (by assumption)} \tag{1} \end{align} $$ Since the denominator cannot be broken down into the product of two marginal distributions, this is the furthest that this equation can be simplified. Hence, equations (2a) and (2b) are incorrect, which is what led to the incorrect answer 1.92.

Finally, the denominator of equation (3) is also incorrect. Equation (3) should read: \begin{align} P(S=1|B=1,C=1) &= \frac{P(B=1|S=1)P(C=1|S=1)P(S=1)}{P(B=1, C=1)} \tag{3} \end{align} The equations that follow equation (3) are correct, however, since they don't actually make use of the (false) marginal independence of $B$ and $C$ and instead marginalize out $S$ from the joint distribution of $B$, $C$, and $S$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.