Naive Bayes calculation

Question

I was following along with an example given here in which we are trying to classify emails as spam ($S \in \{0, 1\}$) based on the occurrence of the words "buy" ($B \in \{0, 1\}$) and "cheap" ($C \in \{0, 1\}$). From the example, we know the following: $$ \begin{align} P(S=1) &= 0.25\\ P(B=1) &= 0.25\\ P(C=1) &= 0.25\\ P(B=1|S=1) &= 0.8\\ P(B=1|S=0) &= 0.067\\ P(C=1|S=1) &= 0.6\\ P(C=1|S=0) &= 0.133 \end{align} $$ Furthermore, from Bayes rule, we can obtain: $$ \begin{align} P(S=1|B=1) = \frac{P(B=1|S=1)P(S=1)}{P(B=1)} = \frac{(0.8)(0.25)}{0.25} = 0.8\\ P(S=1|C=1) = \frac{P(C=1|S=1)P(S=1)}{P(C=1)} = \frac{(0.6)(0.25)}{0.25} = 0.6 \end{align} $$

Using a naive Bayes classifier, we have that $P(B, C | S) = P(B|S)P(C|S)$. Thus, we find that: $$ \begin{align} P(S | B,C) &= \frac{P(B, C| S) P(S)}{P(B, C)}\\ &= \frac{P(B|S)P(C|S)P(S)}{P(B)P(C)} \mbox{ (by assumption)} \tag{1} \end{align} $$

We can rewrite this last line as $\frac{P(S|B)P(S|C)}{P(S)}$ by using Bayes rule: $$ \begin{align} \frac{P(B|S)P(C|S)P(S)}{P(B)P(C)} &= \frac{P(B|S)P(S)}{P(B)}\frac{P(C|S)P(S)}{P(C)}\frac{1}{P(S)} \tag{2a}\\ &= \frac{P(S|B)P(S|C)}{P(S)} \tag{2b} \end{align} $$

Upon substituting, we get: $$ \begin{align} P(S=1|B=1,C=1) &= \frac{P(S=1|B=1)P(S=1|C=1)}{P(S=1)}\\ &= \frac{(0.8)(0.6)}{0.25}\\ &= 1.92 \end{align} $$ which clearly doesn't make sense since probabilities must be between 0 and 1. On the other hand, we can also write: $$ \begin{align} P(S=1|B=1,C=1) &= \frac{P(B=1|S=1)P(C=1|S=1)P(S=1)}{P(B=1)P(C=1)} \tag{3}\\ &= \frac{P(B=1|S=1)P(C=1|S=1)P(S=1)}{\sum_{S \in \{0,1\}}P(B=1|S)P(C=1|S)P(S)}\\ &= \frac{(0.8)(0.6)(0.25)}{(0.067)(0.133)(0.75) + (0.8)(0.6)(0.25)}\\ &= 0.947 \end{align} $$

Why does the second formula work out while the first does not?

Answer 1

You can check each of your calculations with this table

S   B   C   Probability
1   1   1   0.12
1   1   0   0.08
1   0   1   0.03
1   0   0   0.02
0   1   1   0.006666667
0   1   0   0.043333333
0   0   1   0.093333333
0   0   0   0.606666667

This will give you $P(S=1 \mid B=1,C=1) =\dfrac{P(S=1,B=1,C=1)}{P(B=1,C=1)}= \dfrac{0.12}{0.12+0.006666667}\approx 0.9473684$ which is indeed what you found with your second method

This illustrates that $P(B=1,C=1)= 0.12+0.006666667 = 0.126666667$ while $P(B=1)\,P(C=1) = 0.25 \times 0.25= 0.0625$, demonstrating the lack of marginal independence

Answer 2

The assumption made in equation (1) is incorrect. Naive Bayes makes the assumption that the features are conditionally independent, given the class. This is different from marginal independence, which is what I assumed in the denominator of equation (1). Hence, equation (1) should read: $$ \begin{align} P(S | B,C) &= \frac{P(B, C| S) P(S)}{P(B, C)}\\ &= \frac{P(B|S)P(C|S)P(S)}{P(B, C)} \mbox{ (by assumption)} \tag{1} \end{align} $$ Since the denominator cannot be broken down into the product of two marginal distributions, this is the furthest that this equation can be simplified. Hence, equations (2a) and (2b) are incorrect, which is what led to the incorrect answer 1.92.

Finally, the denominator of equation (3) is also incorrect. Equation (3) should read: \begin{align} P(S=1|B=1,C=1) &= \frac{P(B=1|S=1)P(C=1|S=1)P(S=1)}{P(B=1, C=1)} \tag{3} \end{align} The equations that follow equation (3) are correct, however, since they don't actually make use of the (false) marginal independence of $B$ and $C$ and instead marginalize out $S$ from the joint distribution of $B$, $C$, and $S$.