Naive Bayes bias derivation

Question

I am following this course (notebook) and I wonder about how this derivation of the Naive Bayes came to be.

So, there is a $X$ defined to be the term frequency matrix between documents (rows) and words (cols). The documents can be classified to either positive or negative documents. $P_1$ is defined to be the per-word positive count vector (i.e. in how many positive documents did each word appear). Similarly $P_0$ is defined to be the negative count.

in python:

p1 = np.squeeze(np.asarray(x[y.items==positive].sum(0)))
p0 = np.squeeze(np.asarray(x[y.items==negative].sum(0)))

$Prob_1$ is defined to be $P_1$ divided by the total number of positive documents, and likely $Prob_0$ is $P_0$ divided by the total number of negative documents. We add 1 to avoid division by 0.

pr1 = (p1+1) / ((y.items==positive).sum() + 1)
pr0 = (p0+1) / ((y.items==negative).sum() + 1)

Now, we define $r = log(\frac{Prob_1}{Prob_0})$

To predict whether a document is positive or negative, you calculate the dot product between it and the vector $r$.

So far so good, but now comes some bias term $b$ that I don't understand. It is defined in python as:

b = np.log((y.items==positive).mean() / (y.items==negative).mean())

I'm pretty sure they could simply use count instead of mean. But ok.

The next step is to calculate the predictions, which is:

$\hat{y} =$ \begin{cases} 1, & X_{doc} * r + b >0 \\ 0, & X_{doc} * r + b \le 0 \end{cases}

I get it that you might want to correct for the difference ratios of positive to negative documents. If they are similar than $b$ would be 0. But what I don't get is that if you calculate $r$ you see that this bias exists in every-word, so taking the dot product you would also sum it n times (where n is the length of the document), but in the final equation you just correct for it once.

What am I missing?

Answer 1

Ok, so the math works out fine.

Here is the explanation:

Let's define $A$ as the event of getting the specific document (movie review), and $B$ as the event that it is a positive review. According to Bayes theorem we get for positive docs:

$$ P(B \mid A) = \frac{P(A\mid B)P(B)}{P(A)} $$

And similarly for negative docs:

$$ P(B^c \mid A) = \frac{P(A \mid B^c) P(B^c)}{P(A)} $$

Now we are not interested in $P(A)$, our aim is to categorize which is more likely - that the document is positive or negative. So instead of calculating each of the probabilities and seeing which is bigger, we could instead take the log of the ratios, and see if it's above (positive probability is bigger) or below (negative probability is bigger) $0$.

We get:

$$ \log(\frac{P(B \mid A)}{P(B^c \mid A)}) = \log(\frac{P(A\mid B)P(B)}{P(A \mid B^c) P(B^c)}) = \log(\frac{P(A\mid B)}{P(A \mid B^c)}) + \log(\frac{P(B)}{P(B^c)}) $$ The 1st term is the $X \cdot r$, and the 2nd term is the $b$ in the question. The first term will take advantage of the "Naive" assumption that the probability of a document to occur given that it is positive/negative is equal to the multiplication of the probabilities of each word to occur.

$$ \log(\frac{P(A\mid B)}{P(A \mid B^c)}) = \log(\frac{P(a_1\mid B)P(a_2\mid B)...P(a_n \mid B)}{P(a_1\mid B^c)P(a_2\mid B^c)...P(a_n \mid B^c)}) \\ = \log(\frac{P(a_1\mid B)}{P(a_1\mid B^c)}) + \log(\frac{P(a_2\mid B)}{P(a_2\mid B^c)}) + ... + \log(\frac{P(a_n \mid B)}{P(a_n \mid B^c)}) $$

$r$ in itself is the probability ratio for all words in the vocabulary, and once you take the dot product with a specific document (row) in the frequency matrix, you get the 1st term.