I'm reading "Building Machine Learning Systems with Python" by Willi Richert and Luis Pedro Coelho and I got into a chapter concerning sentiment analysis. There is a whole example about classifying a tweet using Naive Bayes method.


How to calculate the probability of features $F_1$ and $F_2$


Assuming that the data set is as follows (content of the tweet / class):

  1. Awesome (positive)
  2. Awesome (positive)
  3. Awesome crazy (positive)
  4. Crazy (positive)
  5. Crazy (negative)
  6. Crazy (negative)

And introducing the following variables:

  • $C = \{pos, neg \}$ Class of the tweet
  • $F_1$ = Counting the occurence of awsome in the tweet
  • $F_2$ = Counting the occurence of crazy in the tweet


The target is to calculate (or estimate)

$$ P(C|F_1,F_2) = \frac {P(C) \cdot P(F_1,F_2|C)}{P(F_1,F_2)} $$

That can be also expressed as:

$$ posterior = \frac {prior \cdot likelihood} {evidence} $$



P(C) is the prior probability of class C without knowing about the data. It's value is as follows: $$ P(C = "pos") = \frac {4}{6} = 0.67 $$ $$ P(C = "neg") = \frac {2}{6} = 0.33 $$


Knowing the fact that the features ane naive we can also calculate $P(F_1,F_2|C)$ using the formula:

$$ P(F_1,F_2|C) = P(F_1|C) \cdot P(F_2|C) $$

So the target is now:

$$ P(C|F_1,F_2) = \frac {P(C) \cdot P(F_1|C) \cdot P(F_2|C)} {P(F_1,F_2)} $$

In this particular problem: $$ P(C="pos"|F_1,F_2) = \frac {P(C="pos") \cdot P(F_1|C="pos") \cdot P(F_2|C="pos")}{P(F_1,F_2} $$ $$ P(C="neg"|F_1,F_2) = \frac {P(C="neg") \cdot P(F_1|C="neg") \cdot P(F_2|C="neg")}{P(F_1,F_2} $$

We can now calculate likelihoods: $$ P(F_1=1|C="pos") = \frac{3}{4} = 0.75 $$

$$ P(F_2=1|C="pos") = \frac{2}{4} = 0.5 $$

$$ P(F_1=1|C="neg") = \frac{0}{2} = 0 $$

$$ P(F_2=1|C="neg") = \frac{2}{2} = 1 $$


In the book it is written that the evidences can be retrieved by calculating the fraction of all training data instances having particular feature value.

The formula is as follows:

$$ P(F_1,F_2) = P(F_1,F_2|C="pos") \cdot P(C="pos") + P(F_1,F_2|C="neg") \cdot P(C="neg") $$

Which leads to the following results: $$ P(F_1=1,F_2=1) = \frac {1}{3} \cdot \frac{4}{6} + 0 \cdot \frac{2}{6} = 0.22 $$

$$ P(F_1=1,F_2=0) = \frac {2}{3} \cdot \frac{4}{6} + 0 \cdot \frac{2}{6} = 0.44 $$

$$ P(F_1=0,F_2=1) = 0 \cdot \frac{4}{6} + 1 \cdot \frac{2}{6} = 0.33 $$

$$ P(F_1=0,F_2=0) = 0 $$

Question: How the four values above are obtained?


1 Answer 1


It seems you found an errata on the book. I did the calculations by hand and my results were quite different. Considering this same example has already an errata reported in the editor's site (wrong value for $P(F_2=1|C="pos")$), these strange values in the final result aren't very surprising.

I'll write down the numbers I found (I'll assume you know how a achieved to them, by replacing the terms of your last formula). In the case something is not clear, just tell me and I can edit the answer and add some clarifications). Here the numbers:

$$ P(F_1=1,F_2=1) = \frac {3}{8} \cdot \frac{4}{6} + 0 \cdot \frac{2}{6} = 0.25 $$

$$ P(F_1=1,F_2=0) = \frac {3}{8} \cdot \frac{4}{6} + 0 \cdot \frac{2}{6} = 0.25 $$

$$ P(F_1=0,F_2=1) = \frac{1}{8} \cdot \frac{4}{6} + 1 \cdot \frac{2}{6} = 0.42 $$

$$ P(F_1=0,F_2=0) = \frac{1}{8} \cdot \frac{4}{6} + 1 \cdot 0 = 0.08 $$

It's hard to tell exactly what the author might have done wrong to achieve the values given in the book, but I suspect he didn't consider the "naïve" assumptions. The probability $P(F_1=0,F_2=0)$ would indeed be zero if they didn't exist. I didn't check though to see if this hypothesis is the right. It's possible also that the results are wrong just because they used incorrect values in previous steps, as the the one mentioned in the linked errata.

I hope the mystery is clarified. Drop a comment if you need some more assistance.

  • $\begingroup$ Thanks for reply. I still cannot understand how do you obtain those values. In my opinion the first (the others are changed consequently) equation should be $P(F_1=1, F_2=1) = \frac {1}{4} \cdot \frac{4}{6} + 0 \cdot \frac {2}{6} = 0.16 $ I undestand it accordingly: #tweets with both awesome and crazy among all positives $\cdot P(C="pos")$ + #tweets with both awesome and crazy among all negatives $\cdot P(C="neg")$ $\endgroup$
    – Khozzy
    Sep 16, 2014 at 21:26
  • 1
    $\begingroup$ The equation you need to use to calculate $P(F_1, F_2|C)$ is $P(F_1,F_2|C) = P(F_1|C) \cdot P(F_2|C)$. So for example, $P(F_1=1, F_2=1|C="pos") = P(F_1=1|C="pos") \cdot P(F_2=1|C="pos")$, which gives us $\frac{3}{4} \cdot \frac{2}{4} = \frac{3}{8}$, not $\frac{1}{4}$ as you said. All other terms are calculated exactly the same way. $\endgroup$ Sep 17, 2014 at 1:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.