I'm reading "Building Machine Learning Systems with Python" by Willi Richert and Luis Pedro Coelho and I got into a chapter concerning sentiment analysis. There is a whole example about classifying a tweet using Naive Bayes method.
Problem
How to calculate the probability of features $F_1$ and $F_2$
Data
Assuming that the data set is as follows (content of the tweet / class):
- Awesome (positive)
- Awesome (positive)
- Awesome crazy (positive)
- Crazy (positive)
- Crazy (negative)
- Crazy (negative)
And introducing the following variables:
- $C = \{pos, neg \}$ Class of the tweet
- $F_1$ = Counting the occurence of awsome in the tweet
- $F_2$ = Counting the occurence of crazy in the tweet
Target
The target is to calculate (or estimate)
$$ P(C|F_1,F_2) = \frac {P(C) \cdot P(F_1,F_2|C)}{P(F_1,F_2)} $$
That can be also expressed as:
$$ posterior = \frac {prior \cdot likelihood} {evidence} $$
Steps
Prior
P(C) is the prior probability of class C without knowing about the data. It's value is as follows: $$ P(C = "pos") = \frac {4}{6} = 0.67 $$ $$ P(C = "neg") = \frac {2}{6} = 0.33 $$
Likelihood
Knowing the fact that the features ane naive we can also calculate $P(F_1,F_2|C)$ using the formula:
$$ P(F_1,F_2|C) = P(F_1|C) \cdot P(F_2|C) $$
So the target is now:
$$ P(C|F_1,F_2) = \frac {P(C) \cdot P(F_1|C) \cdot P(F_2|C)} {P(F_1,F_2)} $$
In this particular problem: $$ P(C="pos"|F_1,F_2) = \frac {P(C="pos") \cdot P(F_1|C="pos") \cdot P(F_2|C="pos")}{P(F_1,F_2} $$ $$ P(C="neg"|F_1,F_2) = \frac {P(C="neg") \cdot P(F_1|C="neg") \cdot P(F_2|C="neg")}{P(F_1,F_2} $$
We can now calculate likelihoods: $$ P(F_1=1|C="pos") = \frac{3}{4} = 0.75 $$
$$ P(F_2=1|C="pos") = \frac{2}{4} = 0.5 $$
$$ P(F_1=1|C="neg") = \frac{0}{2} = 0 $$
$$ P(F_2=1|C="neg") = \frac{2}{2} = 1 $$
Evidence
In the book it is written that the evidences can be retrieved by calculating the fraction of all training data instances having particular feature value.
The formula is as follows:
$$ P(F_1,F_2) = P(F_1,F_2|C="pos") \cdot P(C="pos") + P(F_1,F_2|C="neg") \cdot P(C="neg") $$
Which leads to the following results: $$ P(F_1=1,F_2=1) = \frac {1}{3} \cdot \frac{4}{6} + 0 \cdot \frac{2}{6} = 0.22 $$
$$ P(F_1=1,F_2=0) = \frac {2}{3} \cdot \frac{4}{6} + 0 \cdot \frac{2}{6} = 0.44 $$
$$ P(F_1=0,F_2=1) = 0 \cdot \frac{4}{6} + 1 \cdot \frac{2}{6} = 0.33 $$
$$ P(F_1=0,F_2=0) = 0 $$
Question: How the four values above are obtained?