I'm trying to classify documents into two classes using the Bernoulli Naive Bayes algorithm, as described here in chapter 13.
I've extracted 500 tokens (out of more than 30,000) from my sample documents (about 500) using the mutual information criterion as described in the same chapter.
I calculate the conditional probabilities for all tokens as the empirical frequency of documents in class1 or class2 in my sample to contain this token. I also use the Laplace (+1) smoothing, as suggested in the book chapter. So if in class1, 3 documents contain the token "china" and one does not, then the conditional probability of token "china" being in class1 is:
P(china|class1) = (3 + 1) / (3 + 1 + 2) = 2/3 = 0.6667.
(where the 1 in the numerator is the +1 smoothing and the 2 in the denominator comes from there being 2 classes.)
When I plot these, they look like this:
The two series have a correlation of 0.89. So this means that if a document in class1 contains, say, the token "china", then it is also likely that a document in class2 contains the token "china".
This worries me, as I thought the information criterion would make sure tokens would show up more in one of these classes than in the other.
So my question is:
- has anybody encountered a similar situation?
- is this plausible or does this look wrong?
Thank you very much.
[This is my output from a relatively lengthy and idiosyncratic analysis, so I don't have a minimum working code example right now, but I hope you can follow my verbal explanation here. Thank you very much for any ideas.]