Highly correlated features in text mining despite mutual information criterion

Question

I'm trying to classify documents into two classes using the Bernoulli Naive Bayes algorithm, as described here in chapter 13.

I've extracted 500 tokens (out of more than 30,000) from my sample documents (about 500) using the mutual information criterion as described in the same chapter.

I calculate the conditional probabilities for all tokens as the empirical frequency of documents in class1 or class2 in my sample to contain this token. I also use the Laplace (+1) smoothing, as suggested in the book chapter. So if in class1, 3 documents contain the token "china" and one does not, then the conditional probability of token "china" being in class1 is:

P(china|class1) = (3 + 1) / (3 + 1 + 2) = 2/3 = 0.6667.

(where the 1 in the numerator is the +1 smoothing and the 2 in the denominator comes from there being 2 classes.)

When I plot these, they look like this:

The two series have a correlation of 0.89. So this means that if a document in class1 contains, say, the token "china", then it is also likely that a document in class2 contains the token "china".

This worries me, as I thought the information criterion would make sure tokens would show up more in one of these classes than in the other.

So my question is:

1. has anybody encountered a similar situation?
2. is this plausible or does this look wrong?

Thank you very much.

[This is my output from a relatively lengthy and idiosyncratic analysis, so I don't have a minimum working code example right now, but I hope you can follow my verbal explanation here. Thank you very much for any ideas.]

Answer 1

Ok, I think I know what's going on. I'll first give my tentative explanation and then answer @Wabbits and @Greg Ver Steeg's comments.

Answer

Even after picking tokens that rank highly using the mutual information criterion we still have a lot of tokens in there that appear relatively often in both classes of documents. They all have conditional probabilities of being in either class more than 0.78. That's mainly because they appear many times, all of them more than in more than 400 out of our about 500 training documents.

So apparently the information criterion was still high for them, so that we included them in the top ranked 500 tokens.

Here is a matrix of scatterplots of stats on these tokens against each other:

On the top left (1,1) we have the scatterplot from above that irritated me with the high correlation of the conditional probabilities of 0.88. We can see that the high conditional probabilities are particulary high for tokens that simply appear more often (see the right column, or subplots (1,3) and (2,3)).

But it's interesting to see that they are not necessarily the documents with the high mutual information criterion values (subplots (1,2), (2,2) and (3,3)).

Overall I think things should be fine.

@Wabbit

• Yes, I've build everything myself
• Some classification statistics on the training sample:
  • true positive: 99
  • false positive: 52
  • true negative: 284
  • false negative: 48
  • Recall: 0.66
  • Precision: 0.66
  • AUC: 0.76
• Here are the then most highly ranked documents (with mutual information criterion):
  • 0,116 'automat'
  • 0,0967 'output'
  • 0,0849 'signal'
  • 0,083 'execut'
  • 0,0825 'inform'
  • 0,0782 'input'
  • 0,0766 'detect'
  • 0,0736 'user'
  • 0,0725 'display'
  • 0,07015 'sensor'
• Here are the number 490-500 in the ranked tokens:
  • 0,01108 'lamp'
  • 0,01104 'solid'
  • 0,01101 'concept'
  • 0,01101 'offer'
  • 0,0109 'declin'
  • 0,0109 'diagnosi'
  • 0,0109 'ethernet'
  • 0,0109 'flood'
  • 0,0109 'handheld'
  • 0,0109 'inappropri'
  • 0,0109 'markup'
• I'm not sure what you mean with "MI 2". I've been using the version of the mutual information criterion here on page 272 (there's the base 2 log in there, maybe that's where the name comes from).

@Greg Ver Steeg:

• The unit on the x-axis on my plot in the first question are tokens, not documents. And the same in the scatterplot on the right side.
• The way I've understood your comment is that you think I might classify a lot of documents on the unseen sample because some are longer and some are shorter. Or maybe I got it wrong?
• If that's your concern (classifying documents to easily with longer text), I think it's valid, but it wasn't the question here.

Answer 2

You can check whether this is correct comparing to other feature selection approaches. Also if you can use sklearn SGDClassifier (and others) there is classifier.coef_[0] property where you will find the coefficient for your model after it is fitted.

This is how you can extract feature names (words in your setting) from that

feature_names = vectorizer.get_feature_names()
coefs_with_fns = sorted(zip(classifier.coef_[0], feature_names))