Naive Bayes: Imbalanced Dataset in Real-time Scenario

Question

I am using scikit-learn Multinomial Naive Bayes classifier for binary text classification (classifier tells me whether the document belongs to the category X or not). I use a balanced dataset to train my model and a balanced test set to test it and the results are very promising.

This classifer needs to run in real time and constantly analyze documents thrown at it randomly.

However, when I run my classifier in production, the number of false positives is very high and therefore I end up with a very low precision. The reason is simple: there are many more negative samples that the classifer encounters in the real-time scenario (around 90 % of the time) and this does not correspond to the ideal balanced dataset I used for testing and training.

Is there a way I can simulate this real-time case during training or are there any tricks that I can use (including pre-processing on the documents to see if they are suitable for the classifer)?

I was planning to train my classifier using an imbalanced dataset with the same proportions as I have in real-time case but I am afraid that might bias Naive Bayes towards the negative class and lose the recall I have on the positive class.

Any advice is appreciated.

Answer 1

To create a good model, the model has to be built on training data which is of the same "structure" as the data the model will applied later on. This is the one boring assumption which underlies all classification models.

So by using an balanced data set meanwhile the real world is not balanced, you have already introduced a bias. While there are cases where this is not a problem (imagine perfectly separable (non-linear) classes, a model built on a balanced data set containing all border-relevant points will be still working perfectly on a skewed sample), classifying documents is often a game of probabilities and hence class skew is more problematic.

My suggestions:

• Built the model on the imbalanced set with the same proportions as in production. If you have to sample for this, then perform multiple runs across different samples during validation to improve generalization power.
• The "bias" towards the negative class in an imbalanced set originates from the-best-guess-is-majority-class-if-everything-else-is-equal, something which Naive Bayes is sensitive to (especially when a lot of (irrelevant) features are involved). Use a different classifier which can capture inter-feature/word-dependencies to reduce this. I'd try Gradient Boosting with trees as described in chapter 10 "Boosting and Additive Trees" of The elements of statistical learning.
• You are currently using "plain precision / recall" as metric. Based on your productions requirements, estimate whether a false positive is equally bad as a false negative and adjust the metric accordingly.

Answer 2

In the paper "Tackling the Poor Assumptions of Naive Bayes Text Classifiers" the authors deal with this problem, among others, which stem from the character of the naive bayes algorithm. Having highly skewed data leads to a bias in your weights, which causes the bad precision.

Concretely for the problem of skew data, what they proposed what they call the complementary naive bayes algorithm, where to train each class they use all data, but the sample from that class. The idea is that they get more even training sets.

They idea you mean is usually called stratified sampling, which is also available in scikit-learn, and is worth a try.

In addition to sampling methods, a smart method is to normalize the word counts to correct for weight bias as explained here (Naive Bayes for Text Classification with Unbalanced Classes).

The idea is to make the estimated conditional probabilities insensitive to skewed counts. If you have too few documents of one class, and the are comparable in length to those of the other class, when words appear more often in documents of one class, Naive Bayes will tend to associate them with documents of other classes. By normalizing the word counts across classes, this bias is compensated for.

Good luck!

Answer 3

Any Bayesian classifier can be easily tweaked to incorporate knowledge about how often a particular class is expected. When you train a Bayesian classifier, two sets of parameters are learned:

• P(C=c), the probability that an observation belongs to class C (the class prior probabilities)

• P(F=f | C=c), the probability that an observation has the feature set F given that it belongs class C (i.e. its likelihood).

The classification rule is to choose a c that maximizes P(C=c)*P(F=f|C=c). (See: http://en.wikipedia.org/wiki/Naive_Bayes_classifier#Constructing_a_classifier_from_the_probability_model)

You can modify P(C=c) according to the expected occurrences of positive and negative observations in your production environment. Then your classification criterion will be optimal.

I wouldn't reduce the amount of positive observations in the training dataset. This will indeed change the prior probabilities to better match your test dataset. However, it will hurt the estimation of the likelihood parameters (since you won't use all available data). It's much better to use all available data and then modify the class prior probabilities according to your needs. When using discriminative classifiers such as SVM, the latter approach is less straightforward (since P(C=c) isn't explicitly modeled) and then the logic of keeping both the training and test datasets similarly (im)balanced makes sense.