Strangely imbalanced dataset

Question

I'm new to Machine Learning and this forum. I have a beginner's doubt regarding imbalanced dataset. Here it goes: I have a binary classification task, where I'm more interested in accurately classifying the positive class (which is in minority in the target population). Unlike the common problem of not having enough positive (minority) class instances in the training set, my training dataset contains the positive class in majority.

Here's the target population composition (which I'd expect to find in the environment where my classifier/model would be deployed):

• Positive Class: ~35%
• Negative Class: ~65%

Here's my Training Set Composition:

• Positive Class: ~95%
• Negative Class: ~5%

As my training set composition drastically differs from the target population composition, will the classification algorithm fail to generalize when classifying instances from the target population? As I mentioned earlier, I'm more interested in accurately classifying the positive class instances, which my training set has in abundance.

I read the following description in a publication on imbalanced datasets: "The purpose of machine learning is for the classifier to estimate the probability distribution of the target population. Since that distribution is unknown we try to estimate the population distribution using a sample distribution. Statistics tells us that as long as the sample is drawn randomly, the sample distribution can be used to estimate the population distribution from where it was drawn. Hence, by learning the sample distribution we can learn to approximate the target distribution."

Since my training dataset can not be considered as a random sample of the target distribution, will this affect the generalization power of my classifier? If so, what shall be done to avoid this? Over/under-sampling? Cost Matrices?

PS: I tried to search previous posts about issues similar to mine, but all of them dealt with the problem of not having sufficient examples for the minority class (which is the exact opposite of my scenario).

Thanks in advance -S

Answer 1

There are several components to your question - but first I would ask why is your sample so skewed? You have an under-sampled training set which as you point out is odd. Can you assume that the two classes were sampled randomly from the population? If not, that is your most serious problem and potentially not something you can recover from. The best you can do is build a model, calibrate it and then test it in a pilot on the population.

Assuming representative samples, the issues are:

1) Will this imbalance keep the classifier from properly discriminating between classes? Maybe. You must cross validate any resulting model so this should be testable and you may find you need over sample the negative cases to get the data set into balance. It depends on the type of classifier being used and the data. If you are using random forests or GBM I might not be concerned. If you are using a single decision tree, i would.

2) Will the predicted probabilities from the model align to the population. The answer is no. If this is important to your application (i.e. the model must be well calibrated and not just concerned with ranking or separating the classes) it is a problem but can be overcome. Any time a training data set is used where the class density does not match the population, the resulting probabilities of class membership will be biased. Here is a general purpose way to re-calibrate them:

LINK

Answer 2

I am also dealing with imbalanced dataset. I try to say what I know about this.

--will the classification algorithm fail to generalize when classifying instances from the target population? As I mentioned earlier, I'm more interested in accurately classifying the positive class instances, which my training set has in abundance.

In general such difference in distributions has negative influence on the classifier's generalization ability/performance. But it also depends on the particular classifier used. You can try to run experiments to see how the performance goes. In some case, it might not be that bad.

For the most naive classifier - the majority rule, that is we predict every item as the majority class in the training set. Then of course, in your case the test error rate will be 65%. For probability classifiers that estimate conditional distribution P(y|x), the performance might still be good, as long as the estimation of P(y|x) is good. Say for text classification, it might be the case that the words that appears in the two classes are drastically different, and the within-class distribution P(x|y) of the words stays the same in the training and test set, the classifier might still perform good, even though P(y) changed a lot. And similarly for discriminative classifiers like linear classifers as SVM. But in general, the difference in training set and test set might have some negative influence on classifer's performance. And you might need to note that the difference seems to be the class distribution p(y), but the p(x|y) might be the same.

what do you mean by accurately classifying the positive class instances? Do you want high recall of the positive class? With positive class dominating the training set, the classifer might favor the positive class, so yielding high recall of positive class, but might have low precision for positive class, since it might also get lots of negative instances into positive class (false positive).

Answer 3

One thing you can try out quickly is using SVMLight [1] in a transductive mode [2]. You can go through this lightning quick presentation slides on TSVM by Carlos Guestrin [3]. You can of course read up on TSVM from Joachim's paper linked in Joachim's website.

[1] http://svmlight.joachims.org/
[2] The transductive mode allows you to specify the fraction of true positive examples in your test set. This is usually done by using the "-p" option. Note that you will be using both the train and test set (without labels of course) for learning a classifier here.
[3] http://www.cs.cmu.edu/~guestrin/Class/10701-S06/Slides/tsvms-pca.pdf

Answer 4

Is there no way to get a more realistic sample?

The problem with having such a large positive class is that the model will achieve 95% accuracy on the training set if it just bags all records as positive. The problem with your data collection not being random is that it might show bias towards certain variables (and it often will) depending on the process used. You need to validate that this is not happening.

What method are you using? Many support weights to help rather than undersampling to even out the class size. Undersampling is likely impractical for your case since the negative class is so small (though if n>>m, it might be feasible). Oversampling is, IMO, a dangerous game when the sample data-set is so uncharacteristic of the population.

I've been using rfe from 'caret' in R, but it doesn't support weights (as far as I can tell). In situations like this, I like to even out my sample size to build the model, and then validate against a test set that is truly random sampled and has a 'correct' proportion. I'd have to be very confident in my data collection methods, however.

EDIT: @B_Miner hit the nail on the head-- the unrepresentative/nonrandom sample is a much bigger issue than the uneven classes. +1 to him, wish I could comment.