Train classifier on balanced dataset and apply on imbalanced dataset?

Question

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works best with F-Score ~0.83.

Now, I have to apply the trained classifier on an unlabelled dataset DS2 with ~ 5 million entries (and same features). However, for DS2, the target distribution is expected to be highly unbalanced.

Is this a problem? Will the model reproduce the trained target distribution from DS1 when applied on DS2?

If yes, would another algorithm be more robust?

All the best, Greg

Answer 1

From my understanding, the balance of the unlabeled dataset is not important and should have no effect; provided the training dataset is representative of "real world data" (i.e. the unlabelled data).

What is important is how accurate your classifier is. So, if your classifier has learned meaningful patterns during training, then that should be confirmed by high accuracy when you apply your classifier to your unlabelled dataset.

If you find that the accuracy is lower than expected, then I suggest that:

1. Your classifier has been overfit to the training data. (i.e. the classifier is reporting high accuracy on the training set because it's finely tuned to fit the training set.)
2. That the data used for training is not representative of the data in your unlabelled dataset. (i.e. the patterns learned during training cannot be applied well to the unlabeled dataset.)

Answer 2

If your training set is balanced, but your testing set is imbalanced, you will be fine using a regular model.

However, since you have an imbalanced test set you might care more about correctly labeling samples from the minority class. If you find this to be the case after your implementation, then you might want to try some approaches specific to imbalanced datasets.