Modelling with Unbalanced dataset

Question

I am working with a fairly unbalanced dataset (event class < 5% - it's a binary classification problem). To deal with this imbalance, I am trying out various techniques such as Oversampling the minority class (as well as synthetically generating samples using SMOTE), Under-sampling the minority class etc.

The problem that I am facing is, there is really no linkage between the performance measure than I get on my training sample (synthetically balanced) vis-a-vis what I get on my test sample. I am aware that oversampling can lead to inflated measures of performance, but the difference is stark (I am looking at Kappa statistic - On my training sample, I am getting a value in the range of 0.6-0.7, whereas on the test set it drops to less than 0.1). So my questions are:

a) Is there something than I can do to reduce this deviance between train and test perf. measures? I have been trying out different "ratios" (event/non-event ratio) while sampling, but this hasn't really helped.

b) Given this huge difference, is oversampling even a valid technique to pursue, in this particular case?

c) Finally, I had also tried the other recommended approach for dealing with unbalanced classes (by providing class weights, where the algorithm permits this functionality). This reduces the difference between train/test sets, but then the performance measures are not good for both of these sets! And hence, I was trying out the sampling techniques.

Answer 1

In general, artificially biasing the sampling of parts of the train set (either directly, by weighting, or by synthesis), is problematic.

Virtually every classifier also learns the a-priori label distribution. In Naive Bayes this can be seen very directly. In other techniques (e.g., logistic regression, classification trees, etc.), this appears too indirectly.

---

The cases where oversampling/overweighting makes sense, are different:

The Wikipedia entry, for example, states

The usual reason for oversampling is to correct for a bias in the original dataset. One scenario where it is useful is when training a classifier using labelled training data from a biased source, since labelled training data is valuable but often comes from un-representative sources.

and sites as an example a sample where ~67% are male, whereas they are known to compose ~50% of the population.

A different case where overweighting (possibly by oversampling) is when the missclassification penalties are unequal.

---

Two things to note when dealing with highly imbalanced data:

1. When tuning parameters (using the train set), it might be important to use stratified cross validation.

2. It pays to ensure that the train set and test set are prepared by a stratified split as well. For highly-imbalanced data, you might also need to make the test set larger than you usually do, in order to reduce test noise.