I have a dataset with over 1 million observations, and a few dozens of predictors. My target variable is binary (gold customer vs. not gold customer) and I wish to build a classifier for prediction. Out of the 1 million observations, only around 30,000 are gold customers. I read the file into a couple of computer software packages that struggled with the size. I tried making a prediction, using decision trees (CART), but got a poor result with specificity nearly perfect and sensitivity of less than 30%. I calculated the effect size (cohen's D) for each variable, and 3 variables showed higher values of D. I got the median difference for them, and it was higher than other variables too. Then I took all the gold customers and a random sample of 30,000 non gold customers, and the decision tree gave much better results, with both sensitivity and specificity being over 80%. I tried it on a different random sample, and same again. The results I am mentioning are the validation results, not the training. Can you please help me understand what it means ? Is it possible that a very unbalanced data creates a problem that damages prediction? Thanks!
1 Answer
Imbalanced datasets indeed might damage the predictions. The reason is that when the dataset is imbalance the majority rule gets high accuracy and learner are biased toward it and perform badly on the minority class. You might be interested in this Editorial: Special Issue on Learning from Imbalanced Data Sets and Learning from Imbalanced Data
You have 30K samples of the minority class (3%) so are dealing with a relative imbalance and not absolute imbalance. These are good new since in such scenarios you are likely to end up with enough data after downsampling.
Your intuition of balancing the set was correct - in order to differ between the majority and the minority classes. However, your model is needed to perform well not in differing between them in a balance distribution but on their natural distribution. You can adapt your model back to the natural distribution by learning a new model that will do this adaptation. For details see here.
For the usage of different datasets in order to learn and validate see here
