My dataset is severely imbalanced, negative class is the majority and its records is almost 10000 larger than the positive class records. I'm using Python.

Here's what I have already done:

  1. I used stratified splitting to split the dataset into training and testing data in 0.75:0.25
  2. Then for training data, I tried oversampling, combined sampling and undersampling methods from imblearn. All these sampling methods tend to provide balanced dataset, and they all generate synthetic data.
  3. After sampling, I evaluated each fold of cross validation in training data. Cross validation is stratified 10 fold. In each fold, I checked precision, recall, specificity and average precision score for both original data as well as synthetic data. The results in original data set and synthetic data set are all satisfied.
  4. However, when I applied the trained model on testing data, I got very very lower precision and average precision score, because there are huge amount of false positive. Recall and specificity were ok.

Currently, I haven't used feature selection, param tuning or feature engineering yet. But based on other experiments I did on this dataset before, after feature selection, the results won't improve too much.

So my question is, what can I do so that my testing results can improve a lot?

  1. Should I still use sampling method on training data and still make the classes almost balanced, later rely on feature engineering, feature selection and param tuning?
    • Previously, I tried to use class weighting in the estimator instead of using data sampling, but got very low performance results.
  2. Should I apply sampling on the whole original dataset, which means it will include testing data. Although the right way to do sampling is just to apply it on training data, my cross validation results in each fold of sampled training data are showing good results for both original data and synthetic data, so if my testing data is also included in the sampling, later I just use the performance results on original testing data also looks good.
  3. Or any other better options?
  • 2
    $\begingroup$ If your data set is that unbalanced you may have to consider a different approach, for ex. anomaly detection. With that degree of imbalance it will be very hard to get any useful results out since your models will be heavily biased. $\endgroup$ – user2974951 Oct 10 '18 at 7:43
  • $\begingroup$ Yeah, previously I suggested to try clustering, but I have been required to use classification... $\endgroup$ – Cherry Wu Oct 10 '18 at 16:37
  • $\begingroup$ If possible then, I would severely undersample the majority class (making sure you do not lose too much information), and then either 1) upsample the minority class, 2) use weights to "balance" the data set. There are some algorithms that can do that relatively well, for ex. random forests. $\endgroup$ – user2974951 Oct 11 '18 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.