My dataset is severely imbalanced, negative class is the majority and its records is almost 10000 larger than the positive class records. I'm using Python.
Here's what I have already done:
- I used stratified splitting to split the dataset into training and testing data in 0.75:0.25
- Then for training data, I tried oversampling, combined sampling and undersampling methods from imblearn. All these sampling methods tend to provide balanced dataset, and they all generate synthetic data.
- After sampling, I evaluated each fold of cross validation in training data. Cross validation is stratified 10 fold. In each fold, I checked precision, recall, specificity and average precision score for both original data as well as synthetic data. The results in original data set and synthetic data set are all satisfied.
- However, when I applied the trained model on testing data, I got very very lower precision and average precision score, because there are huge amount of false positive. Recall and specificity were ok.
Currently, I haven't used feature selection, param tuning or feature engineering yet. But based on other experiments I did on this dataset before, after feature selection, the results won't improve too much.
So my question is, what can I do so that my testing results can improve a lot?
- Should I still use sampling method on training data and still make the classes almost balanced, later rely on feature engineering, feature selection and param tuning?
- Previously, I tried to use class weighting in the estimator instead of using data sampling, but got very low performance results.
- Should I apply sampling on the whole original dataset, which means it will include testing data. Although the right way to do sampling is just to apply it on training data, my cross validation results in each fold of sampled training data are showing good results for both original data and synthetic data, so if my testing data is also included in the sampling, later I just use the performance results on original testing data also looks good.
- Or any other better options?