I want to do classification with 2 classes. When I classify without smote I get:
Precision Recall f-1
0,640950987 0,815410434 0,714925374
When I use smote: (oversample the minority class at 200% and k = 5)
Precision Recall f-1
0,831024643 0,783434343 0,804894232
As you can see this works well.
However, when I test this trained model on validation data (which hasn't got any synthetic data)
Precision Recall f-1
0,644335755 0,799044453 0,709791138
which is awful. I used a random decision forest to classify.
Has anyone got any idea why this is happening? Any useful tips regarding extra tests I can try to get more insight are welcome too.
More info: I do not touch the majority class. I work in Python with scikit-learn and this algorithm for smote.
The confusion matrix on the test data (which has synthetic data):
The confusion matrix on the validation data with the same model (real data, which was not generated by SMOTE)
Edit: I read that the problem possibly lies in the fact that Tomek Links were created. Therefore I wrote some code to remove the tomek links. Though this doesn't improve the classification scores.
Edit2: I read that the problem possibly lies in the fact there is too much of an overlap. A solution for this is a more intelligent synthetic sample generation algorithm. Therefore I implemented
ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning
My implementation can be found here. It performed worse than smote.