8
$\begingroup$

I want to do classification with 2 classes. When I classify without smote I get:

Precision       Recall            f-1
0,640950987     0,815410434       0,714925374

When I use smote: (oversample the minority class at 200% and k = 5)

Precision        Recall           f-1
0,831024643      0,783434343      0,804894232

As you can see this works well.

However, when I test this trained model on validation data (which hasn't got any synthetic data)

Precision        Recall           f-1
0,644335755      0,799044453      0,709791138

which is awful. I used a random decision forest to classify.

Has anyone got any idea why this is happening? Any useful tips regarding extra tests I can try to get more insight are welcome too.

More info: I do not touch the majority class. I work in Python with scikit-learn and this algorithm for smote.

The confusion matrix on the test data (which has synthetic data): enter image description here

The confusion matrix on the validation data with the same model (real data, which was not generated by SMOTE)

enter image description here

Edit: I read that the problem possibly lies in the fact that Tomek Links were created. Therefore I wrote some code to remove the tomek links. Though this doesn't improve the classification scores.

Edit2: I read that the problem possibly lies in the fact there is too much of an overlap. A solution for this is a more intelligent synthetic sample generation algorithm. Therefore I implemented

ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning

My implementation can be found here. It performed worse than smote.

Improve this question
$\endgroup$
11
$\begingroup$

If you oversample the training data to change the relative class frequencies in the training set, you are implicitly telling the classifier to expect the validation set to have those same class frequencies. As the class frequencies influence the decision boundary, if the validation set class frequencies are different, then the classification boundary will be wrong as a result and performance will be sub-optimal.

Now it is true that for some classifiers, having imbalanced classes can cause it to perform badly, and re-sampling the data can help to correct this. However if you oversample the minority class too much, you will over-correct and performance will be suboptimal because the the difference in class frequencies between the training and validation sets is more than is needed to correct the imbalance problem.

My recipe then would be to use cross-validation to choose the degree of oversampling that is required to just compensate for the "class imbalance problem" experienced by your classifier, but no more. Note however this leads to an additional parameter that needs to be tuned, which leads to greater computational expense, and a higher likelihood of over-fitting in model selection (if the total number of tunable parameters is non-negligible).

Improve this answer
$\endgroup$
5
  • $\begingroup$ This extra parameter you talk about, is this the % I use ? (in my case 200%). I can't choose a smaller one, they have to be in steps of 100. Though if I understand correctly, if I make validationset which is balanced like the training set, it will give better results ? $\endgroup$ – Olivier_s_j Jun 13 '13 at 11:24
  • $\begingroup$ yes, the %age was indeed the parameter I meant; it is a pity if the software only allows this to be changed in steps of 100. Some machine learning algorithms (e.g. the SVM) allow you to weight positive and negative patterns differently (for the SVM have a different value of the parameter C for each class). This is directly equivalent to SMOTE in the case of the SVM, so that may be another way to get around the problem. $\endgroup$ – Dikran Marsupial Jun 13 '13 at 11:31
  • $\begingroup$ I tried the SVM solution. In sci-kit learn it is possible to set the weights inversely proportional to class frequencies (automatically). Though this gives very bad results ( for linear, polynomial, rbf and sigmoid kernel) $\endgroup$ – Olivier_s_j Jun 13 '13 at 11:45
  • $\begingroup$ If you just set them inversely proportional to the class frequencies that tells the SVM to act as if the class frequencies in the validation set are equal, which again is likely to over-compensate for the class balance problem. Try tuning the weights via cross-validation, it normally works for me. The other thing to do is to consider whether the misclassification costs actually are the same for false-positive and false-negative errors, and if not, include this in setting the C parameters for each class. $\endgroup$ – Dikran Marsupial Jun 13 '13 at 11:48
  • $\begingroup$ Hmmm I've tried tuning the weights manually though what the SVM predicts is or all 1s or all 0s. No in-between. (Btw I've added confusion matrices to the question for clarification, as you can see by using SMOTE, the unbalance is gone when using a test set which is also composed of synthetic data but there is still in unbalance when a test is run on the validationset.) $\endgroup$ – Olivier_s_j Jun 13 '13 at 12:12
2
$\begingroup$

Have you considered using the R implementation of random forest via Rpy? It has an option that allows you to choose both the sample size that each tree is trained on and the fraction falling into each class. I've found it to be the most effective way to deal with imbalance when using random forest.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.