0
$\begingroup$

I am working on a research-based assignment where I suppose to build a 3-class (bad, medium, good) classification using SVM. The dataset provided is imbalanced. The train:test splitting ratio is 75:25 with stratified method.

First Model - I did not oversample the data

Second Model - I oversample minority class using RandomOverSampler() in the the train set

Third Model - I oversample minority class using RandomOverSampler() in the original dataset, then only i split into train and test set.

Based on all 3 models' result, which model should be chosen (even if there is room for improvement for both 3 models) in terms of logicalness, correctness and also why?

3 classification model result evaluation

$\endgroup$
2
  • $\begingroup$ I think there might be room for improvement, but your third model is overfit and you should not consider it. $\endgroup$ – user2974951 Feb 12 '19 at 14:03
  • $\begingroup$ @user2974951 Thanks for you reply. Yes, of course improvement is needed. But at this stage, which model should be consider as the best one and why since you stated not to consider third model? $\endgroup$ – Edmund Feb 12 '19 at 14:07
0
$\begingroup$

Both your second and third models are actually overfit. That the second model is overfit you can see from the big difference in accuracies from the training and test set. Your third model is overfit because you applied upsampling incorrectly. You have to split the data into train / test at the very start. No information from the test set can be used in the training set. So at this moment your first model is best.

$\endgroup$
0
$\begingroup$

You can immediately discard the third approach without even fitting it, as it is conceptually wrong: by oversampling the test set, you are essentially cheating on the evaluation metrics, and leaking information from the training set. Moreover, the data you will get "in production" will be imbalanced anyways, so it clearly does not make sense to evaluate on a different distribution.

The question is then which to choose between the first two, and it is clear that the second is not as performant on the test set. However, as noted by others, the high gap between training and testing performance indicates that the second model is overfitting.

To sum up:

  • at present, use the first model
  • but you might be able to obtain better results with the second approach by tuning the hyper-parameters
  • forget the third approach
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.