Which classification model should I choose and Why?

Question

I am working on a research-based assignment where I suppose to build a 3-class (bad, medium, good) classification using SVM. The dataset provided is imbalanced. The train:test splitting ratio is 75:25 with stratified method.

First Model - I did not oversample the data

Second Model - I oversample minority class using RandomOverSampler() in the the train set

Third Model - I oversample minority class using RandomOverSampler() in the original dataset, then only i split into train and test set.

Based on all 3 models' result, which model should be chosen (even if there is room for improvement for both 3 models) in terms of logicalness, correctness and also why?

Answer 1

Both your second and third models are actually overfit. That the second model is overfit you can see from the big difference in accuracies from the training and test set. Your third model is overfit because you applied upsampling incorrectly. You have to split the data into train / test at the very start. No information from the test set can be used in the training set. So at this moment your first model is best.

Answer 2

You can immediately discard the third approach without even fitting it, as it is conceptually wrong: by oversampling the test set, you are essentially cheating on the evaluation metrics, and leaking information from the training set. Moreover, the data you will get "in production" will be imbalanced anyways, so it clearly does not make sense to evaluate on a different distribution.

The question is then which to choose between the first two, and it is clear that the second is not as performant on the test set. However, as noted by others, the high gap between training and testing performance indicates that the second model is overfitting.

To sum up:

• at present, use the first model
• but you might be able to obtain better results with the second approach by tuning the hyper-parameters
• forget the third approach