Kind of a basic question I guess but I was never clear on that. Many Kaggle tutorials and online courses emphasize the importance of a proper cross validation strategy to evaluate model performance and ensure generalisation. However, very few of these tutorials provide an explanation on how to go about it, and most of the times the explanations are pretty standard.

I am trying to build a spam classifier but I am working with a small dataset. My training dataset has a size of 2600 rows and there is a high class imbalance, 2433 negative observations and 216 positive observations.My approach to deal with this is:

  1. I am trying to find an appropriate oversampling technique using the entire dataset to do cross validation with the StratifiedKFold iterator with 10 folds and shuffling.
  2. After I find which oversampling technique works, I split the training dataset into train-test splits (80/20)
  3. Train an Random Forest model on the train dataset, doing the necessary hyperparameter tuning (GridSearchCV with StratifiedKFold) and using the test dataset for model selection.
  4. When I am done with training I use a random sample that I have kept aside as a validation set and I am doing the final performance evaluation on this dataset.

My question is, is this considered to be a good way to do things or there is a better, intelligent way to go about this?

  1. Split data into train-validation-test
  2. Oversample minority class(es) in the train set.

Note: given that your minority class only represents around 8 % of the total, oversampling may not work well (if at all). You may have to get more creative here.

  1. Tune your model hyperparameters on the validation set
  2. After selecting the best models test them on the test set
| cite | improve this answer | |
  • $\begingroup$ Regarding oversampling, the best approach I have found is to use a combination of over and under sampling, such as SMOTE with Tomek links or Edited Nearest Neighbors. Also that becomes more effective if you do oversampling within cross validation, and not in the entire dataset. There is a good paper about that: researchgate.net/publication/… $\endgroup$ – Dr.Fykos Aug 7 '19 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.