Kind of a basic question I guess but I was never clear on that. Many Kaggle tutorials and online courses emphasize the importance of a proper cross validation strategy to evaluate model performance and ensure generalisation. However, very few of these tutorials provide an explanation on how to go about it, and most of the times the explanations are pretty standard.
I am trying to build a spam classifier but I am working with a small dataset. My training dataset has a size of 2600 rows and there is a high class imbalance, 2433 negative observations and 216 positive observations.My approach to deal with this is:
- I am trying to find an appropriate oversampling technique using the entire dataset to do cross validation with the StratifiedKFold iterator with 10 folds and shuffling.
- After I find which oversampling technique works, I split the training dataset into train-test splits (80/20)
- Train an Random Forest model on the train dataset, doing the necessary hyperparameter tuning (GridSearchCV with StratifiedKFold) and using the test dataset for model selection.
- When I am done with training I use a random sample that I have kept aside as a validation set and I am doing the final performance evaluation on this dataset.
My question is, is this considered to be a good way to do things or there is a better, intelligent way to go about this?