1
$\begingroup$

I am a bit puzzled about the process of experimenting with a model and oversampling and then translating it to the final version of the model that will be used:

  1. I oversample the data (only the training dataset)

  2. Tune parameters and everything else

What do i do next to do it properly?

  1. Do take all the data and oversample it and then re-train the model
  2. Do i just take all the data i have and retrain the model without oversampling?
  3. Or do i just take the model that is trained on the oversampled training data as a 'final model'?

What is the correct way to do things? Thanks!

$\endgroup$
0
$\begingroup$

Great question - I don't think there's one correct way of doing things, but rather more of sensible strategies to get to your end goal. Here's a thought.

Ideally you want to maximize the predictive ability of your model and also train it to be as effective as possible in real life based on the data/input it will likely see. Therefore, one quick check you could do is see if the difference in performance between oversampled and full data is small.

If that's the case, then retrain without oversampling as it will give the model a chance to optimize on what it will likely see vs what you want it to see. If that's not the case, then you'll have to see how much added-value you truly get by training on oversampled data and find a way to try to penalize oversampling and find a trade-off.

$\endgroup$
  • 1
    $\begingroup$ Hmm, yes, this makes sense, however how exactly do i evaluate the performance if i have already used up all of my data to train it? Evaluating on the same data can be very misleading especially with class imbalances. Otherwise your idea does make sense. $\endgroup$ – Emil Filipov Aug 24 at 18:16
  • $\begingroup$ I agree that evaluating on the same data is not the best way to go. Perhaps - prior to training you can do some stratified sampling to create a balanced training and validation set, and do all the experiments on training and compare performance on validation set. $\endgroup$ – Samir Rachid Zaim Aug 24 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.