1
$\begingroup$

This is hypothetical and I would like to hear what people do when the get to the test set and realize they've overfit. Of course, preventing overfitting in the first place is ideal.

You're working on a prediction problem and you split your data into a Train, Validation, and Training set. You're MSE on Validation is within a reasonable value of you're train MSE. However, when you get to your test set, the MSE is very poor. What do you do?

Some Ideas I have

  • Use a less complex model or regularize more
  • Make sure there are no common issues: data leakage, forward looking bias, etc.
  • Get more data

What else?

Say you still do all of this and the model still seems to be overfitting.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Re: "Say you still do all this..." those things are all hard to do, so I would be more concerned with whether I had done them correctly rather than looking for other things to try. $\endgroup$
    – John Madden
    Commented Jun 1 at 19:46
  • $\begingroup$ Okay, but in the case where you have done them correctly...what do you do? $\endgroup$
    – user2330624
    Commented Jun 2 at 14:22

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.