0
$\begingroup$

Let's say we've some training data points (not a huge number), collectively referred to as $X$, and we're trying to fit a curve (a polynomial) to these points. Later, we check training and test set errors.

We know for a fact that higher degree polynomials are bound to overfit the data, and relatively lower degree polynomials might give smaller errors on the test set.

Mathematically, though, a (lower degree) polynomial is just a subset of any polynomial with comparatively greater degree. So, it should be possible (is there some way, rather?) for a higher degree polynomial that overfits the training set, and gives wild results on the test set, to perform at least as good as lower degree polynomial that does not overfit (since that's just a subset)

So, how can the loss term be adjusted to incorporate the test set? Ideally, we're not supposed to see the test set until we're done training, so I'm in bit of a confusion here!

Thanks :)

$\endgroup$
3
  • $\begingroup$ Regularization? $\endgroup$ Jun 11 '20 at 5:33
  • $\begingroup$ @user2974951 It's not obvious to me how regularisation brings us closer to the lower degree solution $\endgroup$ Jun 11 '20 at 5:50
  • $\begingroup$ By penalizing more complex models. $\endgroup$ Jun 11 '20 at 10:28
1
$\begingroup$

Why not just train your model on the test set? You won’t, because this is a bad idea, but for exactly the same reason you shouldn’t “correct” for test set. Test set is ought to be “unseen” data, that is used as a proxy to judge potential performance on future data. If you cheat on test set, you’d know nothing about performance of your model on future data. In such cases you cannot say if your data works well, or just overfits the test set.

Another case is when your test set is very different from train set. Both datasets should come from same population, and it should be same population as where you’d be making predictions for. If that’s not the case, you should find better data, or prepare it better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.