How can we adjust the loss to incorporate beliefs about the test set?

Let's say we've some training data points (not a huge number), collectively referred to as $$X$$, and we're trying to fit a curve (a polynomial) to these points. Later, we check training and test set errors.

We know for a fact that higher degree polynomials are bound to overfit the data, and relatively lower degree polynomials might give smaller errors on the test set.

Mathematically, though, a (lower degree) polynomial is just a subset of any polynomial with comparatively greater degree. So, it should be possible (is there some way, rather?) for a higher degree polynomial that overfits the training set, and gives wild results on the test set, to perform at least as good as lower degree polynomial that does not overfit (since that's just a subset)

So, how can the loss term be adjusted to incorporate the test set? Ideally, we're not supposed to see the test set until we're done training, so I'm in bit of a confusion here!

Thanks :)

• Regularization? Jun 11 '20 at 5:33
• @user2974951 It's not obvious to me how regularisation brings us closer to the lower degree solution Jun 11 '20 at 5:50
• By penalizing more complex models. Jun 11 '20 at 10:28