Techniques to avoid overfitting

Question

I have heard of several techniques to avoid overfitting:

• Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems difficult to me to choose, since we can have a set of parameter which led to better validation score even if the gap between training and validation set for this set of parameter can be bigger than for other set of parameters. When can we say that the gap is too big and then we are overfitting?

• Variance in the cross validation score, it is the same we want a "low" variance but how can we judge what is low?

If you have practical link (not just research) with real example I would be quite interested...

I know the question is not solved and quite general but I am sure that you could help me, even with partial answer, to gain a clearer view!

Answer 1

If avoiding overfitting were the only goal, one could simply choose a model that outputs a constant value (no training even needed!). That's a silly example, of course, but it illustrates that what we want is not just to avoid overfitting. Rather, we typically want to generalize as well as possible from the training data. This means we need to balance between overfitting and underfitting (the extreme example being the constant model). See the bias-variance tradeoff.

The way to do this in the context of holdout/cross validation is to select the model with the best performance on the validation set. As far as we can estimate from the data, this is the model that will perform best on unseen data drawn from the same distribution. As you mentioned, there may be a gap between the training and validation set performance, so some overfitting may be happening. But, selecting a simpler model that overfits less would result in even greater underfitting, reducing the overall performance.

Answer 2

In the context of neural networks you could use dropout and regularization (L1 or L2).

• dropout is a rather simple technique - you simply don't use some the neurons' values in the forward pass (e.g. as you have set their weights to 0). This simulates a sparse network or multiple network architectures at the same time. Having more networks helps with the overfitting because it's like you have more models that know more (different stuff) about your data.
• L1 and L2 regularisations are kind of harder to explain hopefully you'll find good resources

I, personally, found Andrew Ng's Deep learning specialisation very useful in this regard. I think the second course of the specialisation was dealing a lot with similar techniques.