I have heard of several techniques to avoid overfitting:
Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems difficult to me to choose, since we can have a set of parameter which led to better validation score even if the gap between training and validation set for this set of parameter can be bigger than for other set of parameters. When can we say that the gap is too big and then we are overfitting?
Variance in the cross validation score, it is the same we want a "low" variance but how can we judge what is low?
If you have practical link (not just research) with real example I would be quite interested...
I know the question is not solved and quite general but I am sure that you could help me, even with partial answer, to gain a clearer view!