# Techniques to avoid overfitting

I have heard of several techniques to avoid overfitting:

• Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems difficult to me to choose, since we can have a set of parameter which led to better validation score even if the gap between training and validation set for this set of parameter can be bigger than for other set of parameters. When can we say that the gap is too big and then we are overfitting?

• Variance in the cross validation score, it is the same we want a "low" variance but how can we judge what is low?

If you have practical link (not just research) with real example I would be quite interested...

I know the question is not solved and quite general but I am sure that you could help me, even with partial answer, to gain a clearer view!

In the context of neural networks you could use dropout and regularization (L1 or L2).
• dropout is a rather simple technique - you simply don't use some the neurons' values in the forward pass (e.g. as you have set their weights to 0). This simulates a sparse network or multiple network architectures at the same time. Having more networks helps with the overfitting because it's like you have more models that know more (different stuff) about your data.