Early stopping means stopping gradient descent when the validation error starts to increase.

This is commonly used for neural networks, but can also be used for any model trained by gradient descent, such as a high degree polynomial.

Usually when we regularise, we put some kind of penalty on the parameters, and then look for the optimum of this modified cost function. We think of this as balancing goodness of fit, and high variance, which is controlled by the original cost function, against model simplicity, and high bias, which is controlled by the penalty.

But this isn't what we do with early stopping. We're not seeking the optimum of anything -- by definition we are stopping only "part way" to the optimum of the cost function. And we don't seem to have made the model any simpler; we initialise the model randomly, so we can end up anywhere in the parameter space, not in some restricted "simpler" region. Why then is this a good choice of parameters for reducing overfitting? How, intuitively, should we conceptualise what early stopping is doing?

  • $\begingroup$ Actually, the answer likely resides from my experience from my adventures in my Numerical Analysis course. Repetitive procedures either quickly converge with the right starting values, or run into a singularity and just crash. Hence, a rule of thumb. $\endgroup$ – AJKOER May 13 '20 at 22:01
  • $\begingroup$ An analogy, you are attempting to drive home in a thick fog. You know approximately how long it should take. Taking longer likely means you are lost. It is wiser to park then find the lake. $\endgroup$ – AJKOER May 13 '20 at 22:11

This is not a rigorous answer, but just to try to give some intuition behind the concept of early stopping.

We know that neural networks have a very high flexibility, which means that they can model extremely complex functions. The more complex a function gets, the more likely is that it will overfit the data.

Searching for that perfect minimum increases the complexity of the function. You start with a relatively smooth function and start tweaking every point of it until it gets you the best performance on the training data. Which means you have overfitted it.

Instead, you can stop early, to a simpler function. Think of it as something in between a smooth function and a highly complex function. It still gets the general shape you want, but didn't have time to fine tune all the small variations needed for a perfect error.

  • $\begingroup$ Thanks -- could you elaborate on in what sense a neural network is initialised to a "smooth" function? In the polynomial example, I'd have guessed that almost all randomly chosen initial parameters give "unsmooth" functions. $\endgroup$ – Denziloe May 13 '20 at 21:49
  • $\begingroup$ Perhaps I oversimplified my example for illustration purposes. Indeed, the function might not be smooth as we start, but if will definitely be "wrong" (in the sense that is different from our target). The core idea is to stop before the function has learnt the small intricacies of our dataset, before it overfits. This is again, based more empirical evidence than supported by a mathematical argument: we have simply observed that the generalization error decreases, then increases again when overfitting. $\endgroup$ – Paul92 May 13 '20 at 22:04
  • $\begingroup$ you initialise to small weights (normal around zero at least for sigmoid) where function is linear and small variance. you have to check exactly what is done for relu (which is nonlinear only around zero). $\endgroup$ – seanv507 May 13 '20 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.