Early stopping means stopping gradient descent when the validation error starts to increase.
This is commonly used for neural networks, but can also be used for any model trained by gradient descent, such as a high degree polynomial.
Usually when we regularise, we put some kind of penalty on the parameters, and then look for the optimum of this modified cost function. We think of this as balancing goodness of fit, and high variance, which is controlled by the original cost function, against model simplicity, and high bias, which is controlled by the penalty.
But this isn't what we do with early stopping. We're not seeking the optimum of anything -- by definition we are stopping only "part way" to the optimum of the cost function. And we don't seem to have made the model any simpler; we initialise the model randomly, so we can end up anywhere in the parameter space, not in some restricted "simpler" region. Why then is this a good choice of parameters for reducing overfitting? How, intuitively, should we conceptualise what early stopping is doing?