I am confused by the very notion of epochs in neural networks (as well as number of trees in gradient boosting).
Gradient descent method (as most optimization algorithms) keep going until the loss function is "stable", i.e. not changing (within some tolerance) for a certain number of steps.
tolerance and the number of steps in which the loss function is stable after which stop iterating are indeed what i would call external parameters, but why the number of passes of the dataset (a.k.a. epochs) or the number of boosted trees should be fixed a a priori?
My feeling is that the training should just keep going until convergence (in a global or local minimum of the loss function). Where am I wrong?
This question came to me when dealing with early stopping, where you actually stop the training before convergence when a metric computed out-of-sample has reached a stationary point. And this is clear to me, since the training is optimizing in-sample, but you want to stop before to avoid overfitting. But why you need to specify a number of epochs before training is obscure to me.