0
$\begingroup$

I am confused by the very notion of epochs in neural networks (as well as number of trees in gradient boosting).

Gradient descent method (as most optimization algorithms) keep going until the loss function is "stable", i.e. not changing (within some tolerance) for a certain number of steps.

tolerance and the number of steps in which the loss function is stable after which stop iterating are indeed what i would call external parameters, but why the number of passes of the dataset (a.k.a. epochs) or the number of boosted trees should be fixed a a priori?

My feeling is that the training should just keep going until convergence (in a global or local minimum of the loss function). Where am I wrong?

This question came to me when dealing with early stopping, where you actually stop the training before convergence when a metric computed out-of-sample has reached a stationary point. And this is clear to me, since the training is optimizing in-sample, but you want to stop before to avoid overfitting. But why you need to specify a number of epochs before training is obscure to me.

$\endgroup$
1
$\begingroup$

Answering with a question: how would you know that the model has "converged"? Would you wait for test error equal to zero? What if it would be impossible? If the test error would not decrease for 10 epochs would it mean convergence? Or maybe 100? Or maybe 10000? An hour of training? A week? Or maybe a year? "Not decrease" means the difference equal to zero? Or 0.01 is acceptable? Or rather 1e-7?

We need some stopping rule and fixed number of epochs is the simplest one. With fixing the number of epochs you simply decide to wait as long as it is possible for you to wait. If it would find minimum faster, you waisted your time. If not and longer time to wait was unacceptable for you, then nonetheless, you'd have to stop. Nobody says it's the most optimal approach.

$\endgroup$
  • $\begingroup$ I don't understand your answer in 3 respects: 1 why you say "test error"? we are talking about training, so it is training error, that's all the point; 2 why you say error equal to 0? I just want to minimize the error function, whatever that minimum is, zero or not; 3 i see that fixing epochs is a stopping rule, but as far as i know in typical problems of optimization you don't fix that, you fix the tolerance and the number of non-increasing steps. You are saying that neural networks are actually not fitted, since convergence is almost never reached, don't you? $\endgroup$ – deltasun Apr 12 at 18:06
  • $\begingroup$ So would you train until zero training error, while accepting that it terribly overfitts? How would you know that you reached minimum? $\endgroup$ – Tim Apr 12 at 18:10
  • $\begingroup$ well not necessarily: if the model is regularized and/or you have a lot of data you shouldn't be able to reach zero training error. For instance, a linear regression never reaches zero training error (a part degenerate cases). However, to prevent overfitting I would use early stopping on a validation set and don't set epochs a priori. It simply makes little sense to me to fit a model only for some fixed arbitrary number of iterations $\endgroup$ – deltasun Apr 12 at 18:15
  • $\begingroup$ you know that you reach a (local) minimum when your function does not decrease within a chosen tolerance in a fixed number of steps. that's the standard in optimization problems $\endgroup$ – deltasun Apr 12 at 18:19
  • $\begingroup$ @deltasun How choosing the tolerance or number of epochs to wait is less arbitrary? With fixing epochs you simply decide to wait as long as it is possible for you to wait. If it would find minimum faster, you waisted your time. If not and longer time to wait was unacceptable for you, then still you'd have to stop. Nobody says it's the most optimal approach. $\endgroup$ – Tim Apr 12 at 18:25
0
$\begingroup$

TL;DR: Epochs are (unfortunately) the standard way to talk about the length of neural network training, mainly because we use them as a natural checkpoint for checking the model performance on a validation set.


"Epochs" are indeed very deceiving unit used to measure the length of the training. Using "number of updates" would make more sense because it is independent of the actual training set being used, and could be in theory tuned via cross validation. If you use on-line data augmentation, counting epochs starts making even less sense, because there is no finite dataset that one could pass, there are just new unseen samples coming at every iteration.

On the other hand, one has to check the validation loss regularly. Doing it after every iteration would introduce a major overhead; doing it every 100 (1 000, 10 000?) iterations is just arbitrary. Doing it after looking at each training sample once makes some sense, e.g. should give a good estimate how the model improved since the last check, suppressing potential effects of "training with more informative samples since the last time". In this sense, using epochs sounds reasonable.

Since the validation set is used for early stopping, and early stopping is used as the golden standard for checking when to terminate the training, epochs are the standard unit for training length.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.