2
$\begingroup$

A model may take several days to train until it reaches an equilibrium - say if the change in error between epochs is lower than some threshold $\epsilon$, or accuracy reaches some equilibrium.

When tuning hyperparamters (via cross validation or otherwise), is there any problem in only training the model for a small number of epochs, so that training completes faster and more hyperparameters can be evaluated in a shorter amount of time? Training for a small number of epochs means the model hasn't reached an equilibrium as described above, but it has learned something so is performing far better than the random initialization.

$\endgroup$
  • $\begingroup$ Sometimes the "plateau" is a bad place, and just a few steps down the gradient is both "good enough" in terms of quality and "desirable" in the sense of not over-fitting. $\endgroup$ – EngrStudent - Reinstate Monica Dec 4 '19 at 18:25
2
$\begingroup$

This is a somewhat open-ended question, but I would say that it is entirely context-dependent as to whether this is ok or not. Reaching equilibria in the sense you describe only indicates a local minimum has been found in the majority of cases.

Further to this, it is possible that as training continues, you may reach and then pass a minimum in the loss function for a test set, meaning that real-world performance may eventually begin to decay. In this scenario the model may not be stable in training, but the hyperparameters were "better" at (or near) the point where this minimum was reached.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.