It is common to use a validation set to reduce overfitting for machine learning tasks. For traditional (non-neural network based) models, many validation techniques, such as cross-validation, can be used to find the best hyperparameters for the model.

However, for deep learning models, as it requires extensive computation resources, brute force search-based validation techniques may not be applicable. How do we do validation for deep learning model? One possible approach is to use a validation set for early stopping, is there any other techniques?


1) Instead of using brute force search-based techniques, people generally just try out some combinations of different hyper-parameters and choose the one that that gives the best accuracy on the validation set.

2) One technique like early stopping is this: suppose that you evaluate your model on a validation set after each epoch. Now, if on some epoch X you get an accuracy lower than that on epoch X-1 (i.e. the previous epoch), you could discard epoch X completely and resume training from epoch X-1 i.e. you restore the best model. Because the inputs to a model always get divided into mini-batches randomly, this new epoch X will be different than the last epoch X. Of course, you could also only restore the best model only if the difference in the accuracies at the end of the two epochs is greater than some number.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.