Assume one wants to train a Neural Network from scratch. The content of the research is not focused on the optimization, so ideally, I only want to show that with "the standard training set up" it performs well or (maybe) better than state of the art (or simply train a standard Neural Net in standard condition to reproduce old research). Assume the data set is fixed (something like MNIST, cifar-10,cifar-100, ImageNet or even a synthetic data set or a new data set). With a known data set what would be the "standard" way to train my Neural Network? In particular what optimizer and hyper parameters and epochs should I choose? Do I want to train it multiple times or only a single time and choose the best model? Are the models always trained until convergence? If convergence is the standard, how do people estimate the number of epochs or iterations to achieve this convergence most likely?

For example, say we have cifar-10 and I want to train a Neural Network on it. I choose Adam or mini-batch Stochastic Gradient Descent (SGD). Ideally I'd want to train it and say to a fellow researcher: "yea I'm training it the standard way". I guess the main issue is that there seem to be many parameters at play that might affect the (statistical) results of the experiments and simulations. For example, there seems to be a subtle but important interplay with the number of epochs/iterations, learning rate, decay rate and batch size. For example, if someone told me that the standard way is to run any Neural Net experiment with at least 50 epochs for me that wouldn't really mean much because even if I did 50 epochs, if my learning rate is so tiny that my algorithm barely updates the parameters, 50 epochs doesn't really mean much. In other words, it seems that the "effective number of epochs" (term I just made up) should be the deciding factor. So essentially all the parameters combined should be considered. Or maybe the standard thing is to always run at least 50 epochs regardless of the other hyper parameters when using SGD/Adam or something.

Or maybe the standard thing is just to run Begnio's well known random search and choose the one with the lowest (train, validation or test, probably validation) error. But do we really need to run this expensive procedure if we want to train Networks on standard data sets? When is random search the appropriate choice?

As in my earlier paragraph maybe convergence is the method of choice that is "standard". If this is the case, how do people do this? Usually training to convergence on some of these data set takes a lot of time even with GPUs and sometimes it seems convergence can only be tested with human interventions by plotting error vs iterations/epochs (seems gradient zero or small is rather rare?). If this is the case, how do people satisfy this condition?

I am not looking for any theoretical guarantees, just what researchers or partitioners do in practice to train a NN the "standard way".

  • $\begingroup$ It really depends on your application if your research field has converged to a standard way. The best way to find out is to read recent articles from your field and see what others do. $\endgroup$ – Pieter Jan 21 '17 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.