I don't really understand how people begin/start tuning their network and there seems to be a lot of conflicting information.

One online answer I saw said:

Start with the learning rate, tune that, then tune the batch size, tune that, then tune the number of epoches, ..., then add a layer. So basically you tune everything so that the network is optimal for each of the parameter. Tune everything sequentially. The assumption here is that this "optimality" stacks up when you vary other parameters.

Another online answer I saw said:

Come up with several set of {learning rate, batch size, ..., number of hidden units}. Then create a model with each set of parameters. Run the model for each set of parameters, pick the best one on the validation set. This one seems to require a lot more effort than the first!

Which method is correct? And how do I begin tuning? Please provide a good rationale or a reference.


1 Answer 1


I think the most important thing is to run your training for desired number of epochs, log training loss, validation loss, look at the plots of them and ask yourself what you don't like about those plots.

It's highly unlikely that they'll look perfect on the first go (but if they did, congrats, you have optimal set of parameters). Most probably you'll encounter one of a few issues like:

  • Training loss going down but validation loss not
  • Both training loss and validation loss not going down
  • Loss will be unstable (sometimes going up, sometimes down)
  • Loss going down quickly first and then flattening
  • . . .

Depending on what issue or combination of issues is, you'll need to adjust certain parameters of the network and training. After you establish what aspect of the training you want fixed, it should become more clear which parameter should be changed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.