In a Neural Network training, the cost of the model changes throughout the training process when using gradient descent (or something analogous), this is the point of the algorithm. However the cost might not decrease monotonically. In some points the cost might even increase.

Cost in Neural Network model per 100 iterations using gradient descent

So, Is it OK to keep track of the parameters that output the lowest cost and use those as the best parameters for the model? In the image it would imply using the parameters that output the lowest cost instead of the last parameters found.

Assume that the cost returned by the lowest cost outputs an acceptable model accuracy.

Does doing this cause some kind of problem?

  • 1
    $\begingroup$ It is all approximations. That means it isn't going to be "true" lowest cost, but estimated lowest cost. $\endgroup$ – EngrStudent - Reinstate Monica Jun 1 '18 at 19:03

This is a perfectly okay thing to do. Other things you might try are reducing the learning rate to make it more likely that the training cost goes down most of the time and also selecting the model based on the cost on the validation set rather than training.

  • $\begingroup$ Great, thanks for your insights. Do you have any source you can provide to validate this? $\endgroup$ – loco.loop May 29 '18 at 23:57
  • $\begingroup$ What about mini-batch? After each epoch should I do a comparison? Or how would this work? $\endgroup$ – loco.loop Jun 1 '18 at 22:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.