In stacked autoencoders during greed layer-wise training of individual autoencoders using gradient descent and backpropagation to minimize the reconstruction error(squared error or cross entropy) what is the ideal stopping criteria for the pre-training ?

Is it when a minimum reconstruction error is achieved or some other criteria ?

  • $\begingroup$ Convergence of reconstruction error can be good measure. Convergence shows that the network can't get better overtime so there is not much point to continue training. $\endgroup$ May 12 '15 at 9:38
  • $\begingroup$ Can you state a quantifiable measure of what is the optimal reconstruction error % which should be attained by the network ? $\endgroup$ May 14 '15 at 18:01
  • 1
    $\begingroup$ Optimal error depends on the data. What I'm saying is monitor reconstruction error (cost function). As weight captures more common structures, the error will decrease. After some time, error converges to some value and can't decrease anymore. That is an ideal stopping criteria. $\endgroup$ May 15 '15 at 14:13
  • $\begingroup$ @yasin.yazici This is a good sign to stop training, but how do you know if the error achieved is low enough for the whole classifier network, or that you should try a different autoencoder config? Is there a rule of thumb that would save you from trying every autoencoder/classifier layout? $\endgroup$
    – jwalker
    Sep 19 '15 at 7:46
  • $\begingroup$ @jwalker Early stopping find optimum error rate for specific network config (hyperparameters). You should try greedy grid search or bayesian optimization to find optimum hyperparameters. You can make some guess about number of layers and hidden neurons based on complexity of the problem. If you need detailed explanation read this paper. arxiv.org/pdf/1206.5533v2.pdf Especially the title "Hyper-Parameters" $\endgroup$ Sep 19 '15 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.