1
$\begingroup$

I want to compare different hyperparameter settings on the same network and the same task to get an impression of what works good and what works better. I am comparing different initializer, activation functions, optimizer, etc.

I am asking myself what are the relevant criteria to compare neural nets with different hyper parameter settings? Is it more then just:

  • best validation accuricy
  • number of epochs when best validation accuricy was reached
  • time for each epoch (speed of training)

I would be happy for some input.

Thanks!

Improve this question
$\endgroup$
5
  • $\begingroup$ I'd like to add one more to the mix: output fluctuations. If the metric you chose fluctuates a lot from epoch to epoch or not. $\endgroup$
    – Djib2011
    Commented Aug 24, 2018 at 10:49
  • $\begingroup$ @Djib2011: Thanks. Good idea. Do you have any idea about how to calculate this fluctuation? $\endgroup$
    – Dieshe
    Commented Aug 24, 2018 at 18:41
  • $\begingroup$ I've never calculated it exactly, I've just noted it as an observation. For example one run might achieve the highest validation accuracy, but it fluctuates a lot more than another run. $\endgroup$
    – Djib2011
    Commented Aug 25, 2018 at 1:50
  • $\begingroup$ I would say the relevant criteria depend entirely on the application, i.e. validation accuracy for medical diagnosis applicatoins, training speed perhaps important during the design period, and so on $\endgroup$
    – GR4
    Commented Aug 28, 2018 at 10:01
  • $\begingroup$ @GR4 Yes you are right. And I am looking for a collection of all these criteria to compare hperparameter settings independent of the application. $\endgroup$
    – Dieshe
    Commented Sep 10, 2018 at 7:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.