0
$\begingroup$

I wondered whether tuning hyperparameters, such as the learning rate or the amount of layers and neurons, is done seperately or alltogether. E.G. you first tune the amount of neurons and when getting acceptable error rates you continue with tuning the learning rate. I know there are different methods such as the grid search or bayesian optimization, but in my case I have to tune the hyperparameters manually, because I am forced to use SPSS. Therefore I was wondering which way I should try my luck.

Best regards

$\endgroup$
3
  • $\begingroup$ Grid search is likely to get you closer to a joint local optimum faster than the one-at-time search. You can code the grid search in SPSS. I'm not sure why anyone would force you to use a suboptimal tool for the job. $\endgroup$
    – R Carnell
    Commented Aug 31, 2022 at 15:01
  • $\begingroup$ Lets just say I have to use SPSS and a manual search for this method. The question is: Do the hyperparameters affect each other or can I tune them one at a time? $\endgroup$
    – MauM99
    Commented Aug 31, 2022 at 15:23
  • $\begingroup$ I'm not sure I can give a universal answer that learning rate and number of layers do not need to be fitted jointly. If you are using any early stopping criterion for the optimization, then they would absolutely be correlated, for example. My advice is that it is best to tune hyperparameters jointly. $\endgroup$
    – R Carnell
    Commented Sep 1, 2022 at 12:39

1 Answer 1

0
$\begingroup$

Typically people fix a network architecture, and then tune the hyperparameters (learning rate, number of epochs, etc.) for that architecture. There are many ways to do hyperparameter tuning, and no consistent universal agreement on what is "best". Some people have suggested that selecting random choices for hyperparameters is simple and works reasonably well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.