9
$\begingroup$

I have found a set of optimal hyperparameters (e.g. learning rate for gradient descent) using cross validation and bayesian optimisation. While searching for the optimal hyperparameters, my neural net architecture remained constant (same number of layers, same number of nodes etc.).

I chose a relatively small architecture with 2 hidden layers so that the model would train and evaluate faster.

Now that I've found the optimal hyperparameters, I am wondering if I increase the number of hidden layers and nodes per layer, will the hyperparameters still be optimal? All else will remain the same (same training data and validation data).

The reason for making the network deeper and wider now, is that this will serve as the final model, which I will allow to train for more epochs to obtain the highest possible accuracy; I don't mind if it takes a few days to train 1 model now, whereas in optimising the hyperparameters I needed to train a model within a few hours.

Improve this question
$\endgroup$
20
$\begingroup$

Unfortunately, it doesn't work that way. Hyperparameters cooperate in hard-to-predict ways. For example, a bit extreme to make the point.

You have no hidden layers, in other words, you are fitting a logistic regression. A logistic regression will usually not really overfit. So you use a relatively big learning rate and a lot of epochs, and find that that works fine, at least, not worse than other hyperparameter configurations. Then you increase the number of layers. You get a complex model, that is now suddenly prone to overfitting. Then the big learning rate and the many epochs that worked fine earlier are no longer optimal.

Small thing, I would say the number of hidden nodes, or more generally, the whole architecture of the neural network, is also part of the hyperparameters. So your question I read more like, will the same learning rate be optimal if I increase the complexity of the network.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I suspected as much. Although I just gave learning rate as an example; in reality I am tuning other parameters in addition to learning rate. $\endgroup$ – PyRsquared Dec 4 '19 at 12:30
  • 2
    $\begingroup$ Large learning rates are more appropriate for models in which the second derivatives is smaller (i.e., less curvature of log-likelihood/loss function). Is there any reason to think a more complex model will have more curvature than a simpler model? That would be a very interesting fact to me if provable true. $\endgroup$ – Cliff AB Dec 5 '19 at 0:34
  • $\begingroup$ how is the curvature connected to the complexity? $\endgroup$ – Ben Dec 5 '19 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.