0
$\begingroup$

I understand tuning the learning rate, momentum, batch size, etc. and finding the best set of parameters using the validation set.

However, I don't understand when people say that you should also tune the unit of hidden units as well as the number of hidden layers for a neural network.

https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/

hyperparameter tuning in neural networks

  1. Wouldn't tuning the number of hidden units and layers fundamentally alter the structure of your model/architecture? It is no longer that you are comparing the same model with different parameters for training it. You are comparing model A, model B, etc...

  2. Another problem is that you are essentially enlarging your problem. The original problem was to change some parameters of your model and see which performs the best outside of your training data. The new problem becomes, find a neural network model out of the space of all neural networks (as you can just always stack more units and more layers), and pick the best performing one.

In my opinion, for a valid comparison, you need to fix your architecture, then tune some parameters associated with how this architecture was trained, instead of coming up with several (usually small amount, out of infinitely many) architectures and then see which one performs the best.

Can someone chime in?

Improve this question
$\endgroup$
2
  • $\begingroup$ What’s the problem with comparing model A vs model B? $\endgroup$
    – astel
    Commented May 14, 2020 at 1:24
  • $\begingroup$ @astel The scope of your problem becomes extremely large. "Validation" taken to the logical extreme means comparing all the best performing neural network architecture in the last 10-20 years just for MNIST classification. I mean, why bother changing something like batch size or momentum when you can just pick GoogleNet or some ResNet or AlexNet for your classification task? $\endgroup$
    – Fraïssé
    Commented May 14, 2020 at 2:05

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.