How do you go about choosing initial hyperparameters (layer size, # of hidden units in RNN and dense layers, etc.) when training RNNs and MLPs? How do you iteratively tweak these settings -- do you generally start with changing the number of layers, units, or optimizer?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The first part of your question is very closely related to this question (How to choose the number of hidden layers and nodes in a feedforward neural network?). However, keep in mind that mostly, you figure out the layers and unit amount through trial and error.
Once you have found a suitable network architecture, you get on with the optimizer - you don't want your network to overfit the data for example. This part is related to reduce over/under fitting, but also making training faster by initialising your weights correctly. A list of gradient optimization algorithms.
Ofcoures, you can always change the number of layers/units at any given time after that.
