I'm using a neural network for a multi-target regression task and would like to perform hyper-parameter optimization. The network has one hidden layer and uses MSE loss on the output. I have large training and validation sets.
Now I have several questions about hyper-parameter tuning:
- In what cases is random sampling preferred over grid search? I could imagine if one hyper-parameter has a much greater influence than the others then their effects won't become obvious for random sampling (since the dominant parameter changes as well).
- Which hyper-parameters should be included? I sampled the batch size, the kernel initializer (uniform vs. normal), the learning rate and the number of nodes in the hidden layer. Is it appropriate to randomly sample all these together?
- When scanning the batch size, how should the number of epochs be adjusted in order to make the results comparable? If
Nis the number of samples in the training set then the procedure will perform
N / batch_size * n_epochsgradient updates, so I tried to keep this number constant among the different hyper-parameter samples. On the other hand for larger batch sizes the whole data set will be traversed more often and so the results might also increase due to that. Are the any rules (of thumb) for that?
- What is a good way of identifying optimal hyper-parameters for a regression task? I monitored the loss for a few dozens epochs and then checked the final loss (should be small) and the initial loss divided by final loss (should be large) in order to account for different starting values of the loss. However both measures pointed at different optimal configurations whose parameters were not too similar. Also when I checked the best 10 hyper-parameter samples for both measures, their values wildly varied (over one order of magnitude), not really indicating an optimal configuration, although the loss was very similar. Does that mean the variance of values in that range doesn't really have an effect? In addition, even though the loss might be small at the evaluation point, the learning rate could still be too high and the loss would overshoot for subsequent epochs (which I didn't monitor).
- Should I fix the seed for the RNG when comparing different hyper-parameter configurations? It seems to make sense for varying the learning rate for example since the start loss will always be the same, however when varying the network architecture (number of hidden layer nodes) then the start loss will vary anyway.