The question is "how do we determine the (hyper)parameters in deep learning models, usch as CNN, RNN?"
This is a difficult question that so far I am not aware of a solid solution and I want to bring this up in a more specific manner.
(Similar question has been asked before 2 years ago, and I think it is a good time to bring it up again: Guideline to select the hyperparameters in Deep Learning)
The list of parameters/hyperparameters are ,for example:
- Input patch size (e.g., 64x64)
- number of layers
- number of filters in each convolutional layer (if it is a CNN)
- learning rate of the network
Here are some suggestions based on different papers and I wrote my concerns:
- Typically, people suggest that the deeper the network the better, but then how deep is that? Because we cannot just say, 10 is deep enough or 100 is deep enough, especially in the field of biological science research (When you publish a paper; reviewers will judge you)
- Based on No.1, people will suggest to use (a) grid search; (b) random search to find the best combinations of parameters. But the problem is, running all these kind of combinations (let say 200) take A LOT OF TIMES. It is a feasible.
- Based on No.1 and No.2, researchers will suggest to build the model based on some successful models. For example, people nowadays will start with the parameters according to 2012 Imagenet. However, this may not be applied to all the cases. For example, in MRI images analysis, people do pixel classification for segmentation, in that case having 256x256 input patch is just not make sense (since you are predicting the class of the center of the patch).