I am trying to create a Deep Belief Network (DBF) for a binary classification problem. The nolearn package provides a good library for implementing them. I see that there are very many parameters to learn and tune a DBF.
The possible list of parameters are:
- The number of hidden layers and the number of neurons in each.
- Learning rate
- learn_rate_decays - The number with which the learn_rate is multiplied after each epoch of fine-tuning.
- Number of epochs to train (with backprop).
- learn_rates_pretrain – A list of learning rates for pretraining.
- momentum_pretrain – Momentum for pre-training.
- L2 costs per weight layer.
- Dropouts per weight layer.
- Momentum for pre-training.
- L2 costs per weight layer, for pre-training.
- Number of epochs to pre-train (with CDN).
- Scale of the randomly initialized weights.
- Number of nonzero incoming connections to a hidden unit.
- Output activation function. (can be any of Sigmoid, Linear, Softmax)
Among these what are the most important parameters to tune? Do I have to tune all the parameters?