When tuning my neural net with Bayesian optimization I want to determine the optimal number of hidden layers and the corresponding number of neurons in each hidden layer. The problem is that with an increasing number of hidden layersthe vector of hyperparameters gets longer, because for each hidden layer added I need one more entry in the hyperparameter vector indicating the number of neurons in the added layer. For bayesian regression I need the input dimension to be constant. How do I cope with this problem?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Could you give us more details on what exactly are you doing? $\endgroup$– TimCommented Apr 1, 2018 at 15:14
-
$\begingroup$ I want to find the optimal number of hidden layers and neurons per layer for a standard FNN. I want to do this via Bayesian optimization which relies on gaussian process regression. For a Gaussian regression the input dimension must be fixed but with a different number of hidden layers the input dimension changes. For example: when I have three hidden layers the input vector for the Gaussian regression is 4-dimensional $(number\_of\_layers = 3, neurons\_layer1, neurons\_layer2, neurons\_layer3)$. When the number of hidden layer is two, the last entry in the vector is not needed. $\endgroup$– lbf_1994Commented Apr 2, 2018 at 11:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
One way of doing that could be by optimizing over the number of neurons in each layer, where the number of neurons can be 0 (no layer). In order to avoid duplicate configurations (e.g., 10, 10, 0 and 10, 0, 10) one can place an additional constraint for the optimization to ensure that a layer should contain 0 neurons if one of the previous layers has 0 neurons.
Denoting the number of neurons in layer $k$ as $N_k$ with the maximal number of layers equal to $K$, this constraint can be expressed as
\begin{align} \forall k \in \{1, 2, \dots, K-1\}:\ & N_{k+1} \le N_{k+1} \times N_{k} \implies\\ \forall k \in \{1, 2, \dots, K-1\}:\ & N_{k+1} - N_{k+1} \times N_{k} \le 0, \end{align} where $N_k \in \{0, 1, \dots\}$.
If $N_k = 0$ (no layer $k$), then the above inequality is true only when $N_{k+1}$ is also 0, thus ensuring that if there is no layer $k$, then there is no any subsequent layers.
Most Bayesian optimization packages should be able to do that. For example in GPyOpt, allowing for up to 4 layers and passing the number of neurons in matrix x (parameters are passed as a row in a 2D array, more on constrained optimzation in GPyOpt can be foung here) the constraints can be written as
[{'name': 'constraint_on_layer_2', 'constraint': 'x[:, 1] - x[:, 1] * x[:, 0] - 1e-08'},
{'name': 'constraint_on_layer_3', 'constraint': 'x[:, 2] - x[:, 2] * x[:, 1] - 1e-08'},
{'name': 'constraint_on_layer_4', 'constraint': 'x[:, 3] - x[:, 3] * x[:, 2] - 1e-08'}]
