Bayesian optimization neural network When tuning my neural net with Bayesian optimization I want to determine the optimal number of hidden layers and the corresponding number of neurons in each hidden layer. The problem is that with an increasing number of hidden layersthe vector of hyperparameters gets longer, because for each hidden layer added I need one more entry in the hyperparameter vector indicating the number of neurons in the added layer. For bayesian regression I need the input dimension to be constant. How do I cope with this problem?
 A: One way of doing that could be by optimizing over the number of neurons in each layer, where the number of neurons can be 0 (no layer). In order to avoid duplicate configurations (e.g., 10, 10, 0 and 10, 0, 10) one can place an additional constraint for the optimization to ensure that a layer should contain 0 neurons if one of the previous layers has 0 neurons. 
Denoting the number of neurons in layer $k$ as $N_k$ with the maximal number of layers equal to $K$, this constraint can be expressed as 
\begin{align}
\forall k \in \{1, 2, \dots, K-1\}:\ & N_{k+1} \le N_{k+1} \times N_{k} \implies\\
\forall k \in \{1, 2, \dots, K-1\}:\ & N_{k+1} - N_{k+1} \times N_{k} \le 0,
\end{align}
where $N_k \in \{0, 1, \dots\}$. 
If $N_k = 0$ (no layer $k$), then the above inequality is true only when $N_{k+1}$ is also 0, thus ensuring that if there is no layer $k$, then there is no any subsequent layers.
Most Bayesian optimization packages should be able to do that. For example in GPyOpt, allowing for up to 4 layers and passing the number of neurons in matrix x (parameters are passed as a row in a 2D array, more on constrained optimzation in GPyOpt can be foung here) the constraints can be written as
[{'name': 'constraint_on_layer_2', 'constraint': 'x[:, 1] - x[:, 1] * x[:, 0] - 1e-08'},
 {'name': 'constraint_on_layer_3', 'constraint': 'x[:, 2] - x[:, 2] * x[:, 1] - 1e-08'},
 {'name': 'constraint_on_layer_4', 'constraint': 'x[:, 3] - x[:, 3] * x[:, 2] - 1e-08'}]

