Neural Network Tuning with Bayesian Optimisation - Number of layers and size of layers

Question

I'd like to use Bayesian Optimization to tune the hyper parameters of a feed-forward neural network.

Among these hyper parameters, there is the number of hidden layers in the network, as well as the number of nodes in each layer. The issue is that the number of hyper parameters depends on the value chosen for the number of layers. For example, with a single hidden layer, there is only one parameter for the number of nodes. With five hidden layers, there are five number of nodes to choose.

Is there are smart way to handle this? Or do I have do choose between optimizing the number of layers with a fixed layer size and optimizing the number of nodes in each layers with a fixed number of layers?

Answer 1

Recently, I see a paper in arXiv Neural Architecture Search with Bayesian Optimisation and Optimal Transport.

The author use it to explore DNN and CNN by creating a pseudo-metric space consist of neural network by Optimal transport. I don't think it exactly answer your question, but it maybe helpful for you to design a space/heuristic to trade off between optimizing number of layers or nodes in each layer.