3
$\begingroup$

I'd like to use Bayesian Optimization to tune the hyper parameters of a feed-forward neural network.

Among these hyper parameters, there is the number of hidden layers in the network, as well as the number of nodes in each layer. The issue is that the number of hyper parameters depends on the value chosen for the number of layers. For example, with a single hidden layer, there is only one parameter for the number of nodes. With five hidden layers, there are five number of nodes to choose.

Is there are smart way to handle this? Or do I have do choose between optimizing the number of layers with a fixed layer size and optimizing the number of nodes in each layers with a fixed number of layers?

$\endgroup$
3
$\begingroup$

Recently, I see a paper in arXiv Neural Architecture Search with Bayesian Optimisation and Optimal Transport.

The author use it to explore DNN and CNN by creating a pseudo-metric space consist of neural network by Optimal transport. I don't think it exactly answer your question, but it maybe helpful for you to design a space/heuristic to trade off between optimizing number of layers or nodes in each layer.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.