I just finished building a 1D CNN using TensorFlow, and I want to optimize a variety of hyper-parameters using Scikit-Optimize (skopt) (although, I would be willing to use whatever optimization package is most appropriate). My input is always the same size across all examples. The parameters I want to optimize include:
- the number of convolutional layers,
- the size of the convolutional patch for a given layer,
- the stride of the convolutional patch for a given layer,
- the number of outputs per a given layer's patches
This is trickier than the previous optimization project I've worked because the number of layers impacts the maximum possible patch size or stride (if the patch size and strides are too large, then after a few layers, the next layer will try to access a patch larger than the previous layer's output creating an error).
Possible questions and solutions:
- Is it possible to add a constraint to skopt? ie. variable A < variable B
- Should I implement a "check" where I test whether a set of parameters creates a valid graph and if not then my objective function returns positive infinity? (I am trying to minimize my objective function, so positive infinity would suggest the tested parameters are very bad).
Other things I am curious about:
Let's say I only want to optimize: the number of convolutional layers (between 1 and 3) and the number of outputs per layer. So I set up my initial Skopt space as such:
space = [Real(1, 3, name='num_layers'),
Real(1, 16, name='num_outputs_layer1'),
Real(1, 16), name='num_outputs_layer2'),
Real(1, 16), name='num_outputs _layer3')]
Let's say the optimizer is testing "num_layers" = 2. Although, the way I set this up, the optimizer would also be tweaking "num_outputs_layer_3". However, my resulting graph wouldn't contain any layer_3 and so "num_outputs_layer_3" has no impact on it. I'm worried the optimizer would waste effort on it. Is there any way for me to set it up so that the optimizer doesn't worry about tweaking the outputs of a layer that doesn't exist?
EDIT Also, what optimization algorithm would be best for this? Skopt lets me choose between these two (I am currently set to "auto"):