I have designed an autoencoder with a encoder and decoder consiting of 2D convolutational layers (the input are 40'000 2D images). I train the autoencoder using adam optimizer. The autoencoders has the following hyperparameters which I would like to tune (in brackets are my default values):

  • Number of layers in encoder and decoder (I start with 2 in decoder and encoder)
  • Filter size for convolutional layers (I start with 32 and 64)
  • Convolutional kernel size (I start with 3x3)
  • Stride size (I start with 2x2)
  • Dropout (I start with 0.25 after each layer)
  • Learning rate (0.001)
  • learning_rate_decay (0)
  • Latent dimension (I start with 8)
  • Number of units in the dense layer (layer before creating latent space, I start with 16)
  • Batch size (I start with 128)

One possibility would be to use just grid or random search but this is very inefficient and takes a long time with so many hyperparameters. Instead, I would like to observe the training and validation loss (using tensorboard) and adjust the parameters accordingly. For example when observing the training and validation loss there could be overfitting or underfitting (or also an increase in loss etc.).

Are there some general rules or hints how the hyperparameters could be adjusted based on the observed losses or based on other criterions?


1 Answer 1


As you have correctly assumed, there are simply too many parameters to perform a full grid search. Luckily, many of those have relatively small effect compared to others:

  • Convolution kernel size (3x3 works universally quite well)
  • Convolution stride (2x2)
  • Batch size
  • Dropout is not needed for convolutional layers; also, for dense layers use it only if you experience overfitting

There are some rules of thumb for setting the learning rate and its decay, a popular one is to increase the learning rate and record the loss, which should yield a curve like this one (taken from this article): Curve showing the change of loss w.r.t. learning rate, first going down and then growing fast

The region of the steepest loss descent hints to the optimal learning rate choice. Alternatively, optimizers like Adam are robust to the choice of the learning rate (since the amount of change is adaptive for each parameter), but they have their own parameters to tune for optimal performance.

What remains are:

  • number of layers,
  • number of units per layer,
  • latent dimension

These should be easy to tune: start with a small network, increasing the size as long as your validation error gets better.

  • $\begingroup$ Why is dropout not needed for convolutional layers? For example github.com/ekzhang/vae-cnn-mnist/blob/master/model.py used dropout after convolution. $\endgroup$
    – machinery
    Jul 22, 2019 at 13:32
  • $\begingroup$ In the picture, is it the training or validation loss? $\endgroup$
    – machinery
    Jul 22, 2019 at 14:32
  • $\begingroup$ Last but not least, regarding the number of units per layer, i.e. the number of filters for CNN, how should I increase that? Currently I have in the encoder in the first CNN 32 filters and in the second CNN 64. Should I just double the amounts in each layer? Another possibility is also to have the same amount of filters in each CNN... $\endgroup$
    – machinery
    Jul 22, 2019 at 14:38
  • $\begingroup$ There are works advocating dropout for conv.nets, however, it is important to note it has different meaning/semantics than for dense layers, see e.g. towardsdatascience.com/… for deeper discussion. My point is, convnets work great even without dropout, which can't always be said about fully connected networks. If you need to drop some hyperparameters, I would go for dropout. $\endgroup$ Jul 23, 2019 at 10:01
  • $\begingroup$ In the picture it is the training loss $\endgroup$ Jul 23, 2019 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.