I have designed an autoencoder with a encoder and decoder consiting of 2D convolutational layers (the input are 40'000 2D images). I train the autoencoder using adam optimizer. The autoencoders has the following hyperparameters which I would like to tune (in brackets are my default values):
- Number of layers in encoder and decoder (I start with 2 in decoder and encoder)
- Filter size for convolutional layers (I start with 32 and 64)
- Convolutional kernel size (I start with 3x3)
- Stride size (I start with 2x2)
- Dropout (I start with 0.25 after each layer)
- Learning rate (0.001)
- learning_rate_decay (0)
- Latent dimension (I start with 8)
- Number of units in the dense layer (layer before creating latent space, I start with 16)
- Batch size (I start with 128)
One possibility would be to use just grid or random search but this is very inefficient and takes a long time with so many hyperparameters. Instead, I would like to observe the training and validation loss (using tensorboard) and adjust the parameters accordingly. For example when observing the training and validation loss there could be overfitting or underfitting (or also an increase in loss etc.).
Are there some general rules or hints how the hyperparameters could be adjusted based on the observed losses or based on other criterions?