Recurrent Neural Networks: How to find the optimal parameters?

Question

I am trying to fit a recurrent neural network for a binary classification problem using the 'keras' library (https://keras.io/layers/recurrent/). Now these networks have many parameters to tune.

LSTM

keras.layers.recurrent.LSTM(output_dim, init='glorot_uniform', inner_init='orthogonal', forget_bias_init='one', activation='tanh', inner_activation='hard_sigmoid', W_regularizer=None, U_regularizer=None, b_regularizer=None, dropout_W=0.0, dropout_U=0.0)

GRU

keras.layers.recurrent.GRU(output_dim, init='glorot_uniform', inner_init='orthogonal', activation='tanh', inner_activation='hard_sigmoid', W_regularizer=None, U_regularizer=None, b_regularizer=None, dropout_W=0.0, dropout_U=0.0)

Simple RNN

keras.layers.recurrent.SimpleRNN(output_dim, init='glorot_uniform', inner_init='orthogonal', activation='tanh', W_regularizer=None, U_regularizer=None, b_regularizer=None, dropout_W=0.0, dropout_U=0.0)

How do I tune these parameters? Which are the most important ones to tune if am selecting only few parameters to tune?

Answer 1

In these cases, you usually perform a cross validation, or even better, a k-fold cross validation.

The idea is the following:

1. Determine an error threshold
2. Divide your samples into k subsets of the same size (without gaps, potentially overlapping)
3. Repeat until satisfactory performance is reached
   1. Select one particular combination of parameters (e.g. randomly of with a parameter optimization approach)
   2. Initialize an error sum with value $0 $
   3. For each subset
      1. Call this subset "validation set"
      2. For each other subset
         1. Call this subset "test set"
         2. Repeat until the model approximates the validation subset with an error below the error threshold
            1. Call all subsets "training set" that are neither test set nor validation set
            2. Train your model on the training set and with the chosen parameters
         3. Add the model's error for each test set to the error sum
   4. The error sum (divided by $k * (k - 1) $) is a measure for parameter quality: the lower the better