Deep Neural Network tuning hyperparameters

Question

I want to find the most efficient Feed forward Deep Neural Network architecture for my problem (binary classification).

Study roadmap:

1. Create network contains only one hidden layer,
2. Tune hyperparameters
3. Add more layers and goto 1.

I based my neural networks on Keras library with Theano backend.
Let me show my code implements the first two roadmap points.

def create_1nn_to_gridSearch(n_features, n_hidden, activation, has_dropout, p_drop, reg ):
    model = Sequential()
    model.add(BatchNormalization(input_shape=(n_features,)))
    model.add(Dense(n_hidden,input_dim=n_features,
                    W_regularizer=l2(reg), activity_regularizer=activity_l2(reg)))
    # set activation according to the parameter
    if activation is "relu":
        model.add(Activation('relu'))
    elif activation is "PReLU":
        model.add(PReLU(input_shape=(n_hidden,)))
    elif activation is "LeakyReLU":
        model.add(LeakyReLU(input_shape=(n_hidden,)))
    elif activation is "tanh":
        model.add(Activation('tanh'))
    elif activation is "sigmoid":
        model.add(Activation('sigmoid'))

    # enable dropout
    if has_dropout:
        model.add(Dropout(p_drop))
    model.add(Dense(2,input_dim=n_hidden))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy', optimizer="rmsprop")
    return model

And then to tune hyperparameters of above network I use gridsearch (sklearn wrapper).

    train_gs_X, test_gs_X, train_gs_Y, test_gs_Y = train_test_split(features, target, random_state=42,train_size=0.5 )
grid_params = {"clf__n_hidden": [50,100,200,400],
               "clf__reg": [0.008, 0.01, 0.05,0.1],
               "clf__ activation" : [<all possibilities>],
               "clf_has_dropout": [True,False],
               "clf_p_drop": [0.1,0.2,0.5,0.8]   
              }
print(grid_params)

batch_size = 256
nb_epoch = 150

clf_gs = Pipeline([
  ('feature_scale', StandardScaler() ),
  ('clf', KerasClassifier(build_fn = create_1nn_to_gridSearch , 
                         n_features=n_features, 
                          batch_size=batch_size, nb_epoch=nb_epoch, verbose=2) )
])

clf = grid_search.GridSearchCV(estimator = clf_gs,                               
                               param_grid = grid_params,
                               cv=10,
                               scoring='roc_auc',
                               verbose = 3);
clf.fit(train_gs_X, train_gs_Y);

Is my strategy of tuning DNN structure correct? Do you have idea how can I improve it? What hyperparameters should be added? Maybe some of them should be removed. Any comments on NN architecture? Do I need to add/remove some layers?

How can I correctly choose the number of epoch and batch size? Should I also perform gridsearch for it?
In case of 1 hidden layer I need to process about 200 of training. In case of 2 and more deeper networks this number will be much higher.

Do you have any other idea how to tune structures of DNN?

The last but not the least after the training phase I would like to deploy my trained model into C++ project. Do you know any easy way to implement Keras network in C++ standalone project?

Answer 1

Great approach for designing a DNN! I would make the architecture deeper (e.g. add a number of layers and a bypass parameter for each layer). Also, it is recommended to use ReLU activation with He normal weight initialization and a keep probability of 0.5 for dropout during training. It's also common to use Batch Normalization right before the activation layer.

In order to speed up parameter search, I would experiment with random search or bayesian optimization that uses Gaussian Processes (GPs) to explore parameter and trade-off between exploration and exploitation.