# Deep Neural Network tuning hyperparameters

I want to find the most efficient Feed forward Deep Neural Network architecture for my problem (binary classification).

1. Create network contains only one hidden layer,
2. Tune hyperparameters
3. Add more layers and goto 1.

I based my neural networks on Keras library with Theano backend.
Let me show my code implements the first two roadmap points.

def create_1nn_to_gridSearch(n_features, n_hidden, activation, has_dropout, p_drop, reg ):
model = Sequential()
W_regularizer=l2(reg), activity_regularizer=activity_l2(reg)))
# set activation according to the parameter
if activation is "relu":
elif activation is "PReLU":
elif activation is "LeakyReLU":
elif activation is "tanh":
elif activation is "sigmoid":

# enable dropout
if has_dropout:

model.compile(loss='categorical_crossentropy', optimizer="rmsprop")
return model


And then to tune hyperparameters of above network I use gridsearch (sklearn wrapper).

    train_gs_X, test_gs_X, train_gs_Y, test_gs_Y = train_test_split(features, target, random_state=42,train_size=0.5 )
grid_params = {"clf__n_hidden": [50,100,200,400],
"clf__reg": [0.008, 0.01, 0.05,0.1],
"clf__ activation" : [<all possibilities>],
"clf_has_dropout": [True,False],
"clf_p_drop": [0.1,0.2,0.5,0.8]
}
print(grid_params)

batch_size = 256
nb_epoch = 150

clf_gs = Pipeline([
('feature_scale', StandardScaler() ),
('clf', KerasClassifier(build_fn = create_1nn_to_gridSearch ,
n_features=n_features,
batch_size=batch_size, nb_epoch=nb_epoch, verbose=2) )
])

clf = grid_search.GridSearchCV(estimator = clf_gs,
param_grid = grid_params,
cv=10,
scoring='roc_auc',
verbose = 3);
clf.fit(train_gs_X, train_gs_Y);


Is my strategy of tuning DNN structure correct? Do you have idea how can I improve it? What hyperparameters should be added? Maybe some of them should be removed. Any comments on NN architecture? Do I need to add/remove some layers?

How can I correctly choose the number of epoch and batch size? Should I also perform gridsearch for it?
In case of 1 hidden layer I need to process about 200 of training. In case of 2 and more deeper networks this number will be much higher.

Do you have any other idea how to tune structures of DNN?

The last but not the least after the training phase I would like to deploy my trained model into C++ project. Do you know any easy way to implement Keras network in C++ standalone project?

• This is kind of late, but I believe you are on the right track. I think you should also tune the number epochs and the batch size as well. You could also have a look at this link. – darXider Jan 4 '17 at 19:51