# Implementing automated hyperparameter tuning within a manual cross-validation loop

I have several questions I'm going to try to bundle into one here. I am currently trying to implement convolutional neural network training on a public image dataset. I am trying to test and compare several different CNN architectures, eg. InceptionV3, ResNet, DenseNet, and a few others. To that end, I am trying to keep everything else in the code and the approach constant, only changing the architecture. I am using 5-fold cross validation with a hold-out test set of 20%.

I am running into a challenge to implement automated hyperparameter tuning however. My code is architected as follows.

1. Divide data into train (80%) and test (20%)
2. Create KFolds (n=5) from train data (80% train, 20% val for each fold).
3. For loop that iterates through each fold.
4. Within the for loop, I am using a custom data_aumentation function which takes the X_train, X_val, y_train, y_val, and augments the training data using ImageDataGenerator().
5. Fit the model in each fold, save best model performing model by val_loss.
6. Ensemble models and evaluate on the test set (initial hold-out 20%).

I am using Keras functional API for my model creation. Currently I am setting the hyperparameters manually prior to the model fitting + CV steps.

So on my questions:

1. My main question is what is the best approach from here to implement automated hyperparameter tuning? I've found a few different APIs like hype-tune, but they all seem to a) not work well with Keras functional API, or b) have cross-validation baked in, so I can't use my own cross-validation function that I've built.

2. Additionally, I want to check my understanding of what should happen with automated tuning: is it as simple as steps 2-5 above being repeated for each set of candidate hyperparameters, then taking the average performance (val_loss) for all the folds and selecting the set of hyperparameters with the lowest average val_loss?

3. For anyone responding to 1) is there a way to do the more sophisticated hyperparameter tuning (bayesian, gradient descent, evolutionary algorithms).

4. Is this overall, a reasonable approach to the problem? Is there merit to reconsidering CV+holdout, to doing something like nested cross-validation instead?

My relevant code:

Data augmentation function:

def data_aug(X_train,X_test,y_train,y_test,train_batch_size,test_batch_size):
train_datagen = ImageDataGenerator(
rotation_range=60,
# rescale=1.0/255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
vertical_flip=True,
fill_mode='nearest')
test_datagen = ImageDataGenerator()  # nothing applied to test dataset
train_batch = train_datagen.flow(X_train,y_train,batch_size=train_batch_size, seed=33)
test_batch = test_datagen.flow(X_test,y_test,batch_size=test_batch_size, seed=33)
return (train_batch,test_batch)


Cross-validation function

kfold = KFold(n_splits=5, shuffle=True, random_state=33)

cvscores = []
Fold = 1
for train, val in kfold.split(X_train_all, y_train_all):
gc.collect()
K.clear_session()
print ('Fold: ',Fold)

X_train = X_train_all[train]
X_val = X_train_all[val]
X_train = X_train.astype('float32')
X_val = X_val.astype('float32')
y_train = y_train_all[train]
y_val = y_train_all[val]

# Data Augmentation and Normalization
train_batch, val_batch = data_aug(X_train,X_val,y_train,y_val, batch_size, batch_size)

# If model checkpoint is used UNCOMMENT THIS
model_name = 'cnn_keras_Fold_'+str(Fold)+'.h5'

cb = callback()

# create model
model = create_model()      # CUSTOM ARCHITECTURE

# Fit generator for Data Augmentation - UNCOMMENT THIS FOR DATA AUGMENTATION
model.fit(train_batch,
validation_data=val_batch,
epochs=epochs,
validation_steps= X_val.shape[0] // batch_size,
steps_per_epoch= X_train.shape[0] // batch_size,
callbacks=cb,
verbose=2)

# evaluate the model
scores = model.evaluate(X_val, y_val, verbose=0)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
cvscores.append(scores[1] * 100)

Fold += 1

print("%s: %.2f%%" % ("Mean Accuracy: ",np.mean(cvscores)))
print("%s: %.2f%%" % ("Standard Deviation: +/-", np.std(cvscores)))