Implementing automated hyperparameter tuning within a manual cross-validation loop

Question

I have several questions I'm going to try to bundle into one here. I am currently trying to implement convolutional neural network training on a public image dataset. I am trying to test and compare several different CNN architectures, eg. InceptionV3, ResNet, DenseNet, and a few others. To that end, I am trying to keep everything else in the code and the approach constant, only changing the architecture. I am using 5-fold cross validation with a hold-out test set of 20%.

I am running into a challenge to implement automated hyperparameter tuning however. My code is architected as follows.

1. Divide data into train (80%) and test (20%)
2. Create KFolds (n=5) from train data (80% train, 20% val for each fold).
3. For loop that iterates through each fold.
4. Within the for loop, I am using a custom data_aumentation function which takes the X_train, X_val, y_train, y_val, and augments the training data using ImageDataGenerator().
5. Fit the model in each fold, save best model performing model by val_loss.
6. Ensemble models and evaluate on the test set (initial hold-out 20%).

I am using Keras functional API for my model creation. Currently I am setting the hyperparameters manually prior to the model fitting + CV steps.

So on my questions:

1. My main question is what is the best approach from here to implement automated hyperparameter tuning? I've found a few different APIs like hype-tune, but they all seem to a) not work well with Keras functional API, or b) have cross-validation baked in, so I can't use my own cross-validation function that I've built.

2. Additionally, I want to check my understanding of what should happen with automated tuning: is it as simple as steps 2-5 above being repeated for each set of candidate hyperparameters, then taking the average performance (val_loss) for all the folds and selecting the set of hyperparameters with the lowest average val_loss?

3. For anyone responding to 1) is there a way to do the more sophisticated hyperparameter tuning (bayesian, gradient descent, evolutionary algorithms).

4. Is this overall, a reasonable approach to the problem? Is there merit to reconsidering CV+holdout, to doing something like nested cross-validation instead?

My relevant code:

Data augmentation function:

def data_aug(X_train,X_test,y_train,y_test,train_batch_size,test_batch_size):
train_datagen = ImageDataGenerator(
    rotation_range=60,
    # rescale=1.0/255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip=True,
    fill_mode='nearest')
test_datagen = ImageDataGenerator()  # nothing applied to test dataset
train_batch = train_datagen.flow(X_train,y_train,batch_size=train_batch_size, seed=33)
test_batch = test_datagen.flow(X_test,y_test,batch_size=test_batch_size, seed=33)
return (train_batch,test_batch)

Cross-validation function

kfold = KFold(n_splits=5, shuffle=True, random_state=33)

cvscores = []
Fold = 1
for train, val in kfold.split(X_train_all, y_train_all):
    gc.collect()
    K.clear_session()
    print ('Fold: ',Fold)
    
    X_train = X_train_all[train]
    X_val = X_train_all[val]
    X_train = X_train.astype('float32')
    X_val = X_val.astype('float32')
    y_train = y_train_all[train]
    y_val = y_train_all[val]
    
    # Data Augmentation and Normalization
    train_batch, val_batch = data_aug(X_train,X_val,y_train,y_val, batch_size, batch_size)
  
    
    # If model checkpoint is used UNCOMMENT THIS
    model_name = 'cnn_keras_Fold_'+str(Fold)+'.h5'
    
    cb = callback()
    
    # create model
    model = create_model()      # CUSTOM ARCHITECTURE
    
    
    # Fit generator for Data Augmentation - UNCOMMENT THIS FOR DATA AUGMENTATION
    model.fit(train_batch, 
              validation_data=val_batch, 
              epochs=epochs, 
              validation_steps= X_val.shape[0] // batch_size, 
              steps_per_epoch= X_train.shape[0] // batch_size, 
              callbacks=cb, 
              verbose=2)
    
        
    # evaluate the model
    scores = model.evaluate(X_val, y_val, verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
    cvscores.append(scores[1] * 100)
    
    Fold += 1

print("%s: %.2f%%" % ("Mean Accuracy: ",np.mean(cvscores)))
print("%s: %.2f%%" % ("Standard Deviation: +/-", np.std(cvscores)))

Answer 1

I have created a package that might give you some ideas.

It's made to work with sklearn kind of estimators, so I don't think it's ready to be used with keras. But you can take a look, maybe you get some insight for your setting:

https://github.com/JaimeArboleda/nestedcvtraining/blob/master/README.md

My approach is to use a nested CV scheme, building ensembles of inner models (or fitting an only model) and estimating the quality on the outer loop. It's prepared to optimize the hyperparameters (and parameters of the post-process) by using the bayesian approach with the skopt library.