I have several questions I'm going to try to bundle into one here. I am currently trying to implement convolutional neural network training on a public image dataset. I am trying to test and compare several different CNN architectures, eg. InceptionV3, ResNet, DenseNet, and a few others. To that end, I am trying to keep everything else in the code and the approach constant, only changing the architecture. I am using 5-fold cross validation with a hold-out test set of 20%.
I am running into a challenge to implement automated hyperparameter tuning however. My code is architected as follows.
- Divide data into train (80%) and test (20%)
- Create KFolds (n=5) from train data (80% train, 20% val for each fold).
- For loop that iterates through each fold.
- Within the for loop, I am using a custom data_aumentation function which takes the X_train, X_val, y_train, y_val, and augments the training data using ImageDataGenerator().
- Fit the model in each fold, save best model performing model by val_loss.
- Ensemble models and evaluate on the test set (initial hold-out 20%).
I am using Keras functional API for my model creation. Currently I am setting the hyperparameters manually prior to the model fitting + CV steps.
So on my questions:
My main question is what is the best approach from here to implement automated hyperparameter tuning? I've found a few different APIs like hype-tune, but they all seem to a) not work well with Keras functional API, or b) have cross-validation baked in, so I can't use my own cross-validation function that I've built.
Additionally, I want to check my understanding of what should happen with automated tuning: is it as simple as steps 2-5 above being repeated for each set of candidate hyperparameters, then taking the average performance (val_loss) for all the folds and selecting the set of hyperparameters with the lowest average val_loss?
For anyone responding to 1) is there a way to do the more sophisticated hyperparameter tuning (bayesian, gradient descent, evolutionary algorithms).
Is this overall, a reasonable approach to the problem? Is there merit to reconsidering CV+holdout, to doing something like nested cross-validation instead?
My relevant code:
Data augmentation function:
def data_aug(X_train,X_test,y_train,y_test,train_batch_size,test_batch_size): train_datagen = ImageDataGenerator( rotation_range=60, # rescale=1.0/255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True, vertical_flip=True, fill_mode='nearest') test_datagen = ImageDataGenerator() # nothing applied to test dataset train_batch = train_datagen.flow(X_train,y_train,batch_size=train_batch_size, seed=33) test_batch = test_datagen.flow(X_test,y_test,batch_size=test_batch_size, seed=33) return (train_batch,test_batch)
kfold = KFold(n_splits=5, shuffle=True, random_state=33) cvscores =  Fold = 1 for train, val in kfold.split(X_train_all, y_train_all): gc.collect() K.clear_session() print ('Fold: ',Fold) X_train = X_train_all[train] X_val = X_train_all[val] X_train = X_train.astype('float32') X_val = X_val.astype('float32') y_train = y_train_all[train] y_val = y_train_all[val] # Data Augmentation and Normalization train_batch, val_batch = data_aug(X_train,X_val,y_train,y_val, batch_size, batch_size) # If model checkpoint is used UNCOMMENT THIS model_name = 'cnn_keras_Fold_'+str(Fold)+'.h5' cb = callback() # create model model = create_model() # CUSTOM ARCHITECTURE # Fit generator for Data Augmentation - UNCOMMENT THIS FOR DATA AUGMENTATION model.fit(train_batch, validation_data=val_batch, epochs=epochs, validation_steps= X_val.shape // batch_size, steps_per_epoch= X_train.shape // batch_size, callbacks=cb, verbose=2) # evaluate the model scores = model.evaluate(X_val, y_val, verbose=0) print("%s: %.2f%%" % (model.metrics_names, scores*100)) cvscores.append(scores * 100) Fold += 1 print("%s: %.2f%%" % ("Mean Accuracy: ",np.mean(cvscores))) print("%s: %.2f%%" % ("Standard Deviation: +/-", np.std(cvscores)))