I am testing three different neural network architectures on a dataset to see which architecture performs the best.

My methodology for every architecture is to

  • Split the data into train/test
  • Take the training portion and further split this into train/val
  • Perform 5-fold cross validation to measure how well the model performs on average using the validation set (no changes to the hyper-parameters, all models are initialized after every round of CV).
  • Finally, Train a new model (randomly initialized) on the entire training set and evaluate its performance on the test set which I held out in the first step.

I am unsure what metric I should use to reduce the learning rate (or early stopping) on my models during CV. I can choose validation loss or training loss when learning plateaus. But when I am training my model using all of the training data, I am reducing the learning rate based on training loss since I don't have a validation set.

Any suggestions on what metric I should use?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.