I am testing three different neural network architectures on a dataset to see which architecture performs the best.
My methodology for every architecture is to
- Split the data into train/test
- Take the training portion and further split this into train/val
- Perform 5-fold cross validation to measure how well the model performs on average using the validation set (no changes to the hyper-parameters, all models are initialized after every round of CV).
- Finally, Train a new model (randomly initialized) on the entire training set and evaluate its performance on the test set which I held out in the first step.
I am unsure what metric I should use to reduce the learning rate (or early stopping) on my models during CV. I can choose validation loss or training loss when learning plateaus. But when I am training my model using all of the training data, I am reducing the learning rate based on training loss since I don't have a validation set.
Any suggestions on what metric I should use?