Let's say you want to use cross-validation to get the best value for a hyperparameter in a neural network. The network has weights as well, all of which must be learned, but it's impractical to cross-validate the values of the parameters (there are too many). Let's say the hyperparameter of interest is the number of units in a one-layer feed-forward neural network.
You can divide the training set into 10 equally-sized pieces, with balanced classes (assuming this is a classification problem).
Here's what I think the general process would be:
For each value of hyperparameter: For k in 1:10: make the training set and the validation set learn the weights of the network; (stop learning when validation loss stops decreasing) save the best weight setting and the validation loss Calculate the average validation loss over all folds Save the model with lowest average validation loss Open the final best model Run the entire training set through it and the test set.
Am I thinking about this the right way? Is there anything wrong with my process?