# Cross-validation in neural networks

Let's say you want to use cross-validation to get the best value for a hyperparameter in a neural network. The network has weights as well, all of which must be learned, but it's impractical to cross-validate the values of the parameters (there are too many). Let's say the hyperparameter of interest is the number of units in a one-layer feed-forward neural network.

You can divide the training set into 10 equally-sized pieces, with balanced classes (assuming this is a classification problem).

Here's what I think the general process would be:

For each value of hyperparameter:
For k in 1:10:
make the training set and the validation set
learn the weights of the network;
(stop learning when validation loss stops decreasing)
save the best weight setting and the validation loss
Calculate the average validation loss over all folds
Save the model with lowest average validation loss

Open the final best model
Run the entire training set through it and the test set.