Doubts regarding neural network training using k-fold cross validation?

Question

Facing the following problem: Using k-fold cross validation to select network parameters. Early stopping on the held-out fold to obtain the 'best' weights in terms of classification error. Repeat for k-folds. Use lowest average error (from these k runs) to select best network.

Now I have selected the best network but how do I train it on the whole training set for deployment? In the k runs, best 'weights' were obtained at different iterations. Do I use a separate test set as stopping criteria ? If so would this indicate fairly the generalization performance?

Answer 1

Cross-validation could be used to tune the parameters, rather than to set them differently in k-folds.

i.e. parameters are set to run the model k-folds, and get the cross-validation error (mean, as well as standard deviation sometimes). Then we change the values of the combination of parameter and run the new model k-folds and get another cross-validation error mean. ... In the end, we get a curve (if only one parameter) (or curves, surface) about the error vs parameter value, so that we can find the "best" parameter value to minimize the error (or one standard deviation rule).

Note that some parameters are correlated to each other, like iteration and shrinkage in some models.