I am developing a neural network for pattern recognition in Matlab.
I divide my dataset into 6 folds (5 folds CV + 1 fold Test)
I choose 10 different number of hidden neurons
I choose 10 different sets of initial weights
-For each fold (as test)
--For each number of hidden neurons
---For each set of initial weights
----I perform 5 fold CV (4 training and 1 early stop), saving the average performance on Training Validation and Test and the average number of epochs of training
Averaging across the 6 different choices of test folds (10x10x6 -> 10x10) I choose the optimal number of hidden neurons as the value that gives the best mean performance on ten different random sets of initial values.
6 I choose the optimal number of training epochs as the average of training epochs found across the ten iteration of initial weights.
My problem is now how to choose the initial set of weights for the final network that will be trained over all data.
Should I choose again ten sets of initial weights and train 10 different networks with the previous defined parameters to find the best or this could result in a overtraining problem?
Here i attached an image of the performance obtained by different net models on the first fold as Test Set
Between different CV folds, you're using different data to estimate the model. My suggestion is to characterize how large a difference (modern) initializations make when you train the same model (same number of neurons etc.) on the same data a large number of times. If your training is working well, then the values of holdout set loss should be very similar (because initializations should not make an enormous difference, because training is directing the network toward equivalent parameter configurations).
However, even if the values of the loss are different, this may not be important if the two models are not different in their practical utility. Optimization for its own sake is not really the point. The point is whether or not your model solves your problem.