I divide my dataset into 6 folds (5 folds CV + 1 fold Test)
I choose 10 different number of hidden neurons
I choose 10 different sets of initial weights
-For each fold (as test)
--For each number of hidden neurons
---For each set of initial weights
----I perform 5 fold CV (4 training and 1 early stop), saving the average performance on Training Validation and Test and the average number of epochs of training
Averaging across the 6 different choices of test folds (10x10x6 -> 10x10) I choose the optimal number of hidden neurons as the value that gives the best mean performance on ten different random sets of initial values.
6 I choose the optimal number of training epochs as the average of training epochs found across the ten iteration of initial weights.
My problem is now how to choose the initial set of weights for the final network that will be trained over all data. Should I choose again ten sets of initial weights and train 10 different networks with the previous defined parameters to find the best or this could result in a overtraining problem?
Here i attached an image of the performance obtained by different net models on the first fold as Test Set