Finding best neural network structure and inputs using optimization algorithm and cross-validation

I'm using optimization algorithm to find best structure+inputs of a patternnet neural network in MATLAB R2014a using 10-fold cross validation. Where should i initialize weights of my neural network?

*Position_1(rng,configure,randperm?)*

for i=1:number_of_loops

*Position_2(rng,configure,randperm?)*
- repeating cross validation

for i=1:number_of_kfolds

*Position_3(rng,configure,randperm?)*
- Cross validation loop

end
end


I'm repeating 5-fold cross validation (because random selection of cross validation) to have more reliable outputs (average of neural network outputs). Which part is better for weight initialization for neural network (Position_1,Position_2 or Position_3) and why?

There is a discussion here that may help :

Classification+Optimization Mathworks community

We should set rng function in a appropriate position to have unbiased designed and errors. Where is appropriate position for this function? As you know we are calling these codes as cost function of optimization algorithm that is searching for best neurons,layers and inputs combination so i think the logical solution is that we only should have one rng structure in all calling of cost function by optimization algorithm but with that we can't search all space (maybe some initial weights or indexes for train/test/validation are poor and we don't change it in all system. What is your recommendation based on the above link?

My revised solution is that have rng in position 1 and weight initialization (configure) in position 3. It is better than removing rng from all system but can optimization algorithm compare the cost functions because different random number generator of total cost calling (different weights and indexes of train/test/validation) but same random generator in every calling of cost function by optimization algorithm (cost function loops) ?

finally we will use all models from best cost function that will find by optimization algorithm as a pack (number_of_loops*number_of_kfolds models) and average between outputs for out-sample data so we are searching for best pack of models not only structures. What do you think about this?

** So briefly we should set position of configure for neural network weight initialization, rng for random number generator (that we will save and restore it in every calling of cost function) and index=randperm(number_of_samples) for cross validation (after that we separate it to train/test/validation in inner loop).

• If your rng function does not initialize weights, what does it do? You write "We should set rng function in a appropriate position to have unbiased designed and errors." What are "unbiased designed and errors"? – user31264 Sep 9 '14 at 14:35
• Thank you for your comment. it depends on position of rng. We can initialize indexes of train/test/validation or weights or both of them in these loops. We can have rng in position 2,save it and return the saved rng in every call of these codes by optimization algorithm so we have number_of_loops=X weights in every outer loop but these wights are from same saved distribution. The problem is this: we are comparing final output cost of these loops (average) in every calling of these codes by optimization algorithm so we are comparing different structures of neural networks. – user2991243 Sep 9 '14 at 14:41
• If we don't set any rng, we have different initial weights and indexes for cross validation. Can we compare different structures of neural network and so final chosen models? ( we are using all number_of_loops*number_of_kfolds models as a black box and average between outputs of these best models (with lowest classification error) – user2991243 Sep 9 '14 at 14:42