I'm using optimization algorithm to find best structure+inputs of a
patternnet neural network in
MATLAB R2014a using
10-fold cross validation. Where should i initialize weights of my neural network?
*Position_1(rng,configure,randperm?)* for i=1:number_of_loops *Position_2(rng,configure,randperm?)* - repeating cross validation for i=1:number_of_kfolds *Position_3(rng,configure,randperm?)* - Cross validation loop end end
5-fold cross validation (because random selection of cross validation) to have more reliable outputs (average of neural network outputs). Which part is better for weight initialization for neural network (
Position_3) and why?
There is a discussion here that may help :
We should set
rng function in a appropriate position to have unbiased designed and errors. Where is appropriate position for this function? As you know we are calling these codes as cost function of optimization algorithm that is searching for best neurons,layers and inputs combination so i think the logical solution is that we only should have one
rng structure in all calling of cost function by optimization algorithm but with that we can't search all space (maybe some initial weights or indexes for train/test/validation are poor and we don't change it in all system. What is your recommendation based on the above link?
My revised solution is that have
rng in position 1 and weight initialization (
configure) in position 3. It is better than removing
rng from all system but can optimization algorithm compare the cost functions because different random number generator of total cost calling (different weights and indexes of train/test/validation) but same random generator in every calling of cost function by optimization algorithm (cost function loops) ?
finally we will use all models from best cost function that will find by optimization algorithm as a pack (
number_of_loops*number_of_kfolds models) and average between outputs for out-sample data so we are searching for best pack of models not only structures. What do you think about this?
** So briefly we should set position of
configure for neural network weight initialization,
rng for random number generator (that we will save and restore it in every calling of cost function) and
cross validation (after that we separate it to train/test/validation in inner loop).