I'm using optimization algorithm to find best structure+inputs of a
patternnet neural network in
MATLAB R2014a using
10-fold cross validation. Where should i initialize weights of my neural network?
*Position_1(rng,configure,randperm?)* for i=1:number_of_loops *Position_2(rng,configure,randperm?)* - repeating cross validation for i=1:number_of_kfolds *Position_3(rng,configure,randperm?)* - Cross validation loop end end
5-fold cross validation (because random selection of cross validation) to have more reliable outputs (average of neural network outputs). Which part is better for weight initialization for neural network (
Position_3) and why?
There is a discussion here that may help :
Classification+Optimization Mathworks community
We should set
rng function in a appropriate position to have unbiased designed and errors. Where is appropriate position for this function? As you know we are calling these codes as cost function of optimization algorithm that is searching for best neurons,layers and inputs combination so i think the logical solution is that we only should have one
rng structure in all calling of cost function by optimization algorithm but with that we can't search all space (maybe some initial weights or indexes for train/test/validation are poor and we don't change it in all system. What is your recommendation based on the above link?
My revised solution is that have
rng in position 1 and weight initialization (
configure) in position 3. It is better than removing
rng from all system but can optimization algorithm compare the cost functions because different random number generator of total cost calling (different weights and indexes of train/test/validation) but same random generator in every calling of cost function by optimization algorithm (cost function loops) ?
finally we will use all models from best cost function that will find by optimization algorithm as a pack (
number_of_loops*number_of_kfolds models) and average between outputs for out-sample data so we are searching for best pack of models not only structures. What do you think about this?
** So briefly we should set position of
configure for neural network weight initialization,
rng for random number generator (that we will save and restore it in every calling of cost function) and
cross validation (after that we separate it to train/test/validation in inner loop).
rng. We can initialize indexes of
train/test/validationor weights or both of them in these loops. We can have
rngin position 2,save it and return the saved
rngin every call of these codes by optimization algorithm so we have
number_of_loops=Xweights in every outer loop but these wights are from same saved distribution. The problem is this: we are comparing final output cost of these loops (average) in every calling of these codes by optimization algorithm so we are comparing different structures of neural networks. $\endgroup$
rng, we have different initial weights and indexes for cross validation. Can we compare different structures of neural network and so final chosen models? ( we are using all
number_of_loops*number_of_kfoldsmodels as a black box and average between outputs of these best models (with lowest classification error) $\endgroup$