I'm using optimization algorithm to find best structure+inputs of a patternnet
neural network in MATLAB R2014a
using 10-fold cross validation
. Where should i initialize weights of my neural network?
*Position_1(rng,configure,randperm?)*
for i=1:number_of_loops
*Position_2(rng,configure,randperm?)*
- repeating cross validation
for i=1:number_of_kfolds
*Position_3(rng,configure,randperm?)*
- Cross validation loop
end
end
I'm repeating 5-fold cross validation
(because random selection of cross validation) to have more reliable outputs (average of neural network outputs). Which part is better for weight initialization for neural network (Position_1
,Position_2
or Position_3
) and why?
There is a discussion here that may help :
Classification+Optimization Mathworks community
We should set rng
function in a appropriate position to have unbiased designed and errors. Where is appropriate position for this function? As you know we are calling these codes as cost function of optimization algorithm that is searching for best neurons,layers and inputs combination so i think the logical solution is that we only should have one rng
structure in all calling of cost function by optimization algorithm but with that we can't search all space (maybe some initial weights or indexes for train/test/validation are poor and we don't change it in all system. What is your recommendation based on the above link?
My revised solution is that have rng
in position 1 and weight initialization (configure
) in position 3. It is better than removing rng
from all system but can optimization algorithm compare the cost functions because different random number generator of total cost calling (different weights and indexes of train/test/validation) but same random generator in every calling of cost function by optimization algorithm (cost function loops) ?
finally we will use all models from best cost function that will find by optimization algorithm as a pack (number_of_loops*number_of_kfolds models
) and average between outputs for out-sample data so we are searching for best pack of models not only structures. What do you think about this?
** So briefly we should set position of configure
for neural network weight initialization, rng
for random number generator (that we will save and restore it in every calling of cost function) and index=randperm(number_of_samples)
for cross validation
(after that we separate it to train/test/validation in inner loop).
rng
. We can initialize indexes oftrain/test/validation
or weights or both of them in these loops. We can haverng
in position 2,save it and return the savedrng
in every call of these codes by optimization algorithm so we havenumber_of_loops=X
weights in every outer loop but these wights are from same saved distribution. The problem is this: we are comparing final output cost of these loops (average) in every calling of these codes by optimization algorithm so we are comparing different structures of neural networks. $\endgroup$rng
, we have different initial weights and indexes for cross validation. Can we compare different structures of neural network and so final chosen models? ( we are using allnumber_of_loops*number_of_kfolds
models as a black box and average between outputs of these best models (with lowest classification error) $\endgroup$