Large performance variance when using back-propagation in neural networks for feature extraction I'm using a multi-layered neural network for feature extraction (similar to deep belief network).
I test the performance of my model with cross-validation.
When I'm using back-propagation to train my network, I get a very large variance in the performance.
Seems like it is really crucial which samples (and maybe in what order?) are used for back-propagation.
Is it this a known issue with NN?
I tried using methods from here (I have unequal classes) which helped performance but not the variance.
UPDATE: I'm using 256 features and I have about 2000 training samples. I'm training my network with layer-wise pretraining and then back-propagation. I'm using weight decay (around 0.02) and a learning rate of about 0.01 .
Thanks.
 A: Yes, it is well-known that there are many local minimums. How you initialize the network and how you order the data affect the performance, which means there is a large random component.
Even with one hidden layer, the number of local minimums may be exponential in the number of hidden nodes. This is actually easy to see. Suppose you are approximating a function of one variable which is constant outside two small patches. A neuron which is unsaturated in both patches isn't doing much. So, neurons will tend to saturate in one patch or the other, and it is hard for a neuron to switch where it is unsaturated without going through a stage where it contributes nothing. Therefore you get a local minimum for most of the $2^n$ assignments of neurons to patches.
See Erhan et al, "Why Does Unsupervised Pre-training Help Deep Learning?" section 6.3 "Visualization of Model Trajectories During Learning." Although the graph shows that the local minimums found by random initialization differ greatly from the local minimims found by pre-training, it also shows that there isn't a convergence in parameter space of the models with pre-training (or without). Figure 2 shows there is a wide dispersion of error rates even within a training method.
