I'm using a multi-layered neural network for feature extraction (similar to deep belief network).
I test the performance of my model with cross-validation.
When I'm using back-propagation to train my network, I get a very large variance in the performance.
Seems like it is really crucial which samples (and maybe in what order?) are used for back-propagation.
Is it this a known issue with NN?
I tried using methods from here (I have unequal classes) http://www.springerlink.com/content/2wmmd867cyvbbk3h/
which helped performance but not the variance.
UPDATE: I'm using 256 features and I have about 2000 training samples. I'm training my network with layer-wise pretraining and then back-propagation. I'm using weight decay (around 0.02) and a learning rate of about 0.01 .