# Large performance variance when using back-propagation in neural networks for feature extraction

I'm using a multi-layered neural network for feature extraction (similar to deep belief network).

I test the performance of my model with cross-validation.

When I'm using back-propagation to train my network, I get a very large variance in the performance.

Seems like it is really crucial which samples (and maybe in what order?) are used for back-propagation.

Is it this a known issue with NN?

I tried using methods from here (I have unequal classes) http://www.springerlink.com/content/2wmmd867cyvbbk3h/

which helped performance but not the variance.

UPDATE: I'm using 256 features and I have about 2000 training samples. I'm training my network with layer-wise pretraining and then back-propagation. I'm using weight decay (around 0.02) and a learning rate of about 0.01 .

Thanks.

-
How large are your training and test sets? Can you give us more detail on how you trained the network (for example, on which hyperparameters you used)? –  Lucas Feb 20 '12 at 9:42
What is large performance variation? 1%? 10%? 25%? Over how many datapoints? –  carlosdc Feb 20 '12 at 19:49

Even with one hidden layer, the number of local minimums may be exponential in the number of hidden nodes. This is actually easy to see. Suppose you are approximating a function of one variable which is constant outside two small patches. A neuron which is unsaturated in both patches isn't doing much. So, neurons will tend to saturate in one patch or the other, and it is hard for a neuron to switch where it is unsaturated without going through a stage where it contributes nothing. Therefore you get a local minimum for most of the $2^n$ assignments of neurons to patches.