4
$\begingroup$

I'm using a multi-layered neural network for feature extraction (similar to deep belief network).

I test the performance of my model with cross-validation.

When I'm using back-propagation to train my network, I get a very large variance in the performance.

Seems like it is really crucial which samples (and maybe in what order?) are used for back-propagation.

Is it this a known issue with NN?

I tried using methods from here (I have unequal classes) which helped performance but not the variance.

UPDATE: I'm using 256 features and I have about 2000 training samples. I'm training my network with layer-wise pretraining and then back-propagation. I'm using weight decay (around 0.02) and a learning rate of about 0.01 .

Thanks.

Improve this question
$\endgroup$
2
  • $\begingroup$ How large are your training and test sets? Can you give us more detail on how you trained the network (for example, on which hyperparameters you used)? $\endgroup$
    – Lucas
    Feb 20, 2012 at 9:42
  • 1
    $\begingroup$ What is large performance variation? 1%? 10%? 25%? Over how many datapoints? $\endgroup$
    – carlosdc
    Feb 20, 2012 at 19:49

1 Answer 1

Reset to default
1
$\begingroup$

Yes, it is well-known that there are many local minimums. How you initialize the network and how you order the data affect the performance, which means there is a large random component.

Even with one hidden layer, the number of local minimums may be exponential in the number of hidden nodes. This is actually easy to see. Suppose you are approximating a function of one variable which is constant outside two small patches. A neuron which is unsaturated in both patches isn't doing much. So, neurons will tend to saturate in one patch or the other, and it is hard for a neuron to switch where it is unsaturated without going through a stage where it contributes nothing. Therefore you get a local minimum for most of the $2^n$ assignments of neurons to patches.

See Erhan et al, "Why Does Unsupervised Pre-training Help Deep Learning?" section 6.3 "Visualization of Model Trajectories During Learning." Although the graph shows that the local minimums found by random initialization differ greatly from the local minimims found by pre-training, it also shows that there isn't a convergence in parameter space of the models with pre-training (or without). Figure 2 shows there is a wide dispersion of error rates even within a training method.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I don't understand your concrete example of hidden producing hidden minima. Can you elaborate or point me to a more detailed explanation? $\endgroup$
    – bayerj
    Dec 2, 2012 at 21:41
  • $\begingroup$ Additionally, it is generally believed that the more hidden neurons you have, the easier it is to escape local minima. That is because the probability of a subset of hidden neurons being near a better minima will increase with the number of units and tend to dominate during training. $\endgroup$
    – bayerj
    Dec 2, 2012 at 21:43
  • $\begingroup$ Not hidden minimums, local minimums. I don't know if the example is in the literature. This is different from Auer et al "Exponentially many local minima for single neurons," which talks about exponentially many minimums in the dimension of the inputs. $\endgroup$
    – Douglas Zare
    Dec 2, 2012 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.