19
$\begingroup$

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers?

Shouldn't the algorithm calculate the error and work from there, despite the fact that the weights are initially the same?

$\endgroup$
26
$\begingroup$

Symmetry breaking.

If all weights start with equal values and if the solution requires that unequal weights be developed, the system can never learn.

This is because error is propagated back through the weights in proportion to the values of the weights. This means that all hidden units connected directly to the output units will get identical error signals, and, since the weight changes depend on the error signals, the weights from those units to the output units must always be the same. The system is starting out at a kind of unstable equilibrium point that keeps the weights equal, but it is higher than some neighboring points on the error surface, and once it moves away to one of these points, it will never return. We counteract this problem by starting the system with small random weights. Under these conditions symmetry problems of this kind do not arise.

$\endgroup$
  • $\begingroup$ So the initial weights of an NN doesnt only determine the speed at which it trains, but could also be the cause of learning or not learning? $\endgroup$ – user1724140 Dec 4 '12 at 15:23
  • 2
    $\begingroup$ Yes, sometimes, we don't just train (a Neural Network for example) on just one set of weight. One common thing to do, if we have time, is to create several different start weight configurations and train the network on it. Because some times, it can happen that one configuration will end up in a local optima or the weight were unfortunately not really randomize. $\endgroup$ – ThiS Dec 4 '12 at 15:31
11
$\begingroup$

To add to Thierry's answer, you can think of the error as a function of the weight vector i.e. as a function from $R^n \rightarrow R$ which you would like to minimize. The back propagation algorithm works by looking at a local neighborhood of a point and seeing which direction will lead to a smaller error. This will then give you a local minimum.

What you want it a global minimum, but you have no guaranteed way of finding it. And if your surface has several local minima then you may be in trouble.

But if it has only a few then Thierry's strategy should work - performing multiple searches for local minima by starting at randomly selected points should increase the chances of your finding the global minimum.

And in the happy case in which there is only one minimum - any initial weight vector will lead you to it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.