31
$\begingroup$

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers?

Shouldn't the algorithm calculate the error and work from there, despite the fact that the weights are initially the same?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
34
$\begingroup$

Symmetry breaking.

If all weights start with equal values and if the solution requires that unequal weights be developed, the system can never learn.

This is because error is propagated back through the weights in proportion to the values of the weights. This means that all hidden units connected directly to the output units will get identical error signals, and, since the weight changes depend on the error signals, the weights from those units to the output units must always be the same. The system is starting out at a kind of unstable equilibrium point that keeps the weights equal, but it is higher than some neighboring points on the error surface, and once it moves away to one of these points, it will never return. We counteract this problem by starting the system with small random weights. Under these conditions symmetry problems of this kind do not arise.

Improve this answer
$\endgroup$
7
  • $\begingroup$ So the initial weights of an NN doesnt only determine the speed at which it trains, but could also be the cause of learning or not learning? $\endgroup$
    – user1724140
    Commented Dec 4, 2012 at 15:23
  • 2
    $\begingroup$ Yes, sometimes, we don't just train (a Neural Network for example) on just one set of weight. One common thing to do, if we have time, is to create several different start weight configurations and train the network on it. Because some times, it can happen that one configuration will end up in a local optima or the weight were unfortunately not really randomize. $\endgroup$
    – ThiS
    Commented Dec 4, 2012 at 15:31
  • 3
    $\begingroup$ Isnt this wrong, because different inputs can still break symmetry. see: open.wolframcloud.com/env/… $\endgroup$
    – user3180
    Commented Aug 25, 2019 at 3:38
  • 1
    $\begingroup$ +1 on @user3180's comment. I am also totally confused by this answer. Possible reasoning might be that the possible solutions reached from such initialization might still be some small local minima around that $\endgroup$
    – Naman
    Commented Jan 25, 2022 at 18:15
  • 1
    $\begingroup$ @user3180 Your link is broken. Can you clarify? Equal initialization makes hidden layer neurons indistinguishable from each other, so how could they diversify? $\endgroup$
    – user76284
    Commented Apr 25, 2023 at 22:41
15
$\begingroup$

To add to Thierry's answer, you can think of the error as a function of the weight vector i.e. as a function from $R^n \rightarrow R$ which you would like to minimize. The back propagation algorithm works by looking at a local neighborhood of a point and seeing which direction will lead to a smaller error. This will then give you a local minimum.

What you want it a global minimum, but you have no guaranteed way of finding it. And if your surface has several local minima then you may be in trouble.

But if it has only a few then Thierry's strategy should work - performing multiple searches for local minima by starting at randomly selected points should increase the chances of your finding the global minimum.

And in the happy case in which there is only one minimum - any initial weight vector will lead you to it.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Just to clarify, the last sentence here isn't strictly true for e.g. constant/zero initializations (for the reason of the symmetry argument given in the above answer). If we're considering IID initializations, this holds almost surely. $\endgroup$
    – eric.mitchell
    Commented Aug 9, 2021 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.