Danger of setting all initial weights to zero in Backpropagation

Question

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?

Answer 1

edit see alfa's comment below. I'm not an expert on neural nets, so I'll defer to him.

My understanding is different from the other answers that have been posted here.

I'm pretty sure that backpropagation involves adding to the existing weights, not multiplying. The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation.

My understanding is that there are at least two good reasons not to set the initial weights to zero:

• First, neural networks tend to get stuck in local minima, so it's a good idea to give them many different starting values. You can't do that if they all start at zero.

• Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another.

Answer 2

If you thought of the weights as priors, as in a Bayesian network, then you've ruled out any possibility that those inputs could possibly affect the system. Another explanation is that backpropagation identifies the set of weights that minimizes the weighted squared difference between the target and observed values (E). Then how could any gradient descent algorithm be oriented in terms of determining the direction of the system? You are placing yourself on a saddle point of the parameter space.

Answer 3

In each iteration of your backpropagation algorithm, you will update the weights by multiplying the existing weight by a delta determined by backpropagation. If the initial weight value is 0, multiplying it by any value for delta won't change the weight which means each iteration has no effect on the weights you're trying to optimize.