# Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?

-
It seems that classical XOR 2-1 net is good example, but I would appreciate some theoretical reasoning. –  user8078 Apr 25 '12 at 19:19

edit see alfa's comment below. I'm not an expert on neural nets, so I'll defer to him.

My understanding is different from the other answers that have been posted here.

I'm pretty sure that backpropagation involves adding to the existing weights, not multiplying. The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation.

My understanding is that there are at least two good reasons not to set the initial weights to zero:

• First, neural networks tend to get stuck in local minima, so it's a good idea to give them many different starting values. You can't do that if they all start at zero.

• Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another.

-
"The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation." - This is only true for neural networks without hidden layers! But you mentioned two other points, that are good arguments against initializing an ANN with equal weights. –  alfa Apr 27 '12 at 14:54

If you thought of the weights as priors, as in a Bayesian network, then you've ruled out any possibility that those inputs could possibly affect the system. Another explanation is that backpropagation identifies the set of weights that minimizes the weighted squared difference between the target and observed values (E). Then how could any gradient descent algorithm be oriented in terms of determining the direction of the system? You are placing yourself on a saddle point of the parameter space.

-
This is the best answer. It is a saddle point. Backpropagation based optimization algorithms will usually stop immediately. In order to calculate the gradient we multiply deltas with weights and the result will always be zero. –  alfa Apr 27 '12 at 14:35