Danger of setting all initial weights to zero in Backpropagation Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
 A: In each iteration of your backpropagation algorithm, you will update the weights by multiplying the existing weight by a delta determined by backpropagation. If the initial weight value is 0, multiplying it by any value for delta won't change the weight which means each iteration has no effect on the weights you're trying to optimize.
A: It seems to me that one reason why it's bad to initialize weights to the same values (not just zero) is because then for any particular hidden layer all the nodes in this layer would have exactly the same inputs and would therefore stay the same as each other. 
A: edit see alfa's comment below. I'm not an expert on neural nets, so I'll defer to him. 
My understanding is different from the other answers that have been posted here.
I'm pretty sure that backpropagation involves adding to the existing weights, not multiplying.  The amount that you add is specified by the delta rule.  Note that wij doesn't appear on the right-hand-side of the equation.
My understanding is that there are at least two good reasons not to set the initial weights to zero:


*

*First, neural networks tend to get stuck in local minima, so it's a good idea to give them many different starting values.  You can't do that if they all start at zero.

*Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another.
A: It's a bad idea because of 2 reasons:

*

*If you have sigmoid activation, or anything where $g(0) \neq 0$ then it will cause weights to move "together", limiting the power of back-propagation to search the entire space to find the optimal weights which lower the loss/cost.


*If you have $\tanh$ or ReLu activation, or anything where $g(0) = 0$ then all the outputs will be 0, and the gradients for the weights will always be 0. Hence you will not have any learning at all.
Let's demonstrate this (for simplicity I assume a final output layer of 1 neuron):
Forward feed: If all weights are 0's, then the input to the 2nd layer will be the same for all nodes. The outputs of the nodes will be the same, though they will be multiplied by the next set of weights which will be 0, and so the inputs for the next layer will be zero etc., etc. So all the inputs (except the first layer which takes the actual inputs) will be 0, and all the outputs will be the same (0.5 for sigmoid activation and 0 for $\tanh$ and ReLu activation).
Back propagation: Let's examine just the final layer. The final loss ($\mathcal{L}$) depends on the final output of the network ($a^L$, where L denotes the final layer), which depends on the final input before activation ($z^L = W^{L} a^{L-1}$), which depends on the weights of the final layer ($W^{L}$). Now we want to find:
$$dW^{L}:= \frac{\partial\mathcal{L}}{\partial W^{L}} = \frac{\partial\mathcal{L}}{\partial a^L} \frac{\partial a^L}{\partial z^L} \frac{\partial z^L}{\partial W^{L}}$$
$\frac{\partial\mathcal{L}}{\partial a}$ is the derivative of the cost function, $\frac{\partial a}{\partial z}$ is the derivative of the activation function. Regardless of what their ($\frac{\partial\mathcal{L}}{\partial a} \frac{\partial a}{\partial z}$) value is, $\frac{\partial z}{\partial W}$ simply equals to the previous layer outputs, i.e. to $a^{L-1}$, but since they are all the same, you get that the final result $dW^{L}$ is a vector with all element equal.
So, when you'll update $W^L = W^L - \alpha dW^L$ it will move in the same direction. And the same goes for the previous layers.
Point 2 can be shown from the fact that $a^{L-1}$ will be equal to zero's. Hence your $dW^L$ vector will be full of zeros, and no learning can be achieved.
Update: I made a video about it on YouTube, you can check it out here.
A: If you thought of the weights as priors, as in a Bayesian network, then you've ruled out any possibility that those inputs could possibly affect the system. Another explanation is that backpropagation identifies the set of weights that minimizes the weighted squared difference between the target and observed values (E). Then how could any gradient descent algorithm be oriented in terms of determining the direction of the system? You are placing yourself on a saddle point of the parameter space.
A: The answer to this is not entirely "Local Minima/Maxima".
When you have more than 1 Hidden Layer and every weight are 0's, no matter how big/small a change in Weight_i will not cause a change in the Output.
This is because delta Weight_i will be absorbed by the next Hidden Layer.
When there is no change in the Output, there is no gradient and hence no direction.
This shares the same traits as a Local Minima/Maxima, but is actually because of 0's, which is technically different
A: A few reasonable arguments have been provided as to why you should not initialise your network with all weights set to zero.
However, I would like to point out that zero initialisation can work — if done correctly!
To start things of, it is not just zero initialisation that is problematic.
As a matter of fact, when initialising the weights with any constant, such that $w_{ij} = c$, can be considered problematic.
Heck, even an initialisation of the form $w_{ij} = v_j$ would not work very well! The actual problem with all of these approaches is that they induce a certain symmetry among neurons. Concretely, the activations will be the same for all neurons in a single layer. To see this, consider the (pre-)activation, $s_i$, of neuron $i$ in some layer with weight matrix $\boldsymbol{W}$ and inputs $\boldsymbol{x}$:
$$s_i = \boldsymbol{w}_{i:} \cdot \boldsymbol{x} = \sum_j v_j x_j,$$
where $\boldsymbol{w}_{i:}$ is the $i$-th row of $\boldsymbol{W}$.
Note that $s_i$ is completely independent of the index $i$.
This means that all neurons will have the same outputs.
Having symmetric (i.e. duplicated) neurons at initialisation might not seem so bad.
However, the network will never be able to actually break the symmetry.
To see this, we have to consider the back-propagation of errors in terms of $\boldsymbol{\delta} = \frac{\partial L}{\partial \boldsymbol{s}}$. Applying the chain rule, you should be able to find that
$$\delta_j^- = \phi'(s_j)\, \boldsymbol{\delta} \cdot \boldsymbol{w}_{:j} = \phi'(s) \sum_i \delta_i v_j,$$
where $w_{:j}$ is the $j$-th column of $\boldsymbol{W}$ and $\phi$ is the activation function. Also, we used $s = s_j$ given that the (pre-)activations are independent of the individual neurons anyway.
If we now want to compute the update for a single entry in our weight matrix, we find
$$\Delta w_{ij} = \phi(s_i) \delta_j = \phi(s) \delta_j.$$
Again, note that the update will be identical for every row!
This means that the all neurons will output the same value, even after being updated.
Therefore initialising every column in your weight matrices with a constant effectively reduces the effective number of neurons in a each layer to 1, which is generally something you want to avoid.

The most common way to break this symmetry is to use randomly sampled values to initialise the weights.
However, it is by no means the only way.
The entire analysis "ignores" the bias parameters or at least assumes that they are initialised to be zeros, which is common practice.
However, if we simply initialise the bias parameters by sampling from a random distribution, the symmetry of neurons can be broken, even if all initial weights are zero.
TL;DR: the problem is symmetry, which reduces a layer to a single neuron. One solution is random weights, but also biases can be used to break symmetry.
A: Main problem with initialization of all weights to zero mathematically leads to either the neuron values are zero (for multi layers) or the delta would be zero. In one of the comments by @alfa in the above answers already a hint is provided,  it is mentioned that the product of weights and delta needs to be zero.   This would essentially mean that for the gradient descent this is on the top of the hill right at its peak and it is unable to break the symmetry.  Randomness will break this symmetry and one would reach local minimum. Even if we perturb the weight(s) a little we would be on the track.  Reference: Learning from data Lecture 10. 
