30
$\begingroup$

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?

$\endgroup$
  • $\begingroup$ It seems that classical XOR 2-1 net is good example, but I would appreciate some theoretical reasoning. $\endgroup$ – user8078 Apr 25 '12 at 19:19
  • 1
    $\begingroup$ A highly general answer, which may or may not be applicable here, is that whenever new weights are multiples of old weights then zero weights cannot be changed. That's fatal to learning. $\endgroup$ – Nick Cox Nov 26 '15 at 13:49
  • $\begingroup$ There are already good answers available for why not initialize weights to zero. Following link throws some more light on 'how initial weights should be selected?' staff.itee.uq.edu.au/janetw/cmc/chapters/BackProp/index2.html Hope it helps other readers. $\endgroup$ – doer_uvc Feb 8 '16 at 20:48
  • $\begingroup$ @NickCox the weights here are multiples of the next layer of weights, and the last layer is not a multiple of any other weights - so this is not applicable here. $\endgroup$ – David Refaeli Aug 11 at 11:53
21
$\begingroup$

edit see alfa's comment below. I'm not an expert on neural nets, so I'll defer to him.

My understanding is different from the other answers that have been posted here.

I'm pretty sure that backpropagation involves adding to the existing weights, not multiplying. The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation.

My understanding is that there are at least two good reasons not to set the initial weights to zero:

  • First, neural networks tend to get stuck in local minima, so it's a good idea to give them many different starting values. You can't do that if they all start at zero.

  • Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another.

$\endgroup$
  • 6
    $\begingroup$ "The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation." - This is only true for neural networks without hidden layers! But you mentioned two other points, that are good arguments against initializing an ANN with equal weights. $\endgroup$ – alfa Apr 27 '12 at 14:54
  • 1
    $\begingroup$ I think it's primarily the second reason--e.g. in a basic fully connected, feed-forward network, if each layer is initialized with the same weights, then as you suggest, all of the nodes follow the same path and are redundant. My impression is most networks with more than a handful of features will not struggle with local minima. Also, random initialization alone doesn't prevent the network from getting stuck, but repeatedly using different random initializations will cue you in on whether a particular iteration had a local minima problem (I think this was implied, but not explicit). $\endgroup$ – Tahlor Feb 5 '18 at 20:28
  • $\begingroup$ the 2nd reason is false? open.wolframcloud.com/env/… $\endgroup$ – user3180 Aug 25 at 3:39
  • $\begingroup$ different inputs break weight symmetry $\endgroup$ – user3180 Aug 25 at 3:39
  • $\begingroup$ you add the derivatives, and the derivatives are multiplication used through chain rule. $\endgroup$ – David Refaeli Sep 20 at 15:15
7
$\begingroup$

If you thought of the weights as priors, as in a Bayesian network, then you've ruled out any possibility that those inputs could possibly affect the system. Another explanation is that backpropagation identifies the set of weights that minimizes the weighted squared difference between the target and observed values (E). Then how could any gradient descent algorithm be oriented in terms of determining the direction of the system? You are placing yourself on a saddle point of the parameter space.

$\endgroup$
  • 1
    $\begingroup$ This is the best answer. It is a saddle point. Backpropagation based optimization algorithms will usually stop immediately. In order to calculate the gradient we multiply deltas with weights and the result will always be zero. $\endgroup$ – alfa Apr 27 '12 at 14:35
  • $\begingroup$ I think this is a good answer why any particular weight should not be initialized at 0. But the deltas will probably still propagate back--the weights in the output layer do not depend on the weights, so we will have non-zero weights here after the first update. After the next update, these non-zero weights will be used with a delta to adjust the previous layer's weights and so on. I think the bigger issue with initializing a network with all 0's is they're all the same weight, which in your basic fully connected, feed-forward network, is equivalent to having 1 node per layer. $\endgroup$ – Tahlor Feb 5 '18 at 21:02
6
$\begingroup$

In each iteration of your backpropagation algorithm, you will update the weights by multiplying the existing weight by a delta determined by backpropagation. If the initial weight value is 0, multiplying it by any value for delta won't change the weight which means each iteration has no effect on the weights you're trying to optimize.

$\endgroup$
  • 2
    $\begingroup$ >you will update the weights by multiplying the existing weight by a value determined by backpropagation - I don't think so, it's not just multiplication. $\endgroup$ – user8078 Apr 25 '12 at 20:22
  • 1
    $\begingroup$ My general point is that if the initial weights are zero they will stay at zero after each iteration of back propagation. $\endgroup$ – Idr Apr 25 '12 at 21:01
  • $\begingroup$ The 'value determined by backpropagation' is a delta (see e. g. the original paper "Learning representations by back-propagating errors", equation 7). The answer is poorly phrased but it is not completely wrong. $\endgroup$ – alfa Apr 27 '12 at 14:47
  • 1
    $\begingroup$ This is just a wrong answer. The weights will change, but they will change together. This is bad because all hidden units will be identical throughout training and no learning can occur. $\endgroup$ – Amin Sammara Apr 20 at 23:25
  • $\begingroup$ I think this depends on activation function. If you choose tanh, then the final output will be 0, and hence the final layer weights will be 0's, and all the other weights as well. But if you choose a logit, the final output will be $\theta(0) = 0.5$ and hence the final gradient of weights won't be 0, and eventually all the other weights will also won't be 0. $\endgroup$ – David Refaeli Aug 11 at 13:15
2
$\begingroup$

It seems to me that one reason why it's bad to initialize weights to the same values (not just zero) is because then for any particular hidden layer all the nodes in this layer would have exactly the same inputs and would therefore stay the same as each other.

$\endgroup$
1
$\begingroup$

The answer to this is not entirely "Local Minima/Maxima".

When you have more than 1 Hidden Layer and every weight are 0's, no matter how big/small a change in Weight_i will not cause a change in the Output.

This is because delta Weight_i will be absorbed by the next Hidden Layer.

When there is no change in the Output, there is no gradient and hence no direction.

This shares the same traits as a Local Minima/Maxima, but is actually because of 0's, which is technically different

$\endgroup$
0
$\begingroup$

Main problem with initialization of all weights to zero mathematically leads to either the neuron values are zero (for multi layers) or the delta would be zero. In one of the comments by @alfa in the above answers already a hint is provided, it is mentioned that the product of weights and delta needs to be zero. This would essentially mean that for the gradient descent this is on the top of the hill right at its peak and it is unable to break the symmetry. Randomness will break this symmetry and one would reach local minimum. Even if we perturb the weight(s) a little we would be on the track. Reference: Learning from data Lecture 10.

$\endgroup$
0
$\begingroup$

It's a bad idea because of 2 reasons:

  1. If you have sigmoid activation, or anything where $g(0) \neq 0$ then it will cause weights to move "together", limiting the power of back-propagation to search the entire space to find the optimal weights which lower the loss/cost.

  2. If you have $\tanh$ or ReLu activation, or anything where $g(0) = 0$ then all the outputs will be 0, and the gradients for the weights will always be 0. Hence you will not have any learning at all.

Let's demonstrate this (for simplicity I assume a final output layer of 1 neuron):

Forward feed: If all weights are 0's, then the input to the 2nd layer will be the same for all nodes. The outputs of the nodes will be the same, though they will be multiplied by the next set of weights which will be 0, and so the inputs for the next layer will be zero etc., etc. So all the inputs (except the first layer which takes the actual inputs) will be 0, and all the outputs will be the same (0.5 for sigmoid activation and 0 for $\tanh$ and ReLu activation).

Back propagation: Let's examine just the final layer. The final loss ($\mathcal{L}$) depends on the final output of the network ($a^L$, where L denotes the final layer), which depends on the final input before activation ($z^L = W^{L} a^{L-1}$), which depends on the weights of the final layer ($W^{L}$). Now we want to find: $$dW^{L}:= \frac{\partial\mathcal{L}}{\partial W^{L}} = \frac{\partial\mathcal{L}}{\partial a^L} \frac{\partial a^L}{\partial z^L} \frac{\partial z^L}{\partial W^{L}}$$ $\frac{\partial\mathcal{L}}{\partial a}$ is the derivative of the cost function, $\frac{\partial a}{\partial z}$ is the derivative of the activation function. Regardless of what their ($\frac{\partial\mathcal{L}}{\partial a} \frac{\partial a}{\partial z}$) value is, $\frac{\partial z}{\partial W}$ simply equals to the previous layer outputs, i.e. to $a^{L-1}$, but since they are all the same, you get that the final result $dW^{L}$ is a vector with all element equal. So, when you'll update $W^L = W^L - \alpha dW^L$ it will move in the same direction. And the same goes for the previous layers.

Point 2 can be shown from the fact that $a^{L-1}$ will be equal to zero's. Hence your $dW^L$ vector will be full of zeros, and no learning can be achieved.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.