Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ It seems that classical XOR 2-1 net is good example, but I would appreciate some theoretical reasoning. $\endgroup$– user8078Commented Apr 25, 2012 at 19:19
-
3$\begingroup$ A highly general answer, which may or may not be applicable here, is that whenever new weights are multiples of old weights then zero weights cannot be changed. That's fatal to learning. $\endgroup$– Nick CoxCommented Nov 26, 2015 at 13:49
-
1$\begingroup$ There are already good answers available for why not initialize weights to zero. Following link throws some more light on 'how initial weights should be selected?' staff.itee.uq.edu.au/janetw/cmc/chapters/BackProp/index2.html Hope it helps other readers. $\endgroup$– Utsav ChokshiCommented Feb 8, 2016 at 20:48
-
$\begingroup$ @NickCox the weights here are multiples of the next layer of weights, and the last layer is not a multiple of any other weights - so this is not applicable here. $\endgroup$– Maverick MeerkatCommented Aug 11, 2019 at 11:53
-
$\begingroup$ why is: "Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another." true? $\endgroup$– Charlie ParkerCommented Aug 19, 2020 at 19:19
|
Show 1 more comment
8 Answers
$\begingroup$
$\endgroup$
8
edit see alfa's comment below. I'm not an expert on neural nets, so I'll defer to him.
My understanding is different from the other answers that have been posted here.
I'm pretty sure that backpropagation involves adding to the existing weights, not multiplying. The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation.
My understanding is that there are at least two good reasons not to set the initial weights to zero:
First, neural networks tend to get stuck in local minima, so it's a good idea to give them many different starting values. You can't do that if they all start at zero.
Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another.
answered Apr 26, 2012 at 1:40
-
8$\begingroup$ "The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation." - This is only true for neural networks without hidden layers! But you mentioned two other points, that are good arguments against initializing an ANN with equal weights. $\endgroup$– alfaCommented Apr 27, 2012 at 14:54
-
4$\begingroup$ I think it's primarily the second reason--e.g. in a basic fully connected, feed-forward network, if each layer is initialized with the same weights, then as you suggest, all of the nodes follow the same path and are redundant. My impression is most networks with more than a handful of features will not struggle with local minima. Also, random initialization alone doesn't prevent the network from getting stuck, but repeatedly using different random initializations will cue you in on whether a particular iteration had a local minima problem (I think this was implied, but not explicit). $\endgroup$– TahlorCommented Feb 5, 2018 at 20:28
-
$\begingroup$ different inputs break weight symmetry $\endgroup$– user3180Commented Aug 25, 2019 at 3:39
-
3$\begingroup$ why is: "Second, if the neurons start with the same weights, then all the neurons will follow the same gradient, and will always end up doing the same thing as one another." true? $\endgroup$ Commented Aug 19, 2020 at 19:19
-
1$\begingroup$ This answer is wrong, I believe zero initialisation has nothing to do with getting stuck in local minima. That's a completely different problem. The answer by @MaverickMeerkat is correct in my opinion. $\endgroup$ Commented Jul 9 at 13:51
$\begingroup$
$\endgroup$
6
It's a bad idea because of 2 reasons:
If you have sigmoid activation, or anything where $g(0) \neq 0$ then it will cause weights to move "together", limiting the power of back-propagation to search the entire space to find the optimal weights which lower the loss/cost.
If you have $\tanh$ or ReLu activation, or anything where $g(0) = 0$ then all the outputs will be 0, and the gradients for the weights will always be 0. Hence you will not have any learning at all.
Let's demonstrate this (for simplicity I assume a final output layer of 1 neuron):
Forward feed: If all weights are 0's, then the input to the 2nd layer will be the same for all nodes. The outputs of the nodes will be the same, though they will be multiplied by the next set of weights which will be 0, and so the inputs for the next layer will be zero etc., etc. So all the inputs (except the first layer which takes the actual inputs) will be 0, and all the outputs will be the same (0.5 for sigmoid activation and 0 for $\tanh$ and ReLu activation).
Back propagation: Let's examine just the final layer. The final loss ($\mathcal{L}$) depends on the final output of the network ($a^L$, where L denotes the final layer), which depends on the final input before activation ($z^L = W^{L} a^{L-1}$), which depends on the weights of the final layer ($W^{L}$). Now we want to find: $$dW^{L}:= \frac{\partial\mathcal{L}}{\partial W^{L}} = \frac{\partial\mathcal{L}}{\partial a^L} \frac{\partial a^L}{\partial z^L} \frac{\partial z^L}{\partial W^{L}}$$ $\frac{\partial\mathcal{L}}{\partial a}$ is the derivative of the cost function, $\frac{\partial a}{\partial z}$ is the derivative of the activation function. Regardless of what their ($\frac{\partial\mathcal{L}}{\partial a} \frac{\partial a}{\partial z}$) value is, $\frac{\partial z}{\partial W}$ simply equals to the previous layer outputs, i.e. to $a^{L-1}$, but since they are all the same, you get that the final result $dW^{L}$ is a vector with all element equal. So, when you'll update $W^L = W^L - \alpha dW^L$ it will move in the same direction. And the same goes for the previous layers.
Point 2 can be shown from the fact that $a^{L-1}$ will be equal to zero's. Hence your $dW^L$ vector will be full of zeros, and no learning can be achieved.
Update: I made a video about it on YouTube, you can check it out here.
answered Aug 11, 2019 at 13:21
-
2$\begingroup$ best answer here IMO. No other answer even mentions the activation function, which is crucial here. $\endgroup$– hoeflingCommented Mar 16, 2021 at 15:39
-
2$\begingroup$ Agree. Almost a decade for the answer the OP probably expected. All I can say is it is worth the long wait! $\endgroup$– AllohvkCommented Mar 17, 2022 at 14:11
-
$\begingroup$ I don't understand, the gradient of tanh in 0 isn't 1? why do you say it's 0? $\endgroup$ Commented Feb 5, 2023 at 15:12
-
$\begingroup$ @GiladDeutsch The gradient of tanh is 1, but it get's multiplied by other gradients which will be 0. $\frac{\partial a^L}{\partial z^L} \frac{\partial z^L}{\partial W^{L}}=1\cdot 0=0$ - unless you only have a single layer that maps the inputs directly. Then you could actually learn. But if you have more layers, you won't be able to learn, because $\frac{\partial z^L}{\partial W^{L}}=0$. $\endgroup$ Commented Feb 6, 2023 at 12:51
-
1$\begingroup$ @GiladDeutsch consider upvoting if this answer helped you. $\endgroup$ Commented Feb 22, 2023 at 7:33
$\begingroup$
$\endgroup$
5
If you thought of the weights as priors, as in a Bayesian network, then you've ruled out any possibility that those inputs could possibly affect the system. Another explanation is that backpropagation identifies the set of weights that minimizes the weighted squared difference between the target and observed values (E). Then how could any gradient descent algorithm be oriented in terms of determining the direction of the system? You are placing yourself on a saddle point of the parameter space.
-
4$\begingroup$ This is the best answer. It is a saddle point. Backpropagation based optimization algorithms will usually stop immediately. In order to calculate the gradient we multiply deltas with weights and the result will always be zero. $\endgroup$– alfaCommented Apr 27, 2012 at 14:35
-
3$\begingroup$ I think this is a good answer why any particular weight should not be initialized at 0. But the deltas will probably still propagate back--the weights in the output layer do not depend on the weights, so we will have non-zero weights here after the first update. After the next update, these non-zero weights will be used with a delta to adjust the previous layer's weights and so on. I think the bigger issue with initializing a network with all 0's is they're all the same weight, which in your basic fully connected, feed-forward network, is equivalent to having 1 node per layer. $\endgroup$– TahlorCommented Feb 5, 2018 at 21:02
-
$\begingroup$ why is it a problem if all weights have the same value? why would all nodes learn the same function? @Tahlor $\endgroup$ Commented Aug 19, 2020 at 19:21
-
$\begingroup$ @Pinocchio the word "weight" is used with a lack of precision in ML applications. For some models, a weight is used to mean a coefficient value. In this case, no problem provided the observed information is not singular for that set of coefficients. But if a weight is a probabilistic measure, then you're out of luck for the reasons I outlined in the answer. $\endgroup$– AdamOCommented Aug 19, 2020 at 19:34
-
$\begingroup$ what does "In this case, no problem provided the observed information is not singular for that set of coefficients" mean? when I say weight I mean the real number for computing the output function from the image to the labels $w^l_{i,j}$. $\endgroup$ Commented Aug 19, 2020 at 19:38
$\begingroup$
$\endgroup$
5
In each iteration of your backpropagation algorithm, you will update the weights by multiplying the existing weight by a delta determined by backpropagation. If the initial weight value is 0, multiplying it by any value for delta won't change the weight which means each iteration has no effect on the weights you're trying to optimize.
-
4$\begingroup$ >you will update the weights by multiplying the existing weight by a value determined by backpropagation - I don't think so, it's not just multiplication. $\endgroup$– user8078Commented Apr 25, 2012 at 20:22
-
1$\begingroup$ My general point is that if the initial weights are zero they will stay at zero after each iteration of back propagation. $\endgroup$– IdrCommented Apr 25, 2012 at 21:01
-
1$\begingroup$ The 'value determined by backpropagation' is a delta (see e. g. the original paper "Learning representations by back-propagating errors", equation 7). The answer is poorly phrased but it is not completely wrong. $\endgroup$– alfaCommented Apr 27, 2012 at 14:47
-
4$\begingroup$ This is just a wrong answer. The weights will change, but they will change together. This is bad because all hidden units will be identical throughout training and no learning can occur. $\endgroup$ Commented Apr 20, 2019 at 23:25
-
1$\begingroup$ I think this depends on activation function. If you choose tanh, then the final output will be 0, and hence the final layer weights will be 0's, and all the other weights as well. But if you choose a logit, the final output will be $\theta(0) = 0.5$ and hence the final gradient of weights won't be 0, and eventually all the other weights will also won't be 0. $\endgroup$ Commented Aug 11, 2019 at 13:15
$\begingroup$
$\endgroup$
2
It seems to me that one reason why it's bad to initialize weights to the same values (not just zero) is because then for any particular hidden layer all the nodes in this layer would have exactly the same inputs and would therefore stay the same as each other.
-
$\begingroup$ why is " for any particular hidden layer all the nodes in this layer would have exactly the same inputs and would therefore stay the same as each other." true? $\endgroup$ Commented Aug 19, 2020 at 19:22
-
1$\begingroup$ Suppose every node has the same input weights. Then every node in a given layer has the same activation (output value). Then every node in a given layer gets the same weight updates. So the weights are still the same after each update. Adding an additional node in a given layer is then equivalent to having just 1 node in that layer and multiplying the weights by a constant. $\endgroup$– TahlorCommented Aug 19, 2020 at 21:26
$\begingroup$
$\endgroup$
The answer to this is not entirely "Local Minima/Maxima".
When you have more than 1 Hidden Layer and every weight are 0's, no matter how big/small a change in Weight_i will not cause a change in the Output.
This is because delta Weight_i will be absorbed by the next Hidden Layer.
When there is no change in the Output, there is no gradient and hence no direction.
This shares the same traits as a Local Minima/Maxima, but is actually because of 0's, which is technically different
$\begingroup$
$\endgroup$
3
A few reasonable arguments have been provided as to why you should not initialise your network with all weights set to zero. However, I would like to point out that zero initialisation can work — if done correctly!
To start things of, it is not just zero initialisation that is problematic. As a matter of fact, when initialising the weights with any constant, such that $w_{ij} = c$, can be considered problematic. Heck, even an initialisation of the form $w_{ij} = v_j$ would not work very well! The actual problem with all of these approaches is that they induce a certain symmetry among neurons. Concretely, the activations will be the same for all neurons in a single layer. To see this, consider the (pre-)activation, $s_i$, of neuron $i$ in some layer with weight matrix $\boldsymbol{W}$ and inputs $\boldsymbol{x}$:
$$s_i = \boldsymbol{w}_{i:} \cdot \boldsymbol{x} = \sum_j v_j x_j,$$
where $\boldsymbol{w}_{i:}$ is the $i$-th row of $\boldsymbol{W}$. Note that $s_i$ is completely independent of the index $i$ because all rows of $\boldsymbol{W}$ are the same. This means that all neurons will have the same outputs, such that $\forall i,j : s_i = s_j$.
Having symmetric (i.e. duplicated) neurons at initialisation might not seem so bad. However, the network will never be able to actually break the symmetry. To see this, we have to consider the back-propagation of errors in terms of $\boldsymbol{\delta} = \frac{\partial L}{\partial \boldsymbol{s}}$. Applying the chain rule, you should be able to find that
$$\delta_j^- = \phi'(s_j)\, \boldsymbol{\delta} \cdot \boldsymbol{w}_{:j} = \phi'(s) \sum_i \delta_i v_j,$$
where $w_{:j}$ is the $j$-th column of $\boldsymbol{W}$ and $\phi$ is the activation function. Also, we used $s = s_j$ given that the (pre-)activations are independent of the individual neurons anyway.
If we now want to compute the update for a single entry in our weight matrix, we find
$$\Delta w_{ij} = \phi(s_i) \delta_j = \phi(s) \delta_j.$$
Again, note that the update will be identical for every row! This means that the all neurons will output the same value, even after being updated. Therefore initialising every column in your weight matrices with a constant effectively reduces the effective number of neurons in a each layer to 1, which is generally something you want to avoid.
The most common way to break this symmetry is to use randomly sampled values to initialise the weights. However, it is by no means the only way. The entire analysis "ignores" the bias parameters or at least assumes that they are initialised to be zeros, which is common practice. However, if we simply initialise the bias parameters by sampling from a random distribution, the symmetry of neurons can be broken, even if all initial weights are zero.
TL;DR: the problem is symmetry, which reduces a layer to a single neuron. One solution is random weights, but also biases can be used to break symmetry.
answered Oct 27, 2021 at 15:17
Mr Tsjolder from codidactMr Tsjolder from codidact
2,1512 gold badges18 silver badges31 bronze badges
-
$\begingroup$ There is also this paper of course: proceedings.mlr.press/v119/blumenfeld20a.html $\endgroup$ Commented Jun 12, 2023 at 5:28
-
$\begingroup$ the symmetry doesnt hold for the fist layer connected to input . So after first backpropogatio there will be no symmetry in the network $\endgroup$ Commented Feb 19 at 8:51
-
$\begingroup$ @user179156 You are right, the symmetry will typically be broken at the input and the output layer. That is why some people choose to initialise the weights of the last layer to be all zero (e.g. see section 2.3 of this paper) $\endgroup$ Commented Feb 22 at 15:58
$\begingroup$
$\endgroup$
Main problem with initialization of all weights to zero mathematically leads to either the neuron values are zero (for multi layers) or the delta would be zero. In one of the comments by @alfa in the above answers already a hint is provided, it is mentioned that the product of weights and delta needs to be zero. This would essentially mean that for the gradient descent this is on the top of the hill right at its peak and it is unable to break the symmetry. Randomness will break this symmetry and one would reach local minimum. Even if we perturb the weight(s) a little we would be on the track. Reference: Learning from data Lecture 10.
