3
$\begingroup$

The notes for Stanford's online course on CNN's mention not to initialize all the weights to zero, because:

… if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

I have been trying to understand this, but it is still not clear to me. I have searched on the internet, where some sources have only managed to confuse me further, like this stackoverflow answer and some other sources claiming that if the weights are initialized to zero, the weights would not update at all during the whole training process (while according to the notes linked above, the weights will update, but the updates to all the weights will be equal)

I tried doing some calculations on my own, which I have attached below.

Derivation

If I have not made any error then then this derivation is showing that all the weights would undergo updates of equal magnitude, but the weights in the row of the Weight Matrix corresponding to the true class would be updated in the opposite direction compared to the other weights. So if the other weights are increased by delta, the weights in the row corresponding to the true class would be decreased by delta. So when the notes say "(all the neurons would) undergo the exact same parameter updates", they are actually talking about the magnitude – the direction can be different for some weights.

I am no expert in Mathematics, so, if there is any error in the derivation above, or if I am not interpreting it right, your criticism of it would be appreciated. Also, if there is a more intuitive way to see this, that would be helpful

Improve this question
$\endgroup$
6
  • $\begingroup$ Do you want all the magnitudes to be the same? $\endgroup$
    – Adrian Keister
    Commented Mar 13, 2020 at 19:51
  • $\begingroup$ The magnitude are all same, but the directions, as it seems from the derivation I did, might not be the same. But in the notes, they have not clarified whether they are talking about the directions or the magnitudes when they are saying "(all the neurons would) undergo the exact same parameter updates" $\endgroup$
    – Wololo
    Commented Mar 13, 2020 at 20:26
  • 1
    $\begingroup$ Right, but I'm asking the question, because I would think, off-hand, that you would definitely NOT want even the magnitudes to update the same. You would want the weights to update as they need in order to fit your function. Updating the same magnitude sounds like an artificial restriction of freedom - I doubt that's desirable. $\endgroup$
    – Adrian Keister
    Commented Mar 13, 2020 at 20:31
  • $\begingroup$ Oh, right! Yes, it is not desirable $\endgroup$
    – Wololo
    Commented Mar 13, 2020 at 20:37
  • $\begingroup$ You do the math for single layer net. What about more layers, weights will not change and remain zeros? Check it, I did the same math and it appears weights don't change, which confuses me just like you.... $\endgroup$
    – Marie
    Commented Mar 27, 2020 at 20:03

0

Reset to default