# Actual impact of W=0 in a neural network

As far as I understand, one of the main claimed problems with initializing e.g. a feed-forward neural network (with several $$\text{tanh}$$ or $$\text{ReLU}$$ layers) with $$W=0$$ is that it doesn't break "network symmetry", meaning, backpropagation would propagate the same error through all such units (i.e. "nudging all weights in the same direction"). This is, I presume, undesirable because we would not be learning "different" calculations through different paths of the network.

However, I'm confused why that even matters in this case given the fact that if $$W$$ ever drops to 0, we will effectively be propagating no gradients at all through the network, since W=0 would multiply all errors from the output and prevent any learning.

Put another way, even if $$W=0$$ does not break network symmetry (wasting calculations and paths in the network) is it correct to say that if $$W=0$$ (e.g. by initialization) we are effectively killing gradients in a neural network, and thus no learning can take place?

• – Sycorax
Commented Jun 2, 2020 at 15:51

It's simpler. The gradient of objective function will be of the form: $$\frac \partial {\partial w_i} \mathcal L=\alpha$$, i.e. all components of a gradient will be equal (within one layer). Since your step in optimization is proportional to the gradient, you'll be making the same step in every direction $$\Delta w_i=\alpha \eta$$, so your weights will be the same within a layer all the time. I denoted a learning rate $$\eta$$. The step size in parameter direction $$\Delta w_i$$ is usually proportional to a gradient and a learning rate.
• Thanks, but wouldn't you argue that the $\Delta$'s (at least in lower layers) will actually be 0? since they multiply by $W=0$ through the chain rule from upper layers?
• Thanks - (1) I think I need to see an actual full derivation of the gradient formula in a lower layer to see what you are saying (if you know of any links showing this, I'd love to see the actual expression). (2) Separately, what do you mean by "your weights will be the same all the time"? You wrote $\Delta w_i = \alpha \Delta x$. That implies that $w_i$ actually changes in value doesn't it? What am I missing?