2
$\begingroup$

As far as I understand, one of the main claimed problems with initializing e.g. a feed-forward neural network (with several $\text{tanh}$ or $\text{ReLU}$ layers) with $W=0$ is that it doesn't break "network symmetry", meaning, backpropagation would propagate the same error through all such units (i.e. "nudging all weights in the same direction"). This is, I presume, undesirable because we would not be learning "different" calculations through different paths of the network.

However, I'm confused why that even matters in this case given the fact that if $W$ ever drops to 0, we will effectively be propagating no gradients at all through the network, since W=0 would multiply all errors from the output and prevent any learning.

Put another way, even if $W=0$ does not break network symmetry (wasting calculations and paths in the network) is it correct to say that if $W=0$ (e.g. by initialization) we are effectively killing gradients in a neural network, and thus no learning can take place?

$\endgroup$
1

1 Answer 1

2
$\begingroup$

It's simpler. The gradient of objective function will be of the form: $\frac \partial {\partial w_i} \mathcal L=\alpha $, i.e. all components of a gradient will be equal (within one layer). Since your step in optimization is proportional to the gradient, you'll be making the same step in every direction $\Delta w_i=\alpha \eta$, so your weights will be the same within a layer all the time. I denoted a learning rate $\eta$. The step size in parameter direction $\Delta w_i$ is usually proportional to a gradient and a learning rate.

$\endgroup$
8
  • $\begingroup$ Thanks, but wouldn't you argue that the $\Delta$'s (at least in lower layers) will actually be 0? since they multiply by $W=0$ through the chain rule from upper layers? $\endgroup$
    – Josh
    Commented May 28, 2020 at 3:08
  • 1
    $\begingroup$ not necessarily, because they'll be multiplied by nonlinear functions of weights, not the weights themselves in the gradient calculation $\endgroup$
    – Aksakal
    Commented May 28, 2020 at 3:10
  • $\begingroup$ Thanks - (1) I think I need to see an actual full derivation of the gradient formula in a lower layer to see what you are saying (if you know of any links showing this, I'd love to see the actual expression). (2) Separately, what do you mean by "your weights will be the same all the time"? You wrote $\Delta w_i = \alpha \Delta x$. That implies that $w_i$ actually changes in value doesn't it? What am I missing? $\endgroup$
    – Josh
    Commented May 28, 2020 at 3:13
  • 1
    $\begingroup$ the same refers to the weights being the same across a layer $\endgroup$
    – Aksakal
    Commented May 28, 2020 at 3:15
  • 1
    $\begingroup$ no. consider a derivative of the weight in the last layer. it's not zero. so the weight will step out of zero, then in the next iteration the next layer's derivative is not zero etc. $\endgroup$
    – Aksakal
    Commented May 28, 2020 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.