I read here the following:

  • Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving data that is not zero-centered. This has implications on the dynamics during gradient descent, because if the data coming into a neuron is always positive (e.g. $x > 0$ elementwise in $f = w^Tx + b$)), then the gradient on the weights $w$ will during backpropagation become either all be positive, or all negative (depending on the gradient of the whole expression $f$). This could introduce undesirable zig-zagging dynamics in the gradient updates for the weights. However, notice that once these gradients are added up across a batch of data the final update for the weights can have variable signs, somewhat mitigating this issue. Therefore, this is an inconvenience but it has less severe consequences compared to the saturated activation problem above.

Why would having all $x>0$ (elementwise) lead to all-positive or all-negative gradients on $w$?

  • 1
    $\begingroup$ I also had the exact same question watching CS231n videos. $\endgroup$ – subwaymatch Nov 25 '17 at 17:09

$$f=\sum w_ix_i+b$$ $$\frac{df}{dw_i}=x_i$$ $$\frac{dL}{dw_i}=\frac{dL}{df}\frac{df}{dw_i}=\frac{dL}{df}x_i$$

because $x_i>0$, the gradient $\dfrac{dL}{dw_i}$ always has the same sign as $\dfrac{dL}{df}$ (all positive or all negative).

  • $\begingroup$ Please correct me if I am wrong but shouldn't the value of dL/df be transpose of x ie x.T since we would be using idea of Jacobin in here. $\endgroup$ – chinmay Feb 11 '18 at 14:54
  • $\begingroup$ @chinmay sorry for the late reply, I think $f$ here is the outcome of $w^Tx+b$ so the value of dL/df does not depend on x, and usually $L$ is a scalar, $w$ and $x$ are 1d vectors, so dL/df should also be a scalar, right? $\endgroup$ – dontloo Feb 23 '18 at 5:47
  • $\begingroup$ Yes, it is a big typo from my end. I meant df/dw .... but I think it depends more on the vector x and if it is a row vector or a column vector $\endgroup$ – chinmay Mar 28 '18 at 15:46

protected by Community Jan 22 '18 at 12:33

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.