# Non zero centered activation functions

I read the following section from cs231n course notes:

• Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving data that is not zero-centered. This has implications on the dynamics during gradient descent, because if the data coming into a neuron is always positive (e.g. $x > 0$ elementwise in $f = w^Tx + b$)), then the gradient on the weights $w$ will during backpropagation become either all be positive, or all negative (depending on the gradient of the whole expression $f$). This could introduce undesirable zig-zagging dynamics in the gradient updates for the weights. However, notice that once these gradients are added up across a batch of data the final update for the weights can have variable signs, somewhat mitigating this issue. Therefore, this is an inconvenience but it has less severe consequences compared to the saturated activation problem above.

I have understood why the gradients with respect to weights $w$ become all positive or negative during backpropagation since $$\dfrac{\partial f}{\partial w_j}=x_j \text{ , and } \dfrac{\partial L}{\partial w_j}=\dfrac{\partial L}{\partial f}\dfrac{\partial f}{\partial w_j}=\dfrac{\partial L}{\partial f}x_j$$ Thus the gradient of $L$ with respect to weights are all positive or negative depending on the sign of $\frac{\partial L}{\partial f}$.

But I do not understand why it has implications on the dynamics during gradient descent. More precisely, why do we get 'zig-zag' gradient updates if the derivatives with respect to weights are all positive or all negative? Can you provide some intuitions and mathematical arguments to justify this?