What is the problem with training Neural Networks with back propagation with activation functions that only output positive values? [duplicate]

Question

I was watching the new CNNs course by Stanford (CS231n) and they mentioned in lecture 5 that its a bad idea to have activation functions that only output positive values. They explain the intuition on the problem around minute 17:22. It seems to be about two problems that I am having a hard time connecting why they cause a problem:

1. Values in the Network not being centered (probably intermediate activations and the initial data)
2. Activation functions in the network potentially always outputting positive values.

I don't really understand how the two issues are related or exactly how or what problems they cause. In the lecture the instructor asks:

Consider what happens when the input to a neuron (x) is always positive: What can we say about the gradients on w?

which the answer seems to be:

Always all positive or all negative (this is also why you want to zero-mean data!)

Which doesn't quite make sense to me (or I don't understand the reasoning).

Let me tell you what I have thought.

To reason about this I considered the standard equations for backpropagation:

$$ \delta^{(l)}_j = \sum^{D^{(l+1)} }_{k' = 1} \frac{\partial V(w) }{\partial s^{(l+1)}_{k'} } \frac{\partial s^{(l+1)}_{k'} }{\partial a^{(l)}_j } \frac{\partial a^{(l)} }{\partial s^{(l)}_j } = \sum^{ D^{ (l+1) } }_{ k' = 1 } \delta^{ (l+1) }_{k'} W^{ (l+1) } _{j,k'} \theta'( s^{(l)}_j ) $$

where $V$ is the loss function, $s$ is the pre-synaptic activation, $\theta$ is the activation function and $a$ is the activation value. Subscript denote indexing to a vector or matrix and subscripts indicate the layer in question.

Usually $\theta'$ involves the activation like the sigmoid derivative would be $\theta(s) = \theta(s) (1 - \theta(s)) = a (1 - a) $ which means that in the context of the lecture $\theta'( s^{(l)}_j ) > 0 $ is positive. Means that $\delta$'s are positive and if the derivative is computed as:

$$ \frac{ \partial V(w) }{\partial W^{(l)}_{i,j}} = a^{(l-1)}_i \delta^{(l)}_j $$

all the coordinates of the gradient of w (i.e. all the partial derivatives of $w$) are positive. However in the lecture they say that the gradients on $w$ are all positive or all negative. Why is my reasoning wrong and what is the correct reasoning?

I have two issues with this:

1. that its not clear to me what that even means. w is a vector (even a matrix for each unit anyway...), so there is only a single gradient (they say "gradients" which is wrong and confusing). I assume they mean all the partial derivative of w are positive or negative. I sort of agree with the positive but at least for the sigmoid, I don't see why they would all be negative. In fact if any is allowed to be negative, why can't some coordinates of the gradient of w be negative and some positive, why do all have to be positive or negative?
2. Why does the update rule under these conditions give rise to the "zig-zag" behaviour that they talk about in the slides? Are they referring that the update to the parameters can only be positive and negative? and how is the zig-zaging an issue? (for zig-zag discussion see slides from the video)
3. How is centering the data (or the activations? Not sure what needs to be centered from this discussion) related to having all positive or all negative gradients? Not sure what centering has anything to do with this (or the activation).
4. What problems are we even trying to address? Like even if the I accept that we have all positives or all negatives or that we only do zig-zaging, what are these problematic? How do they hinder either the optimizer from exploring the surface loss or how does it hinder generalization?

Essentially I don't understand what arose to the symptoms (i.e. all positives/negatives gradient & zig zaging) and given those symptoms, what the issue is.

My only vague guess is, if we have say $K$ parameters for a activation unit and we can only move in the direction when all coordinates are positive, it would seem to hinder the search space tremendously (essentially it boils down that we wouldn't be able to span a space in $R^K$ with a single vector, meaning the changes to our initial vector w are extremely constrained).

---

At the end of that discussion they also talk about something about a Fisher Matrix and something else about natural gradients to make their argument more rigorous. I don't yet comprehend their "intuitive" argument so it would be good to understand that first. However, any answers providing more mathematical rigor are more than welcome (but will take me some more time to process). Also, if people know what papers or what they are referring to when they talk about those more rigorous concepts that explain this specific idea of gradients not taking helpful values for training when the activations are badly chosen, I'd love to read or listen to those ideas.

Answer 1

The gradient of sigmoid function is $$ (1-\sigma) \sigma $$ So if the input is $[0,1]$ then the output of the gradients on $w$ could only be all positive or all negative. You can check more on the notebook.

Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving data that is not zero-centered. This has implications on the dynamics during gradient descent, because if the data coming into a neuron is always positive (e.g. $x>0$ elementwise in $f=w^Tx+b$)), then the gradient on the weights ww will during backpropagation become either all be positive, or all negative (depending on the gradient of the whole expression $f$).

Answer 2

$$f = w^T.x + b$$ Thus, the gradient is: $$\frac{\partial f}{\partial w} = x $$.
If $x > 0$, then this expression is positive.

$$\sigma = \frac{1}{1+ \exp(-f) }$$ $$\frac{\partial \sigma}{\partial f} = \sigma (1 - \sigma)$$
Now since, $\sigma$ is sigmoid function and it always outputs in the range (0,1) this expression is always positive.

The loss gradient coming upstream is say $\frac{\partial L}{\partial \sigma}$.

Thus by using chain rule, $$ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial \sigma}\sigma (1 - \sigma)x $$ In the above three terms on RHS the rightmost two terms are positive given $x > 0$ and so the sign of gradient is only determined by the gradient of loss function coming upstream in back propagation.

The problem now here is if $x$ which is positive is parameterized by say $n$ $w$'s, then all of these $w$'s will have same sign for gradient. A case where different $w$'s have different signs will not occur. That's what will lead to zig zag dynamics.