I read that in traditional feed-forward neural nets the gradients in the early layers decay very quickly and that this is 'a bad thing'. But I don't understand why. Can someone please explain what does that mean? or at least direct me to places to read about it and understand.


1 Answer 1


Think about the derivative of a cascade of functions,

$$\frac{\partial}{\partial \theta} f(g(h(\theta)) = \frac{\partial}{\partial g} f(g(h(\theta)) \cdot \frac{\partial}{\partial h} g(h(\theta)) \cdot \frac{\partial}{\partial \theta} h(\theta).$$

In a traditional neural network, you have a cascade of linear mappings and point-wise nonlinearities. If your nonlinearity is the logistic sigmoid, the derivatives $\frac{\partial}{\partial g} f$ will be smaller than 1, so that when you multiply many of these you get a very small gradient. The derivatives of parameters in early layers will contain more such factors than later layers, and hence will tend to be smaller.

This is a bad thing if you are using gradient descent, because you will be making tiny updates to the first layer parameters or huge updates to the last layer parameters. You could try to choose a different step width for different types of parameters, or you could use second-order methods which automatically scale the gradients in the right way.

  • $\begingroup$ actually the way I understood it is the following: if the gradients at the lower layers decay very quickly then that means the algorithm with fall quickly in a local minima. Is my intuition correct? $\endgroup$
    – Jack Twain
    Apr 16, 2014 at 16:36
  • $\begingroup$ When you say "decay", do you mean as a function of time (number of iterations) or as a function of network depth? I interpreted it as the latter. Where did you read about it? Maybe a little bit more context will help. $\endgroup$
    – Lucas
    Apr 17, 2014 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.