# Gradient decay in neural networks

I read that in traditional feed-forward neural nets the gradients in the early layers decay very quickly and that this is 'a bad thing'. But I don't understand why. Can someone please explain what does that mean? or at least direct me to places to read about it and understand.

$$\frac{\partial}{\partial \theta} f(g(h(\theta)) = \frac{\partial}{\partial g} f(g(h(\theta)) \cdot \frac{\partial}{\partial h} g(h(\theta)) \cdot \frac{\partial}{\partial \theta} h(\theta).$$
In a traditional neural network, you have a cascade of linear mappings and point-wise nonlinearities. If your nonlinearity is the logistic sigmoid, the derivatives $\frac{\partial}{\partial g} f$ will be smaller than 1, so that when you multiply many of these you get a very small gradient. The derivatives of parameters in early layers will contain more such factors than later layers, and hence will tend to be smaller.