Why do earlier hidden layers learn slower?

Question

I'm reading chapter 5 of Nielsen's textbook about vanishing gradients. He states:

In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. And while we've seen this in just a single network, there are fundamental reasons why this happens in many neural networks. The phenomenon is known as the vanishing gradient problem.

He goes on to say:

One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function f(x) of a single variable. Wouldn't it be good news if the derivative f′(x) was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?

My confusion with the vanishing gradient problem stems from this particular statement:

In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers.

And my question relates to this particular question:

Wouldn't it be good news if the derivative f′(x) was small? Wouldn't that mean we were already near an extremum?

---

Specifically, don't small derivatives indicate that the parameters have essentially converged and, therefore, finished learning? Since the gradients of the earlier layers are smaller compared to later layers, shouldn't this mean that earlier layers learn faster?

Answer 1

There are at least two issues here:

1. We are optimising a function of many variables. A partial derivative along one variable doesn't tell us anything about the function's distance from the optimum. It only tells us how much a change of that variable is related to the change of the function.

2. A partial derivative close to zero doesn't necessarily mean we are close to the (global) optimum. We may be as well close only to a local optimum, or at a plateau, which can both be far, far away from the global optimum.

So, as the consequence, the partial derivative doesn't tell you much about learning. Only the function value (the error) itself contains that information.

Answer 2

Recall that the backpropagation algorithm involves heavy usage of the chain rule, which means that the derivatives of early layers with respect to the output is represented as a long series of multiplications.

The gradient signal at the start maybe be fairly strong. But if your derivatives tend to be small -- say, on the order of 0.1, just as an example -- the last layer will receive a gradient of order 0.1, the penultimate layer will receive a gradient of order 0.01, then 0.001, and so on. At the start of your network, so many small numbers have been multiplied in the computation of the gradient that your parameters won't move much at all.

So a small derivative in this case does not necessarily tell you that you have converged, as you can see. Think of the "speed of learning" as, very loosely, the amount by which your parameters step toward the optimum. Then with this formulation, even if the gradient is very small, if the optima is far away, then you've really only moved an infinitesimal part of the way -- which isn't fast at all.