Why can't we handle vanishing gradient problem in neural nets using large step sizes?

Question

In the training of very deep neural nets, the vanishing gradient is the problem that the gradients of weights associated with early layers are too small leading to small overall steps in gradient descent. This seems like a valid problem if the gradient is too small to be represented using available bits. But if this is not the case, then why can't we overcome this problem by having huge learning rates (step sizes) for parameters with small gradient?

Note: This question is different from questions that have already been asked on this on stackexchange as other questions ask about role of activation function in vanishing gradient problem.

Answer 1

First, what if the gradient is small because you are close to the true solution? By using a large stepwise, you are going to overshoot.

Second, if the gradient is small then it has a very little directional information due to cancellation, so it would be like going in a random direction with a large stepsize.

Answer 2

The problem is that the gradient is not small for all layers, but small for the deeper layers (further away from the output). For the layers near the output, the gradient may be big.

If you now have a huge learning rate, the weights in the near-to-output layers will shoot around erratically and the net will not converge.