RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, each update to the weight should be still similar in magnitude, right? Then why do we need other methods like batch normalization to deal with vanishing gradients?


1 Answer 1


Adam and RMSProp utilized by Adam are created to speed up the optimization by accelerating the gradient descent. RMSProp's function is mainly to adapt the learning rate to the features by dividing the previous squared gradients.

enter image description here

Source: RMSProp by Ng

You see that after the RMSProp the trajectory becomes much smooth toward the minimum point since the dimensions causing oscillations are divided by a much larger number.

And batch normalization

can turn the contours of your learning problem from something that might be very elongated to something that is more round, and easier for an algorithm like gradient descent to optimize. So this works, in terms of normalizing the input feature values to a neural network, alter the regression.

Source: normalizing Activations in a Network (C2W3L04) and you can refer to this answer also(we just do the similar thing to the hidden layers and make the mean and variance be "learnable").

I don't think RMSProp solves vanishing gradient problem, or at least it was not designed to solve that.

To avoid vanishing gradient, I thought we should not use sigmoid as activation functions in hidden layers, use Relu or others instead; and initialize the weights carefully, for instance the Xaiver or He or Glorot and etc; batch norm and Residual sturcture and etc.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.