RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, each update to the weight should be still similar in magnitude, right? Then why do we need other methods like batch normalization to deal with vanishing gradients?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.