2
$\begingroup$

SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate ?

I dont understand why he claimed that. I couldnt find any reference about it in any paper.

However, it has been shown that when we slowly decrease the learning rate, SGD shows the same convergence behaviour as batch gradient descent, almost certainly converging to a local or the global minimum for non-convex and convex optimization respectively.

Source : http://sebastianruder.com/optimizing-gradient-descent/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.