Why does the neural network stops at a certain loss and does not converge with a big learning rate?

Question

When increasing the learning rate of an neural network I expect a divergence of the gradient because the gradient moves further in every step of the learning rate which should result in endless cost values. In this neural network simulation I configured the learning rate to $10$ (see image) and expect an divergence in loss because of that but the loss stops at a certain value. Why is this the case and the loss/cost function doesn't diverge?

Answer 1

It depends on the definition of loss. Try regression.