0
$\begingroup$

Can we say that if our learning is too small then vanishing gradient will occur.

And if our learning rate is too large then exploding gradient will occur?

very small learning rate == vanishing gradient,

very large learning rate == exploding gradient

Is this concept right or wrong?

Improve this question
$\endgroup$
9
  • $\begingroup$ Exploding and vanishing gradients are about the gradients, not the learning rate. Setting the learning rate too small/large is also a problem, but it's a different problem. $\endgroup$
    – Sycorax ♦
    Dec 16, 2021 at 18:59
  • $\begingroup$ Hi, Thank you for the answer but I am reading an article where a well known professor explained that "Small learning rates consume a lot of time to converge or will not be able to converge because of the vanishing gradient, i.e. the gradient goes to 0. Large learning rates puts the model at risk of overshooting the minima so it will not be able to converge: what is known as exploding gradient." $\endgroup$
    – ZaHid
    Dec 16, 2021 at 21:11
  • $\begingroup$ If the gradient goes to zero, then any finite learning rate $c$ won't update the weights: $c \cdot 0 = 0$. $\endgroup$
    – Sycorax ♦
    Dec 16, 2021 at 21:12
  • $\begingroup$ Welcome to Cross Validated! Please give a full citation for the article so we can review it ourselves. $\endgroup$
    – Dave
    Dec 16, 2021 at 21:12
  • $\begingroup$ Here is the link to the article "towardsdatascience.com/…". $\endgroup$
    – ZaHid
    Dec 16, 2021 at 21:12

0

Reset to default

Browse other questions tagged or ask your own question.