I am doing an online course that states that the reason we use LSTMs and similar variations of vanilla RNNs is because of the vanishing/exploding gradients problems with vanilla RNNs.
However, an earlier part of the course said that gradient clipping can be used to combat issues with exploding and vanishing gradients by scaling the norm of the gradient to a certain value. If this is the case, then why do we use LSTM models instead of just applying gradient clipping on a vanilla RNN?
Yes you can solve them, but it doesn't work.
Exploding gradient clipping implies that you are not learning anything, as clipping zeroes out the gradient, so long term relationship (which are the ones causing usually the explosion) will have 0 gradient
Vanishing gradient renormalization cause the fact that everything then it's weighted a certain value you pick, completely destroying the optimization:
Now, what happens if you always normalize $\nabla L$ to a specific value? you never converge, as the magnitude of gradient tells how far you are from the minima (given a strongly convex function, which can be reasonably assumed in a neighborhood of a minima of a non convex function)
In other words, think about a person going down a hill, you need to reduce the length of the step proportionally to the distance to the valley... if you do always 1 meter long step, then you will jump from one side of the hill to the other, never reaching the minima
Yes, you can induce it via the stepsize of the optimization, but the stepsize and the norm size will be veyr painful to nail, as they strongly influence the optimization process
LSTM instead attenuates these problems by construction (and then you'll meet the transformers which completely removes this problem, but are sweated only for some tasks, so you cannot really replace RNNs-like nets with them always)
