Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm currently doing some stuff with ML, and I created a LSTM model to recognize activities such as walking or running. I have read that LSTM has advantages over traditional RNNs due to addressing the vanishing gradient problem.
Could you please explain it to me with a simple example?
The vanishing gradient problem: [1]
As more layers using certain activation functions are added to neural networks, the gradients of the loss function approaches zero, making the network hard to train.
The reason that recurrent neural networks (RNN) suffer from the vanishing gradient problem is due to the nature of backpropagation through time. [2]
- An unrolled RNN tends to be a very deep network.
- In an unrolled RNN the gradient in an early layer is a product that (also) contains many instances of the same term.
Long short-term memory (LSTM) neural networks solve the vanishing gradient problem by allowing network cells to forget part of their previously stored memory.
For a concise mathematical summary, see How does LSTM prevent the vanishing gradient problem?.
[1] Wang, Chi-Feng. "The Vanishing Gradient Problem: The Problem, Its Causes, Its Significance, and Its Solutions." towards data science. Available from: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484.
[2] Milman, Oren. Answer to question: "Why do RNNs have a tendency to suffer from vanishing/exploding gradient?." Cross Validated (a Stack Exchange site). Available from: https://stats.stackexchange.com/a/370732/295223.
