For very long input sequence, cell state can grow without bound and as a result output activation function (tanh) is saturated. In this case, can vanishing gradient occur in lstm without forget gate (i.e. forget gate equals to 1). what is the reason, if vanishing gradient can occure?
1 Answer
$\begingroup$
$\endgroup$
0
Let's first look at the cell state memory of a vanilla RNN:
$$a_t = g(W[a_{t-1}, x_t] + b_a)$$ where $g$ is some activation function (sigmoid, tanh), $a_{t-1}$ the previous state and $W$ and $b_a$ the weights and biases.
For a vanilla RNN, we incur a vanishing gradient because as we continually compute the activation function, our previous state $a_{t-n}$ will multiplicatively get smaller. For a small gradient, the sigmoid function will continually shrink the gradient close to $0$ until it virtually "vanishes". Exploding gradients occur in much the same way.
Let's look at LSTM's. For a LSTM cell, we compute the cell state by $$ c_t = \Gamma_u \times \tilde{c}^t + \Gamma_f \times c ^{t-1}$$
where the forget gate is a sigmoid output between $0$ and $1$ computed by $$\Gamma_f = \sigma(W_f[a^{t-1}, x^t] + b_f).$$
In a nutshell, we see that the forget gate allows us to capture the previous gradient in an additive manner; the current cell state is merely a (learned) linear combination of the candidate cell state and the the previous cell state. Even if we use backpropogation and take the derivative of our gradients, we can see that they no longer shrink multiplicatively.
Quora seems to have some good high-level answers as well, and I think this medium post shows the mathematical details behind the backpropogation.
