I need to implement an LSTM, so I'm reading a tutorial of how does it work. Quoted (the relevant parts; if you're aware of LSTM internals, you're probably want to just skip down to the question, and then look at any picture):

The first step in our LSTM is to decide what information we’re going to throw away from the cell state. This decision is made by a sigmoid layer called the “forget gate layer.” It looks at $h_{t−1}$ and $x_t$, and outputs a number between $0$ and $1$ for each number in the cell state $C_{t−1}$. A $1$ represents “completely keep this” while a $0$ represents “completely get rid of this.” forget gate

The next step is to decide what new information we’re going to store in the cell state. This has two parts. First, a sigmoid layer called the “input gate layer” decides which values we’ll update. Next, a tanh layer creates a vector of new candidate values, $\tilde{C}_t$, that could be added to the state. In the next step, we’ll combine these two to create an update to the state. enter image description here

It’s now time to update the old cell state, $C_{t−1}$, into the new cell state $C_t$. The previous steps already decided what to do, we just need to actually do it.

We multiply the old state by $f_t$, forgetting the things we decided to forget earlier. Then we add $i_t∗\tilde{C}_t$. This is the new candidate values, scaled by how much we decided to update each state value. input gate

The problem is that when $f_t$ (forget gate) is close enough to zero to drop $C_{t-1}$ (prev. state) upon multiplication, then $i_t$ (input gate) would also destroy the new state upon $i_t * \tilde{C}_t$, because accordingly to an image $f_t = i_t$.

So, how is the new state supposed to squeeze to the next step? Wouldn't the new state be forgotten at the same time with the prev. state? How is that supposed to work?


1 Answer 1


If you check the equations near diagrams, you can see that $ f_t \neq i_t$ because weights($ W_f$,$W_i$) and biases($ b_f$,$b_i$) are not tied together. You can't see them on the diagrams because, yellow rectangles implicitly include them.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.