I need to implement an LSTM, so I'm reading a tutorial of how does it work. Quoted (the relevant parts; if you're aware of LSTM internals, you're probably want to just skip down to the question, and then look at any picture):
The first step in our LSTM is to decide what information we’re going to throw away from the cell state. This decision is made by a sigmoid layer called the “forget gate layer.” It looks at $h_{t−1}$ and $x_t$, and outputs a number between $0$ and $1$ for each number in the cell state $C_{t−1}$. A $1$ represents “completely keep this” while a $0$ represents “completely get rid of this.”
…
The next step is to decide what new information we’re going to store in the cell state. This has two parts. First, a sigmoid layer called the “input gate layer” decides which values we’ll update. Next, a tanh layer creates a vector of new candidate values, $\tilde{C}_t$, that could be added to the state. In the next step, we’ll combine these two to create an update to the state.
…
It’s now time to update the old cell state, $C_{t−1}$, into the new cell state $C_t$. The previous steps already decided what to do, we just need to actually do it.
We multiply the old state by $f_t$, forgetting the things we decided to forget earlier. Then we add $i_t∗\tilde{C}_t$. This is the new candidate values, scaled by how much we decided to update each state value.
The problem is that when $f_t$ (forget gate) is close enough to zero to drop $C_{t-1}$ (prev. state) upon multiplication, then $i_t$ (input gate) would also destroy the new state upon $i_t * \tilde{C}_t$, because accordingly to an image $f_t = i_t$.
So, how is the new state supposed to squeeze to the next step? Wouldn't the new state be forgotten at the same time with the prev. state? How is that supposed to work?
