Understanding RNN/LSTM

Question

I am trying to understand how to use RNN/LSTM; but I cannot understand some concepts related to topology. I found really useful information in this article: http://colah.github.io/posts/2015-08-Understanding-LSTMs/. But then, when comparing that topology 1 to the one Alex Graves says in his LSTM writing2, and the one described here 3, we can see the differences (in the first one the inputs never go to previous states, whereas they do in the other two topologies). What is going on?

Thank you

Answer 1

There exist several LSTM variants, this may explain the difference you have observed.

{1} gives a list:

4.3. LSTM Variants

The vanilla LSTM from Section 2 is referred as Vanilla (V). The derived eight variants of the V architecture are the following:

1. No Input Gate (NIG)
2. No Forget Gate (NFG)
3. No Output Gate (NOG)
4. No Input Activation Function (NIAF)
5. No Output Activation Function (NOAF)
6. No Peepholes (NP)
7. Coupled Input and Forget Gate (CIFG)
8. Full Gate Recurrence (FGR)

The first six variants are self-explanatory. The CIFG variant uses only one gate for gating both the input and the cell recurrent self-connection (an LSTM modification proposed in GRU (Cho et al., 2014)). This is equivalent to setting $f_t = 1 − i_t$ instead of learning the forget gate weights independently. The FGR variant adds recurrent connections between all the gates as in the original formulation of the LSTM (Hochreiter & Schmidhuber, 1997). It adds nine additional recurrent weight matrices, thus significantly increasing the number of parameters.

---

References:

• {1} Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber LSTM: A Search Space Odyssey. https://arxiv.org/abs/1503.04069