1
$\begingroup$

I am trying to understand how to use RNN/LSTM; but I cannot understand some concepts related to topology. I found really useful information in this article: http://colah.github.io/posts/2015-08-Understanding-LSTMs/. But then, when comparing that topology 1 to the one Alex Graves says in his LSTM writing2, and the one described here 3, we can see the differences (in the first one the inputs never go to previous states, whereas they do in the other two topologies). What is going on?

Thank you

$\endgroup$
2
$\begingroup$

There exist several LSTM variants, this may explain the difference you have observed.

{1} gives a list:

4.3. LSTM Variants

The vanilla LSTM from Section 2 is referred as Vanilla (V). The derived eight variants of the V architecture are the following:

  1. No Input Gate (NIG)
  2. No Forget Gate (NFG)
  3. No Output Gate (NOG)
  4. No Input Activation Function (NIAF)
  5. No Output Activation Function (NOAF)
  6. No Peepholes (NP)
  7. Coupled Input and Forget Gate (CIFG)
  8. Full Gate Recurrence (FGR)

The first six variants are self-explanatory. The CIFG variant uses only one gate for gating both the input and the cell recurrent self-connection (an LSTM modification proposed in GRU (Cho et al., 2014)). This is equivalent to setting $f_t = 1 − i_t$ instead of learning the forget gate weights independently. The FGR variant adds recurrent connections between all the gates as in the original formulation of the LSTM (Hochreiter & Schmidhuber, 1997). It adds nine additional recurrent weight matrices, thus significantly increasing the number of parameters.


References:

  • {1} Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber LSTM: A Search Space Odyssey. https://arxiv.org/abs/1503.04069
$\endgroup$
  • $\begingroup$ I see. But I have something unclear: For time series, If I am training my model with 400 sequences of length 6 (data of 6 weeks, the label will be the 7th week value), and in order to predict a value in the future I just use 1 sequence of length 6: am I really using 400 states? Why did I calculate the value of those weights if then I am just feeding one sequence? $\endgroup$ – user3528810 Nov 30 '16 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.