0
$\begingroup$

I'm having some problems to understand LSTM theory and how exactly works a time series data in a Deep learning. To see if my thoughts are correct, I will make some statements (explanation) about how I think it's works and if I'm made mistakes, please corrects me! After that I will make some questions:

Let's say that I want to create a time series regression based on this data:

time series

To evaluate my model, after the yellow line will be my validation set, which means that my training set will be the sequence before that line. To train my model, I will follow the same steps as in DNN (Dense Neural Network) model:

Feed the data in B batchs >> Evaluate the loss >> apply Backpropagation

But, to do this in a sequential data I need to create my samples, following the image bellow: sample

Where X is in red and Y is in green. Basically I will feed [X1, X2, X3] and evaluate my loss in Y. Considering a time step equals to one, my training set will become something like this:

(([X1, X2, X3], Y)
 ([X2, X3, X4], Y)
 ...
)

Where X4 in the first row was Y, X5 will be Y in the second row and so on... Now I get stuck:

  • 1 - let's say that I just have one LSTM layer with just one cell and considering the ([X1, X2, X3], Y) I will insert X1, process (gates magic), insert X2, process, insert X3, process and then compare Yprediction to Y, take my loss and apply backpropagation? or I just insert [X1, X2, X3] at the same time as a vector(because as I discovered each LSTM gate is a small DNN) and then evaluate my loss?
  • 2 - As I said earlier, each gate is indeed a DNN? Their architectures are defined or I just need to specify (how many layers and so on)?
  • 3 - Following DNN model, the so called LSTM cells are fully connected? lets say that I have two LSTM layers, they will follow this same structure where every cell from first layer is connected to each cell from the second one?

PS: Yes, I read colah post, but I did not figure out those points.

$\endgroup$

1 Answer 1

0
$\begingroup$

Answering out of order:

2 - Each gate is NOT a DNN, it is an LSTM Cell, which is a single neuron, albeit one whose inner workings are more complex than the basic neurons used in DNN. There are two key concepts in LSTM:

  • Recurrence, where the output is fed back to the input. This is a property of RNN's in general, of which LSTM is a special case. DNN cannot have the recurrence feature, which is why they are sometimes referred to as feed-forward networks - the signal goes in only one direction. In RNNs on the other hand the signal can flow backwards. Recurrence allows the storage of "short term memories", by the previous values of the sequence in with the current values of the sequence.
  • The memory cell: This is unique to LSTM, and allows the LSTM to store not just short term interactions, but also long term ones.

The reason why it is easy to confuse a single LSTM cell with an entire DNN (I had this misconception myself when I first started studying them) is that LSTM are typically represented as rolled out over time. A single LSTM cell, because of its recurrence relation, can take in a sequence step by step and hence is functionally equivalent to multiple DNN inputs taking in each value of the sequence simultaneously.

From Chris Olah's post, the single LSTM cell is rolled out over time, so that it looks like multiple DNN cells:

enter image description here

Why do we not just stick to DNN then? Well, with the LSTM cell, the sequence can be arbitrarily long, whereas for the DNN the length of an input sequence is limited by the number of neurons you defined in your DNN's input layer.

1 - Here, it gets little confusing: as mentioned above, LSTM are supposed to handle arbitrarily long sequences, compared to DNN, which have the sequence length predefined by the network architecture. And this is indeed the case for LSTM applied to language problems. In theory this should work for LSTM applied to time series as well. But in practice, it doesn't give good results, so instead we feed the LSTM sequences of fixed length, or sequence windows (see here for how input and output windows are used for forecasting with LSTM). So to answer your question: Do you insert [X1, X2, X3] in one pass, or do you insert the sequence one by X1, then X2, then X3, etc...? You insert them one by one in sequence, but the sequence is a fixed window length sequence, so from the outside they look like they are [X1, X2, X3] vectors, even though they are not. So yeah, your data looking like this:

(([X1, X2, X3], Y)
 ([X2, X3, X4], Y)
 ...
)

Is fine, just think of them as windowed sequences.

(If you wondering how the hell are you going to code this windowing protocol, don't worry: Keras handles it for you).

3 - "Following DNN model, the so called LSTM cells are fully connected? lets say that I have two LSTM layers, they will follow this same structure where every cell from first layer is connected to each cell from the second one?" - No, that is not the case. As explained above, LSTM cells are different from normal neurons in that they are inherently sequential. If you want to connect two LSTM layers, the issue is not whether they are fully connected or not. The detail you have to be careful about is that you are feeding sequences to the second layer, not just individual values (In Keras, this is done by setting the "return sequences" argument to true when specifying an LSTM layer which connects to another LSTM layer).

Chris Olah's post on LSTM is excellent, but it focuses mostly on the internal mechanics of a single LSTM cell. For a more comprehensive functional view of LSTM, I recommend Andrej Karpathy's blog on the topic: The Unreasonable Effectiveness of Recurrent Neural Networks, even though it focuses mostly on language examples, not time series.

$\endgroup$
1
  • $\begingroup$ First, thanks to answer! @Skander H. - Reinstate Monica Could you implement those infos? (A)In Q2, where can I find more about how those gates works? (B) Let's say that I have one LSTM layer with 2 cells, each cell will receive the [X1, X2, X3]. What I'm saying is: each cell receives the same input? (C) Related to backpropagation, will work as I said? (feed my input sequence, predict Y and then apply backpropagation) $\endgroup$ Jan 10, 2020 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.