Answering out of order:
2 - Each gate is NOT a DNN, it is an LSTM Cell, which is a single neuron, albeit one whose inner workings are more complex than the basic neurons used in DNN. There are two key concepts in LSTM:
- Recurrence, where the output is fed back to the input. This is a property of RNN's in general, of which LSTM is a special case. DNN cannot have the recurrence feature, which is why they are sometimes referred to as feed-forward networks - the signal goes in only one direction. In RNNs on the other hand the signal can flow backwards. Recurrence allows the storage of "short term memories", by the previous values of the sequence in with the current values of the sequence.
- The memory cell: This is unique to LSTM, and allows the LSTM to store not just short term interactions, but also long term ones.
The reason why it is easy to confuse a single LSTM cell with an entire DNN (I had this misconception myself when I first started studying them) is that LSTM are typically represented as rolled out over time. A single LSTM cell, because of its recurrence relation, can take in a sequence step by step and hence is functionally equivalent to multiple DNN inputs taking in each value of the sequence simultaneously.
From Chris Olah's post, the single LSTM cell is rolled out over time, so that it looks like multiple DNN cells:
Why do we not just stick to DNN then? Well, with the LSTM cell, the sequence can be arbitrarily long, whereas for the DNN the length of an input sequence is limited by the number of neurons you defined in your DNN's input layer.
1 - Here, it gets little confusing: as mentioned above, LSTM are supposed to handle arbitrarily long sequences, compared to DNN, which have the sequence length predefined by the network architecture. And this is indeed the case for LSTM applied to language problems. In theory this should work for LSTM applied to time series as well. But in practice, it doesn't give good results, so instead we feed the LSTM sequences of fixed length, or sequence windows (see here for how input and output windows are used for forecasting with LSTM). So to answer your question: Do you insert [X1, X2, X3] in one pass, or do you insert the sequence one by X1, then X2, then X3, etc...? You insert them one by one in sequence, but the sequence is a fixed window length sequence, so from the outside they look like they are [X1, X2, X3] vectors, even though they are not. So yeah, your data looking like this:
(([X1, X2, X3], Y)
([X2, X3, X4], Y)
Is fine, just think of them as windowed sequences.
(If you wondering how the hell are you going to code this windowing protocol, don't worry: Keras handles it for you).
3 - "Following DNN model, the so called LSTM cells are fully connected? lets say that I have two LSTM layers, they will follow this same structure where every cell from first layer is connected to each cell from the second one?" - No, that is not the case. As explained above, LSTM cells are different from normal neurons in that they are inherently sequential. If you want to connect two LSTM layers, the issue is not whether they are fully connected or not. The detail you have to be careful about is that you are feeding sequences to the second layer, not just individual values (In Keras, this is done by setting the "return sequences" argument to true when specifying an LSTM layer which connects to another LSTM layer).
Chris Olah's post on LSTM is excellent, but it focuses mostly on the internal mechanics of a single LSTM cell. For a more comprehensive functional view of LSTM, I recommend Andrej Karpathy's blog on the topic: The Unreasonable Effectiveness of Recurrent Neural Networks, even though it focuses mostly on language examples, not time series.