I'm having some problems to understand LSTM theory and how exactly works a time series data in a Deep learning. To see if my thoughts are correct, I will make some statements (explanation) about how I think it's works and if I'm made mistakes, please corrects me! After that I will make some questions:
Let's say that I want to create a time series regression based on this data:
To evaluate my model, after the yellow line will be my validation set, which means that my training set will be the sequence before that line. To train my model, I will follow the same steps as in DNN (Dense Neural Network) model:
Feed the data in B batchs >> Evaluate the loss >> apply Backpropagation
But, to do this in a sequential data I need to create my samples, following the image bellow:
Where X is in red and Y is in green. Basically I will feed [X1, X2, X3]
and evaluate my loss in Y
. Considering a time step equals to one, my training set will become something like this:
(([X1, X2, X3], Y)
([X2, X3, X4], Y)
...
)
Where X4
in the first row was Y
, X5
will be Y
in the second row and so on...
Now I get stuck:
- 1 - let's say that I just have one LSTM layer with just one cell and considering the
([X1, X2, X3], Y)
I will insertX1
, process (gates magic), insertX2
, process, insertX3
, process and then compareYprediction
toY
, take my loss and apply backpropagation? or I just insert[X1, X2, X3]
at the same time as a vector(because as I discovered each LSTM gate is a small DNN) and then evaluate my loss? - 2 - As I said earlier, each gate is indeed a DNN? Their architectures are defined or I just need to specify (how many layers and so on)?
- 3 - Following DNN model, the so called LSTM cells are fully connected? lets say that I have two LSTM layers, they will follow this same structure where every cell from first layer is connected to each cell from the second one?
PS: Yes, I read colah post, but I did not figure out those points.