Basically, my input can be thought of timeseries of timeseries (where each individual point of the timeseries is a timeseries as well - those subseries are all of the same length).
You can interpret this data as a 2D tensor of shape
(series length, subseries length)
What I'd like to achieve is to have a LSTM network, where first layer processes those subseries and outputs a vector for each of them; then second LSTM layer operates normally on this representation (series of embedded subseries) - as a model in tensorflow/keras which allows full backpropagation for training.
With convolutional nets this would be trivial using column-wise convolutions first until we are left with only one row, then a row-wise convolution.
Any idea if this has been done already using LSTMs?