"Concurrent" LSTM network

Question

Basically, my input can be thought of timeseries of timeseries (where each individual point of the timeseries is a timeseries as well - those subseries are all of the same length).

You can interpret this data as a 2D tensor of shape (series length, subseries length)

What I'd like to achieve is to have a LSTM network, where first layer processes those subseries and outputs a vector for each of them; then second LSTM layer operates normally on this representation (series of embedded subseries) - as a model in tensorflow/keras which allows full backpropagation for training.

With convolutional nets this would be trivial using column-wise convolutions first until we are left with only one row, then a row-wise convolution.

Any idea if this has been done already using LSTMs?

Answer 1

Yes this type of thing is commonly done in natural language processing. It is referred to as hierarchical rather than concurrent. Example uses are to have the first LSTM process the characters in a word to create a representation for each word and the second LSTM operates on words in a sentence. Another example is to have the first LSTM process words in a sentence and the second can do sentences in a paragraph.