What are the advantages, why would one use multiple LSTMs, stacked one side-by-side, in a deep-network? I am using a LSTM to represent a sequence of inputs as a single input. So once I have that single representation— why would I pass it through again?
I am asking this because I saw this in a natural-language generation program.
I think that you are referring to vertically stacked LSTM layers (assuming the horizontal axes is the time axis.
In that case the main reason for stacking LSTM is to allow for greater model complexity. In case of a simple feedforward net we stack layers to create a hierarchical feature representation of the input data to then use for some machine learning task. The same applies for stacked LSTM's.
At every time step an LSTM, besides the recurrent input. If the input is already the result from an LSTM layer (or a feedforward layer) then the current LSTM can create a more complex feature representation of the current input.
Now the difference between having a feedforward layer between the feature input and the LSTM layer and having another LSTM layer is that a feed forward layer (say a fully connected layer) does not receive feedback from its previous time step and thus can not account for certain patterns. Having an LSTM in stead (e.g. using a stacked LSTM representation) more complex input patterns can be described at every layer
From {1}:
While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that deep RNNs work better than shallower ones on some tasks. In particular, Sutskever et al (2014) report that a 4-layers deep architecture was crucial in achieving good machine-translation performance in an encoder-decoder framework. Irsoy and Cardie (2014) also report improved results from moving from a one-layer BI-RNN to an architecture with several layers. Many other works report result using layered RNN architectures, but do not explicitly compare to 1-layer RNNs.
FYI:
References:
From playing around with LSTM for sequence classification it had the same effect as increasing model capacity in CNNs (if you're familiar with them). So you definitely get gains especially if you are underfitting your data.
Of course double edged as you can also over fit and get worse performance. In my case I went from 1 LSTM to a stack of 2 and get pretty much instant improvement.
In my experience, stacking LSTM layers (beyond 3) seems to offer worse performance.
The purple has 2 layers, pink has 3 and green has 6. Everything else is held constant. It does, I'm sure, depend on task. My task is a sequence-to-sequence of fixed length input and output.
Adam
$\endgroup$