What are the advantages of stacking multiple LSTMs?

What are the advantages, why would one use multiple LSTMs, stacked one side-by-side, in a deep-network? I am using a LSTM to represent a sequence of inputs as a single input. So once I have that single representation— why would I pass it through again?

I am asking this because I saw this in a natural-language generation program.

• Did you really mean LSTMs stacked side by side as in horizontally ( along time steps) or did you mean vertically stacked (multiple LSTM cells for each time step)? Mar 8 '16 at 18:04

I think that you are referring to vertically stacked LSTM layers (assuming the horizontal axes is the time axis.

In that case the main reason for stacking LSTM is to allow for greater model complexity. In case of a simple feedforward net we stack layers to create a hierarchical feature representation of the input data to then use for some machine learning task. The same applies for stacked LSTM's.

At every time step an LSTM, besides the recurrent input. If the input is already the result from an LSTM layer (or a feedforward layer) then the current LSTM can create a more complex feature representation of the current input.

Now the difference between having a feedforward layer between the feature input and the LSTM layer and having another LSTM layer is that a feed forward layer (say a fully connected layer) does not receive feedback from its previous time step and thus can not account for certain patterns. Having an LSTM in stead (e.g. using a stacked LSTM representation) more complex input patterns can be described at every layer

• LSTM cells within a layer are already fully, recurrently connected with each other (the outputs of a layer have connections to all inputs of the same layer). Therefore, individual cells can already combine features on top of the outputs of other cells, all within one layer. Could you elaborate on why multiple layers result in more complex patterns, please? May 11 '16 at 0:06

From {1}:

While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that deep RNNs work better than shallower ones on some tasks. In particular, Sutskever et al (2014) report that a 4-layers deep architecture was crucial in achieving good machine-translation performance in an encoder-decoder framework. Irsoy and Cardie (2014) also report improved results from moving from a one-layer BI-RNN to an architecture with several layers. Many other works report result using layered RNN architectures, but do not explicitly compare to 1-layer RNNs.

FYI:

References:

From playing around with LSTM for sequence classification it had the same effect as increasing model capacity in CNNs (if you're familiar with them). So you definitely get gains especially if you are underfitting your data.

Of course double edged as you can also over fit and get worse performance. In my case I went from 1 LSTM to a stack of 2 and get pretty much instant improvement.

In my experience, stacking LSTM layers (beyond 3) seems to offer worse performance.

The purple has 2 layers, pink has 3 and green has 6. Everything else is held constant. It does, I'm sure, depend on task. My task is a sequence-to-sequence of fixed length input and output.

• This shows that this particular green model is worse, but since changing the number of layers changes the difficulty of the optimization task, I wonder if we can change how the model is trained to overcome this. Does the performance gap between the 6-layer green model and the 2-layer purple model disappear if you change model configuration details like the optimizer settings, the initialization, the sequence length, or adding regularization, or other model hyper-parameters?
– Sycorax
Mar 10 '20 at 13:43
• I suppose I can - but I'm holding those constant. What would you suggest to start with? For optimizer, both use Adam Mar 10 '20 at 16:18
• Right, and my point is that holding them constant doesn't tell us whether changing the configuration (e.g. the learning rate) improves the green model. When we're training neural networks, we're not constrained to use the same configuration across all models, so a fair comparison would need to tune each of these models separately and report the best results for each. I have some suggestions on where to start here: stats.stackexchange.com/questions/352036/…
– Sycorax
Mar 10 '20 at 16:20