What are the advantages of stacking multiple LSTMs? What are the advantages, why would one use multiple LSTMs, stacked one side-by-side, in a deep-network? I am using a LSTM to represent a sequence of inputs as a single input. So once I have that single representation— why would I pass it through again?
I am asking this because I saw this in a natural-language generation program.
 A: From playing around with LSTM for sequence classification it had the same effect as increasing model capacity in CNNs (if you're familiar with them). So you definitely get gains especially if you are underfitting your data.
Of course double edged as you can also over fit and get worse performance. 
In my case I went from 1 LSTM to a stack of 2 and get pretty much instant improvement.
A: I think that you are referring to vertically stacked LSTM layers (assuming the horizontal axes is the time axis. 
In that case the main reason for stacking LSTM is to allow for greater model complexity. In case of a simple feedforward net we stack layers to create a hierarchical feature representation of the input data to then use for some machine learning task. The same applies for stacked LSTM's. 
At every time step an LSTM, besides the recurrent input. If the input is already the result from an LSTM layer (or a feedforward layer) then the current LSTM can create a more complex feature representation of the current input. 
Now the difference between having a feedforward layer between the feature input and the LSTM layer and having another LSTM layer is that a feed forward layer (say a fully connected layer) does not receive feedback from its previous time step and thus can not account for certain patterns. Having an LSTM in stead (e.g. using a stacked LSTM representation) more complex input patterns can be described at every layer
A: From {1}:

While it is not theoretically clear what is the additional power gained by the deeper
  architecture, it was observed empirically that deep RNNs work better than shallower ones
  on some tasks. In particular, Sutskever et al (2014) report that a 4-layers deep architecture
  was crucial in achieving good machine-translation performance in an encoder-decoder
  framework. Irsoy and Cardie (2014) also report improved results from moving from a one-layer
  BI-RNN to an architecture with several layers. Many other works report result using
  layered RNN architectures, but do not explicitly compare to 1-layer RNNs.

FYI:


*

*The same question on the data science Stack Exchange: Advantages of stacking LSTMs?

*Is anyone stacking LSTM and GRU cells together and why?

References:


*

*{1} Goldberg, Yoav. "A Primer on Neural Network Models for Natural Language Processing." J. Artif. Intell. Res.(JAIR) 57 (2016): 345-420. https://scholar.google.com/scholar?cluster=3704132192758179278&hl=en&as_sdt=0,5 ;
http://u.cs.biu.ac.il/~yogo/nnlp.pdf
A: In my experience, stacking LSTM layers (beyond 3) seems to offer worse performance.

The purple has 2 layers, pink has 3 and green has 6. Everything else is held constant. It does, I'm sure, depend on task.  My task is a sequence-to-sequence of fixed length input and output.
