# Different ways to deepen LSTMs

Quick introduction of the notation I'm working with:

With regular RNNs we have $$h^{\left\langle t\right\rangle }=g_{h}\left(W_{h}\left[h^{\left\langle t-1\right\rangle },x^{\left\langle t\right\rangle }\right]+b_{h}\right)$$ $$\hat{y}^{\left\langle t\right\rangle }=g_{y}\left(W_{y}h^{\left\langle t\right\rangle }+b_{y}\right)$$

Where

$$x^{\left\langle t\right\rangle }\in\mathbb{R}^{n}$$ is the input at time $$t$$, $$h^{\left\langle t\right\rangle }\in\mathbb{R}^{d}$$ is the hidden state at time $$t$$ .$$\hat{y}^{\left\langle t\right\rangle }\in\mathbb{R}^{m}$$ is the prediction for time $$t$$

$$g_{h}$$ and $$g_{y}$$ are activation functions (say pointwise $$\tanh$$ or sigmoids)

$$W_{h}\in\mathbb{R}^{d\times(d+n)}$$ and $$W_{y}\in\mathbb{R}^{m\times d}$$ are the weight matrices.

For LSTMs, we add a vector $$c^{\left\langle t\right\rangle }\in\mathbb{R}^{d}$$, the memory cell, modify $$W_{h}$$ to be a $$4d\times(d+n)$$ matrix and use the $$4d$$ dimensional vector resulting from $$W_{h}\left[h^{\left\langle t-1\right\rangle },x^{\left\langle t\right\rangle }\right]$$ together with $$c^{\left\langle t-1\right\rangle }$$ through various gates to compute $$c^{\left\langle t\right\rangle }$$, $$h^{\left\langle t\right\rangle }$$ and $$\hat{y}^{\left\langle t\right\rangle }$$.

Iv'e seen in multiple places the idea of deepening this model by stacking LSTM units one on top of the other. That is, using the prediction $$\hat{y}^{\left\langle t\right\rangle }$$ as the input to another LSTM unit, with its own hidden state and memory cell. But it accrues to me there are other ways to modify this model to make it deeper. Namely, we can add MLPs (fully connected feed forward networks of various depths) at any connection in the way. The obvious places would be from the input $$x^{\left\langle t\right\rangle }$$ to $$x'^{\left\langle t\right\rangle }$$ before it engages with the cell, and from the prediction $$\hat{y}^{\left\langle t\right\rangle }$$ to $$\hat{y'}^{\left\langle t\right\rangle }$$, but really anywhere else inside the cell too.

Based on the idea of hierarchical representation learning, I'd excpect that stacking LSTMs would lead to the network learning more sophisticated memory features, a MLP from the input as transforming it into higher level features before processing it in the cell etc.

Are these observations correct or are there reasons for some of the ideas above to not be valid? Are there any theoretical or empirical results comparing these kind of approaches?