Difference between feedback RNN and LSTM/GRU

Question

I am trying to understand different Recurrent Neural Network (RNN) architectures to be applied to time series data and I am getting a bit confused with the different names that are frequently used when describing RNNs. Is the structure of Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) essentially an RNN with a feedback loop?

Answer 1

All RNNs have feedback loops in the recurrent layer. This lets them maintain information in 'memory' over time. But, it can be difficult to train standard RNNs to solve problems that require learning long-term temporal dependencies. This is because the gradient of the loss function decays exponentially with time (called the vanishing gradient problem). LSTM networks are a type of RNN that uses special units in addition to standard units. LSTM units include a 'memory cell' that can maintain information in memory for long periods of time. A set of gates is used to control when information enters the memory, when it's output, and when it's forgotten. This architecture lets them learn longer-term dependencies. GRUs are similar to LSTMs, but use a simplified structure. They also use a set of gates to control the flow of information, but they don't use separate memory cells, and they use fewer gates.

This paper gives a good overview:

Chung et al. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.

Answer 2

Standard RNNs (Recurrent Neural Networks) suffer from vanishing and exploding gradient problems. LSTMs (Long Short Term Memory) deal with these problems by introducing new gates, such as input and forget gates, which allow for a better control over the gradient flow and enable better preservation of “long-range dependencies”. The long range dependency in RNN is resolved by increasing the number of repeating layer in LSTM.

RNN

LSTM

For Further Details :Understanding LSTM

Answer 3

LSTMs are often referred to as fancy RNNs. Vanilla RNNs do not have a cell state. They only have hidden states and those hidden states serve as the memory for RNNs.

Meanwhile, LSTM has both cell states and a hidden states. The cell state has the ability to remove or add information to the cell, regulated by "gates". And because of this "cell", in theory, LSTM should be able to handle the long-term dependency (in practice, it's difficult to do so.)

Answer 4

TL;DR

We can say that, when we move from RNN to LSTM (Long Short-Term Memory), we are introducing more & more controlling knobs, which control the flow and mixing of Inputs as per trained Weights. And thus, bringing in more flexibility in controlling the outputs. So, LSTM gives us the most Control-ability and thus, Better Results. But also comes with more Complexity and Operating Cost.

_Reference

---

[NOTE]:

GRU is better than LSTM as it is easy to modify and doesn't need memory units, therefore, faster to train than LSTM and give as per performance. Actually, the key difference comes out to be more than that: Long-short term (LSTM) perceptrons are made up using the momentum and gradient descent algorithms.

---

This image demonstrates the difference between them:

Answer 5

I think the difference between regular RNNs and the so-called "gated RNNs" is well explained in the existing answers to this question. However, I would like to add my two cents by pointing out the exact differences and similarities between LSTM and GRU.

The original definitions mostly come in the following form (I omitted bias terms, as well as time indices where not needed, in order to reduce some noise in the formulas): $$\begin{align} & \text{GRU} & & \text{LSTM} \\ \\ \boldsymbol{z} &= \mathrm{gate}(\boldsymbol{x}, \boldsymbol{h}) & \boldsymbol{i} &= \mathrm{gate}(\boldsymbol{x}, \boldsymbol{h}) \\ \boldsymbol{r} &= \mathrm{gate}(\boldsymbol{x}, \boldsymbol{h}) & \boldsymbol{o} &= \mathrm{gate}(\boldsymbol{x}, \boldsymbol{h}) \\ && \boldsymbol{f} &= \mathrm{gate}(\boldsymbol{x}, \boldsymbol{h}) \\ \hat{\boldsymbol{h}}[t] &= \phi(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{U} (\boldsymbol{r} \odot \boldsymbol{h}[t-1])) & \boldsymbol{c}[t] &= \boldsymbol{f} \odot \boldsymbol{c}[t-1] + \boldsymbol{i} \odot \phi_1(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{U} \boldsymbol{h}[t-1]) \\ \boldsymbol{h}[t] &= (\boldsymbol{1} - \boldsymbol{z}) \odot \boldsymbol{h}[t-1] + \boldsymbol{z} \odot \hat{\boldsymbol{h}}[t] & \boldsymbol{h}[t] &= \boldsymbol{o} \odot \phi_2(\boldsymbol{c}[t]). \end{align}$$

Note that the GRU has only 2 gates, whereas the LSTM has 3. Also, the LSTM has two activation functions, $\phi_1$ and $\phi_2$, whereas the GRU has only 1, $\phi$. This immediately gives the idea that GRU is slightly less complex than the LSTM.

Now, if we rewrite the recurrences of both models so that they fit on a single line (instead of spreading over two lines), we would end up with the following equations: \begin{align*} \boldsymbol{h}[t] &= (\boldsymbol{1} - \boldsymbol{z}) \odot \boldsymbol{h}[t-1] + \boldsymbol{z} \odot \phi(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{U} (\boldsymbol{r} \odot \boldsymbol{h}[t-1])) \tag{GRU} \\ \boldsymbol{c}[t] &= \boldsymbol{f} \odot \boldsymbol{c}[t-1] + \boldsymbol{i} \odot \phi_1(\boldsymbol{W} \boldsymbol{x} + \boldsymbol{U} (\boldsymbol{o}\odot \phi_2(\boldsymbol{c}[t-1]))) \tag{LSTM} \end{align*} By putting the recurrences next to each other like this, it should become clear that if all of the follwing equalities hold $$\begin{align} \boldsymbol{f} & = \boldsymbol{1} - \boldsymbol{z} & \boldsymbol{i} & = \boldsymbol{z} & \boldsymbol{o} & = \boldsymbol{r} & \phi_2(\boldsymbol{x}) & = \boldsymbol{x}, \end{align}$$ the GRU and LSTM models implement the same recurrence. The only difference that remains, is that, in the LSTM, the gates are computed from $\boldsymbol{c}[t-1]$, whereas the GRU directly uses the result of the recurrence, $\boldsymbol{h}[t-1]$.

In this sense, the GRU is a strictly simplified version of the LSTM.