Is there any difference between forget gates and remember gates?

Question

In this youtube tutorial by nervada the following diagrams can be seen:

The remember gate in Gated Recurrent Units (GRUs) in the diagram to the right seems to have an analogous function to the forget gate in LSTM (left diagram): filter out how much of the previous state will be kept or discarded.

Is there a more mathematical or technical difference between them?

Answer 1

Yes, the output of both the LSTM's forget gate and the GRU's remember gate are vectors with dimension equal to the LSTM's cell memory and GRU's hidden state, respectively, and pointwise values between 0 and 1.

They act by multiplying pointwise by the previous LSTM cell memory or the previous GRU hidden state, respectively. (Note, unlike the LSTM, the GRU does not maintain a cell memory separate from its hidden state.) Forget/remember gate values closer to 1 ensure that the previous cell memory/hidden state is retained, while values close to 0 wipe out the previous cell memory/hidden state.

Concretely (see notation definitions below), the GRU's remember gate is defined as $r_t = \sigma(W_r[h_{t-1},x_t])$, and the "remember-gated" hidden state is $h_{t-1}* r_t$.

Similarly, the LSTM's forget gate is defined as $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$ and the "forget-gated" cell memory is $C_{t-1}* f_t$.

That is, mathematically they are pretty much the same thing, except for the bias term and the fact that the forget gate is pointwise-multiplied by the LSTM's previous cell memory, while the GRU's remember gate is pointwise-multiplied directly by its previous hidden state.

But after this initial step, what the LSTM does with its forget-gated cell memory vs. what the GRU does with its remember-gated hidden state is quite different.

Check out Chris Olah's blog post for a great explanation for the inner workings of an LSTM, featuring some particularly elegant and intuitive diagrams.

Notation

• $x_t \in \mathbf{R}^x$ is the current input,
• $h_{t-1} \in \mathbf{R}^h$ is the previous hidden state,
• $C_{t-1} \in \mathbf{R}^c$ is the LSTM's cell memory,
• $W_r$ is a weight matrix for the remember gate with dimension $h \times (x + h)$,
• $W_f$ is a weight matrix for the forget gate with dimension $c \times (x + h)$,
• $b_f \in \mathbf{R}^c$ is a bias vector for the forget gate,
• $\sigma(\cdot)$ is the sigmoid function,
• $[\cdot, \cdot]$ is vector concatenation, and
• $*$ is pointwise multiplication.

Answer 2

The remember gate from GRU is rather similar to the output gate of the lstm. The role of the forget gate in the LSTM is represented by $(1 - z)$, where $z$ is the update gate from GRU.

For the mathematical details, I refer to my answer on a somewhat similar question.