1
$\begingroup$

I understand how the forget gate works. My understanding of the forget gate:

A sigmoid function is used to make each of the gate tensor's values $\Gamma_f^{<t>}$ range from 0 to 1. The forget gate $\Gamma_f^{<t>}$ has the same dimensions as the previous cell state $c^{⟨t−1⟩}$. This means that the two can be multiplied together, element-wise. Multiplying the tensors $\Gamma_f^{<t>}*c^{⟨t−1⟩}$ is like applying a mask over the previous cell state. If a single value in $\Gamma_f^{<t>}$ is 0 or close to 0, then the product is close to 0.

What I don't understand is the purpose of the update gate and how does it achieve its purpose?

$\endgroup$

2 Answers 2

0
$\begingroup$

LSTM gates are, as generally referred as, forget gate, input gate and output gate. The update gate is a combination of forget and input gates and it is introduced in GRUs. Basically, input gate decides how much of the input contributes into the current state, and is independent of forgetting mechanism. So, if we forget a cell value and doesn't choose to place any input, it's like the cell remains stale. In GRU, the input gate multiplier is complement of the forget gate, i.e. $i_t=1-f_t$. Therefore, if the cell chooses to forget a state, it fills in the corresponding state with new input.

$\endgroup$
0
$\begingroup$

While it's common to read that LSTMs have three gates, as @gunes noted, I believe @A-ar is asking about the "cell update" or "candidate gate" in LSTMs, which is often included when discussing the input gate. The input, forget, and output gates use the sigmoid function to make decisions (0 or 1) - thus referred to as "gates". The "candidate gate", also known as the activation function, transforms the data.
Similar SO question
Example of documentation stating that LSTMs have four gates
Excellent article for understanding LSTMS

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.