3
$\begingroup$

In GRU units, I don't understand the effective difference between the update and reset gate, $z_t$ and $r_t$ respectively. \begin{align} z_t &= \sigma_g(W_{z} x_t + U_{z} h_{t-1} + b_z) \\ r_t &= \sigma_g(W_{r} x_t + U_{r} h_{t-1} + b_r) \\ \hat{h}_t &= \phi_h(W_{h} x_t + U_{h} (r_t \odot h_{t-1}) + b_h) \\ h_t &= z_t \odot \hat{h}_t + (1-z_t) \odot h_{t-1} \end{align}

in the final rule, $z_t$ should select how much information preserve from $\hat{h}_t$ and the old state $h_{t-1}$, but the amount of $h_{t-1}$ to retain is already selected in $\hat{h}_t$ by $r_t$. So, why a new selection of $h_{t-1}$ is made again with $1-z_t$?

$\endgroup$

1 Answer 1

1
$\begingroup$

You are correct, the "forget" gate doesn't fully control how much the unit forgets about the past $h_{t-1}$. Calling it a "forget gate" was meant to facilitate an intuition about its role, but as you noticed, the unit is more complicated than that. The current hidden state $\hat h_t$ is a non-linear function of the current input $x_t$ and the past state $h_{t-1}$. We want it to be like this because the point of using recurrent neural networks is about modeling the non-linear changes over time. If $\hat h_t$ was only a non-linear function of the current inputs and we had a linear relationship between $\hat h_t$ and the past like $z_t \hat h_t + (1- z_t) h_{t-1}$, this would be just a kind of exponential smoothing with changing weights. But we want to model more complicated, non-linear changes over time. Also, keep in mind that the GRU unit is able to learn the simpler version of the model by zeroing the $r_t$ weights, so the simpler version is possible under GRU.

Another reason we have more complicated units like GRU or LSTM is the problems of vanishing and exploding gradients. While simpler RNNs should work, we noticed that there are computational problems when estimating their parameters, and GRU and LSTM were designed to overcome them.

$\endgroup$
6
  • $\begingroup$ ok, But if we want to model more complicated, non-linear changes over time, is it not enough just to model $\hat{h}_t$ as function of $x$ and $h_{t-1}$ (as already done) and make simply $h_t = z_t\hat{h}_t$? $\endgroup$
    – volperossa
    Aug 25, 2022 at 10:17
  • $\begingroup$ @volperossa "enough" in what sense? For some problems, RNN would be "enough", for some GRU, for some LSTM, for some using classical time-series models like ARIMA, or exponential smoothing, for some models "enough" would be to always predict the constant like historical average. We have many models because not every one of them works for all cases. Sure, nobody prohibits you from trying simpler ones before going to something more complicated. $\endgroup$
    – Tim
    Aug 25, 2022 at 10:21
  • 1
    $\begingroup$ Also keep in mind that the unit was designed to prevent problems like vanishing and exploding gradients, not only to model the data. $\endgroup$
    – Tim
    Aug 25, 2022 at 10:23
  • $\begingroup$ I try to better explain myself (in two messages for answer length limits): in the GRU general form I see a redundancy of $h_{t-1}$, appearing twice: the first time in the $\hat{h}_t$ expression, the second one in ${h}_t$ (that is function of both $\hat{h}_t$ and, again $h_{t-1}$). I'm searching the ratio behind this. I understand your point that explain that it can help to find non-linear changes. $\endgroup$
    – volperossa
    Aug 25, 2022 at 10:40
  • 1
    $\begingroup$ @volperossa every neural network has a lot of redundancies. Classification of the neural network without redundancies is logistic regression, every more complicated NN has redundant nodes. They are complicated because we want to model complicated, non-linear functions & for computational reasons like vanishing/exploding gradients. $\endgroup$
    – Tim
    Aug 25, 2022 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.