4
$\begingroup$

I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}{\partial h_{i-1}}$ in 3 cases: (1). $z= 1$. (2). $z=0, r=0$, and (3). $z=0, r=1$.

Here $z,r$ are update and reset gates respectively.

To clarify the notations and provide some details, the loss gradient at step $t$ with respect to the input-to-hidden weight in the update gate $z_t$, $U_z$ has form \begin{align} \frac{\partial L_t}{\partial U_z} &= \frac{\partial L_t}{\partial h_t} \sum_{k=1}^t \{ \left(\prod_{i=k+1}^t \frac{\partial h_i}{\partial h_{i-1}} \right) \frac{\partial^+ h_k}{\partial U_z} \} \end{align} To keep the notations simple I will drop the matrix transposes and Hadamard products in the partial derivative below: \begin{align} \frac{\partial h_i}{\partial h_{i-1}} &= \frac{\partial h_i}{\partial \tilde{h}_i} \frac{\partial \tilde{h}_i}{\partial h_{i-1}} + \frac{\partial h_i}{\partial z_i} \frac{\partial z_i}{\partial h_{i-1}} + \frac{\partial^{+} h_i}{\partial h_{i-1}} \\ &= (1-z_i) \{ W_r W_h (1- \tilde{h}_i^2) h_{i-1} r_i (1- r_i) + W_h(1- \tilde{h}_i^2) r_i\} \\ &+ W_z (h_{i-1}- \tilde{h}_i) z_i (1-z_i) + z_i \end{align} where $h_i$ is the hidden state at step $i$, $\tilde{h}_i$ is the current memory of the network at step $i$.

The 3 cases mentioned earlier is explained in more details here. On slide 4 the author mentioned that units with short-term dependencies will have active reset gates $r_t$; and units with long term dependencies have active update gates $z_t$.

I would like some help with (1). Understanding the status of the gates in relation to short/long-term dependencies.
(2). In the 2nd case , when $z=r=0$, $\frac{\partial h_i}{\partial h_{i-1}}$ is $0$. Why $\frac{\partial L_t}{\partial U_z}$ still avoids shrinking to zero


Rehmer & Kroll (2020).

(8): enter image description here (9): enter image description here

$\endgroup$

2 Answers 2

3
$\begingroup$

(1). Understanding the status of the gates in relation to short/long-term dependencies.

Active update ($z_t$) gate means that the information inside $h_{t-1}$ is propagated to $h_t$, because the update equation is $h_t=z_t h_{t-1} + (1-z_t)\tilde h_t$. So, the information travels through the time steps. This means long-term dependency.

Active reset ($r_t$) gate means the $\tilde h$ has information integrated into it from the previous time step due to the equation $\tilde h_t=\tanh(Wx_t+r_tUh_{t-1})$. At the same time, if $z_t$ is not very active, this information will be conveyed to the next steps, but will die out quickly. That is short-term dependency.

(2). In the 2nd case , when $z=r=0$, $\frac{\partial h_i}{\partial > h_{i-1}}$ is $0$. Why $\frac{\partial L_t}{\partial U_z}$ still avoids shrinking to zero

The derivative is not zero. Even in your formula, it looks like $W_h$, though I'm not sure the exact form. However, since $z_t$ and $r_t$ also depend on $h_{t-1}$, the derivative is not zero, at least not in a simple way.

$\endgroup$
10
  • $\begingroup$ Thanks for the answer. <br> I have thought about the reset gate, when the update gate is not active. When it is active, $r_t= 1$, info in $h_{t-1}$ and new info in $x_t$ are both passed on to $\tilde{h}_t$. With $z_t = 0$ they are further passed on to $h_t$. How do they 'die out quickly'? <br> When the reset gate is not active, only new info $x_t$ is passed on to $\tilde{h}_t$, and $h_t$ received only the new info. I find this sounds more like 'capturing short-term dependency' $\endgroup$
    – siegfried
    Dec 15, 2021 at 11:41
  • $\begingroup$ Dependencies aside, does GRU alleviate vanishing gradients with the additive partial derivatives instead of the multiplicative ones in BPTT? I have read in LSTM that addition is more well-behaved and the learnable gates will adjust to avoid the loss gradient shrinking to 0 $\endgroup$
    – siegfried
    Dec 15, 2021 at 11:45
  • 1
    $\begingroup$ Depending on inputs is neither short nor long term dependency, because it's the input not the memory. When $z_t=1$, the effect of the past is much more prevalent on the current state, compared to $z_t=0$. If $z_t$ is active, most of the information is conveyed through the steps, but this is not the case for $z_t=0$. Yes, the past state is blended with the input, but over the time steps, it's been blended with the inputs so many times that it's not possible to talk about any long term dependency. $\endgroup$
    – gunes
    Dec 16, 2021 at 22:10
  • $\begingroup$ About the learnable gates, yes. If the gates were not learnable, you wouldn't have the second term in your differentiation, since $z_t$ was constant. And, the whole expression would be $0$ if $z_t=0, r_t=1$ (vanilla RNN), but it's not when $z_t$ depends on something. $\endgroup$
    – gunes
    Dec 16, 2021 at 22:12
  • $\begingroup$ now that i think of this, if we just look at the equations of $\tilde{h}_t, h_t$, inactive $z_t$ removes $h_{t-1}$ from $h_t$, inactive $r_t$ removes $h_{t-1}$ from $\tilde{h}_{t}$. Then isn't $h_t$ solely dependent on $x_t$? $\frac{\partial h_t}{\partial h_{t-1}} = 0$ in this case? $\endgroup$
    – siegfried
    Mar 14, 2022 at 1:10
2
$\begingroup$

I want to add to the accepted answer and the question itself.

I think there is a confusion as to how GRU solves the vanishing gradient issue (title of the question but, not the actual question itself) when z=r=0 which makes ∂hi/∂hi−1 = 0 and therefore, ∂Lt/∂Uz = 0.

From the backward pass equations in the given links, it is in fact true that ∂Lt/∂Uz = 0 when z=r=0 because ∂hi/∂hi−1 = 0.

One thing to keep in mind that this does not necessarily mean that the entire gradient vanishes because the entire gradient is not ∂Lt/∂Uz but, ∂L/∂Uz which is the sum of all ∂Lt/∂Uz at different time steps. You need to repeat the above equation for all ∂Lt/∂Uz at multiple time steps because each time step has its own loss (error). This is true if you have a label y (ground truth) for each time step and the error is some form of a loss function that involves this label y. If you only have a label at the very end of the sequence (a classification type problem), then I believe you only have a single error ∂Lt/∂Uz for that sequence (or time series) hence the gradient in fact vanishes if z=r=0 at any point in time but, your hope is that you can find a path (set of weights) where z and r are not equal to zero (at least not both r and z) for the entire sequence so information from previous time steps keep moving forward but, that might be very difficult if you have a very long sequence and the longer the sequence is the harder it becomes to find these set of weights that avoids the vanishing gradient issue.

So, technically from what I observed so far, you can run into the vanishing gradient issues in GRUs depending on the weights you find during training and once you run in to the vanishing gradient issue the training ultimately stops because the weights no longer change or you get stuck at a local minima and no matter how many iterations you run, it won't help unless of course, you are using some fancy optimizer that let's you escape the local minima (which I don't know about). Also, what if such set of weights do not exist for the type of problem you are solving (no one can guarantee that it does exist - at least not without running some serious analysis).

You should also not ignore the context of the problem: whether you are solving a classification type problem (many-to-one) or regression type problem (many-to-many) and watch your gradients closely.

I also want to share this wonderful and intuitive paper which explains the derivation of the GRU gradients via BPTT and when & why the gradients vanish or explode (mostly in the context of gating mechanisms):

Rehmer, A., & Kroll, A. (2020). On the vanishing and exploding gradient problem in gated recurrent units. IFAC-PapersOnLine, 53(2), 1243–1248. https://doi.org/10.1016/j.ifacol.2020.12.1342

I'm not sure if you can access the paper directly but, there should be a version of the paper available by the authors online which you can read. I did not want to provide a link to avoid future issues since the paper may not be open access.

I hope this answer resolves some of the confusion and helps someone else that come across this post as well.

$\endgroup$
18
  • $\begingroup$ very insightful answer! regarding the task of sequence classification, do you mean that only the final step loss will be computed, because at each step t there will only be the hidden state $h_t$ computed? $\endgroup$
    – siegfried
    Mar 29, 2022 at 23:37
  • $\begingroup$ Sorry, It seems I didn't pay attention to the sum near the product term. Even if ∂hi/∂hi−1 = 0 at some i (or time step), you need to repeat this multiple times through all the time steps. So, the gradient may still not vanish entirely because it does not necessarily mean that zi=ri=0 is true for every time step. Remember z and r values depend on the input, weights, sigmoid activation, hidden state values, and bias values. Please go to this source (dbs.ifi.lmu.de/Lehre/DLAI/WS18-19/script/05_rnns.pdf) and just look at slide 70. It's BPTT for simple RNN but, still useful to look at. $\endgroup$ Apr 1, 2022 at 0:06
  • 1
    $\begingroup$ To answer your 2nd question: Using GRU, your hope is that you can learn the long term dependency in a given task but, when you update your weights it's possible to encounter a "vanishing gradient" issue at some time step t. Check the very first link I sent, they explain how this vanishing gradient may happen in GRU. In my opinion, the real question should be: after encountering this vanishing gradient problem at certain time step t, can the gradient descent algorithm change the weights such that the real dynamics within the task can be learned? I can't answer this question yet myself. $\endgroup$ Apr 4, 2022 at 2:57
  • 1
    $\begingroup$ To answer your 3rd question: another question you should ask yourself is: Should GRU learn to forget? In certain tasks, it may be useful to learn to forget or ignore certain inputs. In other words, you may not want certain inputs at certain time steps to affect your final answer. Think of a signal of a moving creature and GRU trying to guess its age. Let's say it should really pay attention to max speed to accurately guess the age and one of the creatures decides to stop for a long time. GRU should learn to pay attention to the max speed part of the sequence or learn to ignore certain inputs $\endgroup$ Apr 4, 2022 at 3:17
  • 1
    $\begingroup$ Great, when I read their paper their arguments made sense to me however, if you find out that for some reason it doesn't make sense and you actually find out why it doesn't make sense, I would also really appreciate it if you can let me know about your findings. Also, not sure about your end goal but, if you are dealing with language tasks it may not be a bad idea to look at BERT or similar as they seem to be the current state of the art especially for language tasks. Additionally, remember that there is also exploding gradient issues (gradient being higher is not necessarily a good thing). $\endgroup$ Apr 4, 2022 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.