How GRU solves vanishing gradient

Question

I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}{\partial h_{i-1}}$ in 3 cases: (1). $z= 1$. (2). $z=0, r=0$, and (3). $z=0, r=1$.

Here $z,r$ are update and reset gates respectively.

To clarify the notations and provide some details, the loss gradient at step $t$ with respect to the input-to-hidden weight in the update gate $z_t$, $U_z$ has form \begin{align} \frac{\partial L_t}{\partial U_z} &= \frac{\partial L_t}{\partial h_t} \sum_{k=1}^t \{ \left(\prod_{i=k+1}^t \frac{\partial h_i}{\partial h_{i-1}} \right) \frac{\partial^+ h_k}{\partial U_z} \} \end{align} To keep the notations simple I will drop the matrix transposes and Hadamard products in the partial derivative below: \begin{align} \frac{\partial h_i}{\partial h_{i-1}} &= \frac{\partial h_i}{\partial \tilde{h}_i} \frac{\partial \tilde{h}_i}{\partial h_{i-1}} + \frac{\partial h_i}{\partial z_i} \frac{\partial z_i}{\partial h_{i-1}} + \frac{\partial^{+} h_i}{\partial h_{i-1}} \\ &= (1-z_i) \{ W_r W_h (1- \tilde{h}_i^2) h_{i-1} r_i (1- r_i) + W_h(1- \tilde{h}_i^2) r_i\} \\ &+ W_z (h_{i-1}- \tilde{h}_i) z_i (1-z_i) + z_i \end{align} where $h_i$ is the hidden state at step $i$, $\tilde{h}_i$ is the current memory of the network at step $i$.

The 3 cases mentioned earlier is explained in more details here. On slide 4 the author mentioned that units with short-term dependencies will have active reset gates $r_t$; and units with long term dependencies have active update gates $z_t$.

I would like some help with (1). Understanding the status of the gates in relation to short/long-term dependencies.
(2). In the 2nd case , when $z=r=0$, $\frac{\partial h_i}{\partial h_{i-1}}$ is $0$. Why $\frac{\partial L_t}{\partial U_z}$ still avoids shrinking to zero

---

Rehmer & Kroll (2020).

(8): (9):

Answer 1

(1). Understanding the status of the gates in relation to short/long-term dependencies.

Active update ($z_t$) gate means that the information inside $h_{t-1}$ is propagated to $h_t$, because the update equation is $h_t=z_t h_{t-1} + (1-z_t)\tilde h_t$. So, the information travels through the time steps. This means long-term dependency.

Active reset ($r_t$) gate means the $\tilde h$ has information integrated into it from the previous time step due to the equation $\tilde h_t=\tanh(Wx_t+r_tUh_{t-1})$. At the same time, if $z_t$ is not very active, this information will be conveyed to the next steps, but will die out quickly. That is short-term dependency.

(2). In the 2nd case , when $z=r=0$, $\frac{\partial h_i}{\partial > h_{i-1}}$ is $0$. Why $\frac{\partial L_t}{\partial U_z}$ still avoids shrinking to zero

The derivative is not zero. Even in your formula, it looks like $W_h$, though I'm not sure the exact form. However, since $z_t$ and $r_t$ also depend on $h_{t-1}$, the derivative is not zero, at least not in a simple way.

Answer 2

I want to add to the accepted answer and the question itself.

I think there is a confusion as to how GRU solves the vanishing gradient issue (title of the question but, not the actual question itself) when z=r=0 which makes ∂hi/∂hi−1 = 0 and therefore, ∂Lt/∂Uz = 0.

From the backward pass equations in the given links, it is in fact true that ∂Lt/∂Uz = 0 when z=r=0 because ∂hi/∂hi−1 = 0.

One thing to keep in mind that this does not necessarily mean that the entire gradient vanishes because the entire gradient is not ∂Lt/∂Uz but, ∂L/∂Uz which is the sum of all ∂Lt/∂Uz at different time steps. You need to repeat the above equation for all ∂Lt/∂Uz at multiple time steps because each time step has its own loss (error). This is true if you have a label y (ground truth) for each time step and the error is some form of a loss function that involves this label y. If you only have a label at the very end of the sequence (a classification type problem), then I believe you only have a single error ∂Lt/∂Uz for that sequence (or time series) hence the gradient in fact vanishes if z=r=0 at any point in time but, your hope is that you can find a path (set of weights) where z and r are not equal to zero (at least not both r and z) for the entire sequence so information from previous time steps keep moving forward but, that might be very difficult if you have a very long sequence and the longer the sequence is the harder it becomes to find these set of weights that avoids the vanishing gradient issue.

So, technically from what I observed so far, you can run into the vanishing gradient issues in GRUs depending on the weights you find during training and once you run in to the vanishing gradient issue the training ultimately stops because the weights no longer change or you get stuck at a local minima and no matter how many iterations you run, it won't help unless of course, you are using some fancy optimizer that let's you escape the local minima (which I don't know about). Also, what if such set of weights do not exist for the type of problem you are solving (no one can guarantee that it does exist - at least not without running some serious analysis).

You should also not ignore the context of the problem: whether you are solving a classification type problem (many-to-one) or regression type problem (many-to-many) and watch your gradients closely.

I also want to share this wonderful and intuitive paper which explains the derivation of the GRU gradients via BPTT and when & why the gradients vanish or explode (mostly in the context of gating mechanisms):

Rehmer, A., & Kroll, A. (2020). On the vanishing and exploding gradient problem in gated recurrent units. IFAC-PapersOnLine, 53(2), 1243–1248. https://doi.org/10.1016/j.ifacol.2020.12.1342

I'm not sure if you can access the paper directly but, there should be a version of the paper available by the authors online which you can read. I did not want to provide a link to avoid future issues since the paper may not be open access.

I hope this answer resolves some of the confusion and helps someone else that come across this post as well.