I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}{\partial h_{i-1}}$ in 3 cases: (1). $z= 1$. (2). $z=0, r=0$, and (3). $z=0, r=1$.
Here $z,r$ are update and reset gates respectively.
To clarify the notations and provide some details, the loss gradient at step $t$ with respect to the input-to-hidden weight in the update gate $z_t$, $U_z$ has form \begin{align} \frac{\partial L_t}{\partial U_z} &= \frac{\partial L_t}{\partial h_t} \sum_{k=1}^t \{ \left(\prod_{i=k+1}^t \frac{\partial h_i}{\partial h_{i-1}} \right) \frac{\partial^+ h_k}{\partial U_z} \} \end{align} To keep the notations simple I will drop the matrix transposes and Hadamard products in the partial derivative below: \begin{align} \frac{\partial h_i}{\partial h_{i-1}} &= \frac{\partial h_i}{\partial \tilde{h}_i} \frac{\partial \tilde{h}_i}{\partial h_{i-1}} + \frac{\partial h_i}{\partial z_i} \frac{\partial z_i}{\partial h_{i-1}} + \frac{\partial^{+} h_i}{\partial h_{i-1}} \\ &= (1-z_i) \{ W_r W_h (1- \tilde{h}_i^2) h_{i-1} r_i (1- r_i) + W_h(1- \tilde{h}_i^2) r_i\} \\ &+ W_z (h_{i-1}- \tilde{h}_i) z_i (1-z_i) + z_i \end{align} where $h_i$ is the hidden state at step $i$, $\tilde{h}_i$ is the current memory of the network at step $i$.
The 3 cases mentioned earlier is explained in more details here. On slide 4 the author mentioned that units with short-term dependencies will have active reset gates $r_t$; and units with long term dependencies have active update gates $z_t$.
I would like some help with
(1). Understanding the status of the gates in relation to short/long-term dependencies.
(2). In the 2nd case , when $z=r=0$, $\frac{\partial h_i}{\partial h_{i-1}}$ is $0$. Why $\frac{\partial L_t}{\partial U_z}$ still avoids shrinking to zero
Rehmer & Kroll (2020).