GRU unit: difference between Update and Forget gates In GRU units,
I don't understand the effective difference between the update and reset gate, $z_t$ and $r_t$ respectively. 
\begin{align}
z_t &= \sigma_g(W_{z} x_t + U_{z} h_{t-1} + b_z) \\
r_t &= \sigma_g(W_{r} x_t + U_{r} h_{t-1} + b_r) \\
\hat{h}_t &= \phi_h(W_{h} x_t + U_{h} (r_t \odot h_{t-1}) + b_h) \\
h_t &=   z_t \odot \hat{h}_t + (1-z_t) \odot h_{t-1}
\end{align}

in the final rule, $z_t$ should select how much information preserve from $\hat{h}_t$ and the old state $h_{t-1}$, but the amount of $h_{t-1}$ to retain is already selected in $\hat{h}_t$ by $r_t$. So, why a new selection of $h_{t-1}$ is made again with $1-z_t$?
 A: You are correct, the "forget" gate doesn't fully control how much the unit forgets about the past $h_{t-1}$. Calling it a "forget gate" was meant to facilitate an intuition about its role, but as you noticed, the unit is more complicated than that. The current hidden state $\hat h_t$ is a non-linear function of the current input $x_t$ and the past state $h_{t-1}$. We want it to be like this because the point of using recurrent neural networks is about modeling the non-linear changes over time. If $\hat h_t$ was only a non-linear function of the current inputs and we had a linear relationship between $\hat h_t$ and the past like $z_t \hat h_t + (1- z_t) h_{t-1}$, this would be just a kind of exponential smoothing with changing weights. But we want to model more complicated, non-linear changes over time. Also, keep in mind that the GRU unit is able to learn the simpler version of the model by zeroing the $r_t$ weights, so the simpler version is possible under GRU.
Another reason we have more complicated units like GRU or LSTM is the problems of vanishing and exploding gradients. While simpler RNNs should work, we noticed that there are computational problems when estimating their parameters, and GRU and LSTM were designed to overcome them.
