2
$\begingroup$

I was looking around for a good explanation as to why LSTMs are better able to handle vanishing and exploding gradients compared to vanilla RNNs. I know it is due to the cell memory $c_t$ acting as a constant error carousel (CEC): $$c_t=f_t\circ c_{t-1}+i_t\circ\widetilde{c}_t$$ So when I compute the gradient of the loss at time step $t$ w.r.t. a weight $W$ other than from the output gate, I get: $$\frac{\partial L_t}{\partial W}=\frac{\partial L_t}{\partial\hat{y}_t}\frac{\partial\hat{y}_t}{\partial h_t}\frac{\partial h_t}{\partial c_t}\left(\prod_{i=2}^t\frac{\partial c_i}{\partial c_{i-1}}\right)\frac{\partial c_{1}}{\partial W}.$$ If I made a mistake here please let me know. So my question is in regards to $\frac{\partial c_t}{\partial c_{t-1}}$, given the equation for $c_t$, it should be: $$\frac{\partial c_t}{\partial c_{t-1}}=f_t.$$ But I read a post that derived this expression: \begin{align*} \frac{\partial C_t}{\partial C_{t-1}} &= \frac{\partial C_t}{\partial f_{t}}\frac{\partial f_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}} + \frac{\partial C_t}{\partial i_{t}}\frac{\partial i_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}} \\ &+ \frac{\partial C_t}{\partial \widetilde{C}_{t}}\frac{\partial \widetilde{C}_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}} + \frac{\partial C_t}{\partial C_{t-1}} \end{align*} https://weberna.github.io/blog/2017/11/15/LSTM-Vanishing-Gradients.html

Which has me a bit confused. So my question is where did I or they go wrong?

Any help is much appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.