I was looking around for a good explanation as to why LSTMs are better able to handle vanishing and exploding gradients compared to vanilla RNNs. I know it is due to the cell memory $c_t$ acting as a constant error carousel (CEC): $$c_t=f_t\circ c_{t-1}+i_t\circ\widetilde{c}_t$$ So when I compute the gradient of the loss at time step $t$ w.r.t. a weight $W$ other than from the output gate, I get: $$\frac{\partial L_t}{\partial W}=\frac{\partial L_t}{\partial\hat{y}_t}\frac{\partial\hat{y}_t}{\partial h_t}\frac{\partial h_t}{\partial c_t}\left(\prod_{i=2}^t\frac{\partial c_i}{\partial c_{i-1}}\right)\frac{\partial c_{1}}{\partial W}.$$ If I made a mistake here please let me know. So my question is in regards to $\frac{\partial c_t}{\partial c_{t-1}}$, given the equation for $c_t$, it should be: $$\frac{\partial c_t}{\partial c_{t-1}}=f_t.$$ But I read a post that derived this expression: \begin{align*} \frac{\partial C_t}{\partial C_{t-1}} &= \frac{\partial C_t}{\partial f_{t}}\frac{\partial f_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}} + \frac{\partial C_t}{\partial i_{t}}\frac{\partial i_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}} \\ &+ \frac{\partial C_t}{\partial \widetilde{C}_{t}}\frac{\partial \widetilde{C}_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}} + \frac{\partial C_t}{\partial C_{t-1}} \end{align*} https://weberna.github.io/blog/2017/11/15/LSTM-Vanishing-Gradients.html

Which has me a bit confused. So my question is where did I or they go wrong?

Any help is much appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.