0
$\begingroup$

I am calculating the backpropagation across LSTM cell gates, and I notice that many online sources treat the gates as independent. For example, in this Aidan Gomez blog, he derives the backpropagation update for weight $W_a$ using $\delta a_1$:

$\delta a = \delta state \cdot i \cdot $ tanh'$(a)$

Presumably, he derives the $i_1$ term by assuming that $\frac{d}{da} (i \cdot f(a)) = i \cdot \frac{d}{da} f(a)$, which necessitates that a and i are independent.

However, $i_1$ is a function of $out_0$, which is itself a function of $a_0$. In this manner, would we not have to consider $\frac{d}{da}(i)$ when updating the weight matrix $W_a$?

In other words, the weights $W_a$ affect the state output at all times $t$, so any term which depends on state $H_{t-1}$ then must also have some dependency on $\Delta W_a$, right?

The same logic applies to all of the other weight matrices and their interdependencies.


I've tried to work out some of the implications of this below. Hopefully you can follow the long usage of the chain rule here.

Let $E$ be the error function we are using for gradient descent. Suppose we are calculating the update for weight matrix $W_A$ of the input activation gate. Note that the linked example uses explicit sigmoid and tanh functions, but I will use general notation $f_k$ for gate $k$ activation function.

$\frac{\delta E}{\delta W_A} = \frac{\delta E}{\delta H_t} \frac{\delta H_t}{\delta W_A} $, where $H_t = o_t * f_S(S_t)$.

$ \frac{\delta H_t}{\delta W_A} = \frac{\delta o_t}{\delta W_A} f_S(S_t) + o_t f_S ' (S_t) \frac{\delta S_t}{\delta W_A}$ using product rule on $H_t$.

Note that $o_t = f_o (W_o X_t + U_o H_{t-1})$. Let us call $W_o X_t + U_o H_{t-1} = Z_o$ for easy reference. Then,

$ \frac{\delta o_t}{\delta W_A} = f_o ' (Z_o) * U_o * \frac{\delta H_{t-1}}{\delta W_A} $,

The complication here is that the previous output $H_{t-1}$ also has some dependency on $W_A$. This is the crux of the problem!!! If we assume independence between gates, this would cancel out nicely to zero, but that doesn't seem mathematically sound.

Now expanding this term in the previous derivation, we get

$ \frac{\delta H_t}{\delta W_A} = \bigg( f_o ' (Z_o) * U_o * \frac{\delta H_{t-1}}{\delta W_A} \bigg) * f_S(S_t) + o_t f_S ' (S_t) \frac{\delta S_t}{\delta W_A}$

which requires the calculation of $\frac{\delta S_t}{\delta W_A}$, where $S_t = a_t * i_t + f_t * S_{t-1}$.

$\frac{\delta S_t}{\delta W_A} = \frac{\delta a_t}{\delta W_A} * i_t + a_t * \frac{\delta i_t}{\delta W_A} + \frac{\delta S_{t-1}}{\delta W_A} * f_t + S_{t-1} * \frac{\delta f_t}{\delta W_A}$ by using the product rule a bunch of times. Solving each term:

$\frac{\delta a_t}{\delta W_A} = X_t + U_A \frac{\delta H_{t-1}}{\delta W_A}$

$\frac{\delta i_t}{\delta W_A} = U_i * \frac{\delta H_{t-1}}{\delta W_A}$

$\frac{\delta f_t}{\delta W_A} = U_f * \frac{\delta H_{t-1}}{\delta W_A}$

From here, $\frac{\delta S_{t-1}}{\delta W_A}$ and $\frac{\delta H_{t-1}}{\delta W_A}$ are now circular, since we're currently trying to figure out those same terms for time $t$.

I'm banging my head against a wall here. If all the cross-gate terms truly cancel out, then the math is easy. I'm just not seeing justification for why they cancel out. Any help is appreciated!


The only explanation I can think of is that $H_{t-1}$ is considered a "fixed" input, such that we don't consider backpropagation through that channel. This seems like a cop-out though, as $H_{t-1}$ is certainly an adjustable quantity in terms of the various weight matrices. Anybody have further insight on this?

$\endgroup$
  • $\begingroup$ Quite similar to this question. $\endgroup$ – Oren Milman Oct 8 '18 at 10:45
  • $\begingroup$ @OrenMilman Thank you for the link! That is precisely my question, although it seems that neither that page - nor the linked Reddit and Quora threads - seem to answer the question very well. The Reddit page has an answer that seemingly says "we ignore this for convenience and call it 'truncated brackpropagation' " - which is obviously unsatisfying :p . Especially given that truncated bptt is already a thing that refers to only going back K time steps, nothing to do with actual propagation of the gradient through gate terms. $\endgroup$ – Joey F. Oct 12 '18 at 16:55
  • $\begingroup$ To me it also feels unsatisfying, though I am not an expert (and maybe we both just missed something here). $\endgroup$ – Oren Milman Oct 12 '18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.