1
$\begingroup$

I read several posts about BPTT for RNN, but I am actually a bit confused about one step in the derivation. Given

$$h_t=f(b+Wh_{t-1}+Ux_t)$$

when we compute $\frac{\partial h_t}{\partial W}$, does anyone know why is it simply

$$\frac{\partial h_t}{\partial W}=\frac{\partial h_{t}}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial W}$$

not

$$\frac{\partial h_t}{\partial W}=\frac{\partial h_{t}}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial W}+\frac{\partial h_t}{\partial f}h_{t-1}$$

?

What I mean is, since both $W$ and $h_{t-1}$ depends on $W$, why is the second term in the expression above missing?

Thank you!

$\endgroup$
  • $\begingroup$ How does W depend on W? Also, are you trying to take a total derivative? $\endgroup$ – nbro Nov 26 '19 at 4:02
  • $\begingroup$ @nbro Because W is W itself, $\frac{\partial W}{\partial W} = 1$. I am not trying to do a total derivative, but only partial derivative wrt W. $\endgroup$ – username123 Nov 26 '19 at 15:30
  • $\begingroup$ can you provide a reference for the first version of $\frac{\partial h_t}{\partial W}$ $\endgroup$ – shimao Nov 30 '19 at 21:55
  • $\begingroup$ @shimao Sure, one example is kharshit.github.io/blog/2019/02/22/backpropagation-through-time, where they denoted "h" as "s". $\endgroup$ – username123 Dec 2 '19 at 17:19
  • 1
    $\begingroup$ @username123 in that link, they correctly write $\frac{\partial h_t}{W} = \sum_{i<t} \frac{h_t}{h_i} \frac{\partial h_i}{W}$ $\endgroup$ – shimao Dec 6 '19 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.