1
$\begingroup$

I'm building a custom LSTM net based on this article. I got questions on how to implement the backpropagation, based on these formulas of the derivatives in an LSTM layer: enter image description here

Question 1: The weights (w.., v.., b..) in the formulas for dHt and dXt have a T above them. Does that mean we have to use the "original" weights that have been used at that time step during propagation (and not current weights which might have been updated since then)?

Question 2: When we start backpropagation through time, where do we get dCt+1 and dHt+1 from? As we go back in time, they are accumulated (as shown in the formulas), but at the first iteration, do we just set them to 0?

$\endgroup$

1 Answer 1

0
$\begingroup$

If you want to get a general understanding of BPTT with LSTM units check out this numerical example: https://medium.com/@aidangomez/let-s-do-this-f9b699de31d9

$\endgroup$
2
  • $\begingroup$ Could you explain why they use a matrix of two elements as inputs (x0[1,2], x1[0.5,3]), two W weights, and what the "with label: ..." means in relation to that? (sorry, English math is not my native) $\endgroup$ Jan 28, 2021 at 19:46
  • $\begingroup$ Input x0 is the input of the first time step and x1 is the input of the second time step. Both have to elements because the example uses and input layer containing 2 input input nodes. Therefore there are two weights for each gate, corresponding to the to inputs of every time step. The label is just the target output. $\endgroup$
    – ValK
    Jan 30, 2021 at 1:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.