For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have the size of the LSTM layer. Suppose that:

  • The number of time steps is $n=3$ and the size of the LSTM is $d=4$.
  • The output of the memory cell $h_t$ has dimensions $(n, d)$.
  • I want the output of my network $\hat{y}$ to be of dimensions $(n, 1)$.
  • I want to use Euclidean norm as loss function and sigmoid as activation function for the dense layer.

I implemented a dense layer with weights $w_{dx1}$ in order to obtain just one label per time step and be able to compare the output to the target. I started to have problems in implementing the backpropagation part and some questions come to my mind:

  • Should I update the weights of LSTM and the weights of the MLP at the end of the structure separately?

  • It is correct to compute $E_0 = (\hat{y} - y_{target}) * \sigma^{'}(h_t*w)$ and $dw_0 = E_0 * \hat{y}$ in order to update the weight of the fully connected layer as $w_0 \mathrel{-}= \eta * dw_0$? Once I have done that I don't understand how to compute $\delta h_t$ in order to start backpropagating through the LSTM layer. If $\delta h_t = \frac{\partial{E}}{\partial{ h_t}}$, how can I compute the error function for that layer? It looks like to me that I don't have any target to compare with at that point.

I'm not able to relate the two steps of backpropagation, maybe also because it is hard to find some example of this kind of implementation, given that the main ML framework do the work for you.

  • $\begingroup$ Welcome to CV. I think this is a nice technical question. I've not worked with LSTM's, but have with MLP & relatives, and I like Eric Wan's 1996 diagrammatic gradient because it is universal and intuitive. Have you considered making a Wan-style adjoint network? It should make sure your gradient is correctly formulated given your topology. It should, in theory, allow you to unify your backpropagation steps into one. If you wanted to have my version of fun you could watch how the gradient changes and maybe think about advanced update approaches for weights, or for culling. $\endgroup$ – EngrStudent Jan 3 at 13:01
  • $\begingroup$ I have never heard about it before. From your question I suppose that there is no way to consider the 2 steps of backpropagation at one time. Anyway, I will be glad to have a look at your proposed way to overcome the problem. $\endgroup$ – Alexbrini Jan 3 at 13:43
  • $\begingroup$ citeseerx.ist.psu.edu/viewdoc/summary?doi= $\endgroup$ – EngrStudent Jan 3 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.