LSTM and GRU are models that were proposed in order to solve the vanishing gradient issue. However, I have noticed that with long sequences these models also suffer from it, which makes sense. I am using sequences from 60 to 180 timesteps for video classification.

Is there any method to avoid vanishing gradient for long sequences?

  • $\begingroup$ were you able to find an answer to this question? I am dealing with a similar scenario in time series and I would love to hear any insight you have. $\endgroup$ – Greg Aug 23 '18 at 8:02
  • 1
    $\begingroup$ @Greg I couldnt find any direct approach for LSTM or GRU that was useful for me. Most people suggest using ReLU instead of sigmoid, using Adam optimizer like always. However, I found different approaches like Hessian free optimizer or the Echo State Network that seems to work better in this case but I could not test it because there is not a direct implementation in Tensorflow. I do not remember the source where I found it but I'll edit this as soon I find it. $\endgroup$ – Guillem Aug 23 '18 at 9:45
  • $\begingroup$ Have you tried adapting the ideas of attention models? Where the attention makes some sort of weighted decision based directly on ALL previous activations? $\endgroup$ – Pinocchio Jun 4 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.