2
$\begingroup$

As far as I understand many RNN:s are trained with back propagation over a sequence of $k$ datapoints.

The RNN is "unrolled" for each datapoint, i.e. its output is fed into itself together with the next datapoint, to compute the gradient of the loss function for these $k$ datapoints.

How is this $k$ chosen? And what consequences does different choices of $k$ have?

(If results are design-specific I am mostly interested in LSTM networks.)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ All the research I have read implies that the longer k the better, until you run into vanishing gradient problems. But a question I asked here - stats.stackexchange.com/questions/238496/… seems to show the exact opposite. $\endgroup$
    – Frobot
    Oct 17, 2016 at 18:27
  • $\begingroup$ Thank you for replying and for asking similar questions. Regarding your question - (I cannot comment there due to stats reputation requirements) I was thinking SIBP should clearly outperform BPTTT for problems where RNN-memory is much less important than the input at the same timestep. I.e. that information at the same timestep is basically enough to predict and the RNN-memory unnecessary. I don't know if this is relevant for your case, but if this is the problem here it should be easy to check. I'm curious about more results regarding this topic so please keep us updated :) $\endgroup$
    – LearnOPhile
    Oct 18, 2016 at 13:38

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.