27
$\begingroup$

I'm looking into using a LSTM (long short-term memory) version of a recurrent neural network (RNN) for modeling timeseries data. As the sequence length of the data increases, the complexity of the network increases. I am therefore curious what length of sequences would be feasible to model with a good accuracy?

I would like to use a relatively simple version of the LSTM without any difficult to implement state-of-the-art approaches. Each observation in my timeseries would likely have 4 numeric variables and the number of observations would be around 100.000 to 1.000.000.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
15
$\begingroup$

It totally depends on the nature of your data and the inner correlations, there is no rule of thumb. However, given that you have a large amount of data a 2-layer LSTM can model a large body of time series problems / benchmarks.

Furthermore, you don't backpropagate-through-time to the whole series but usually to (200-300) last steps. To find the optimal value you can cross-validate using grid search or bayesian optimisation. Furthermore, you can have a look at the parameters here: https://github.com/wojzaremba/lstm/blob/master/main.lua.

So, the sequence length doesn't really affect your model training but it's like having more training examples, that you just keep the previous state instead of resetting it.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Say that I need to do sentiment analysis, which is a many-to-one approach (see karpathy.github.io/2015/05/21/rnn-effectiveness). Each of these senteces are very long (>200 words). If I only backpropagate-through-time the usual 35 steps, wouldn't that be an issue? Since it's supervised learning I assume that it can only backpropagate when it "hits" the binary classification target, y. In this way how would the RNN ever adjust weights based on anything prior to the 35 steps selected for BPTT? $\endgroup$
    – pir
    Aug 17, 2015 at 12:45
  • 2
    $\begingroup$ Well, it wouldn't be an issue as the same weights are reused in the next training step. Furthermore, if you see the source code in step 36 (let's say) the initialization vector is not zero's, but the states of step 35. So, by doing small steps you optimize your cost function using BPTT. $\endgroup$
    – Yannis Assael
    Aug 17, 2015 at 12:48
  • 2
    $\begingroup$ Just to clarify: Is the BPTT run a single time or multiple times for a single sentence? If it is run a single time, then patterns affecting only the first couple of words in the sentence will only affect the hidden state, right? I mean.. the gradients will never be computed with regards to that part of the input. $\endgroup$
    – pir
    Aug 17, 2015 at 12:57
  • 2
    $\begingroup$ I've asked this as separate question that I hope you will look at :) stats.stackexchange.com/questions/167482/… $\endgroup$
    – pir
    Aug 17, 2015 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.