Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it.

I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and then make it to predict new sequence of vectors of length $n$ based on several priming vectors.

I cannot find simple implementation which would done this. My base language is Python, but anything what doesn't install for days will hold.

I tried to use Lasagne, but implementation of RNN is not ready yet and it's in separated package nntools. Anyway, I tried the latter one but I can't figure out how to train it, then prime it by some test vectors and let it predict the newone(s). Blocks are the same problem - no documentation is available for LSTM RNN, although it seems that there are some classes and functions which could work (e.g. blocks.bricks.recurrent).

There are several implementation of RNN LSTM in Theano, like GroundHog, theano-rnn, theano_lstm and code for some papers, but non of those have tutorial or guide how to do what I want.

The only usable solution I've found was using Pybrain. But unfortunately it lacks the features of Theano (mainly GPU computation) and is orphaned (no new features and support).

Does anyone know where I could find what I'm asking for? Easy to work with RNN LSTM for predicting sequences of vectors?

Edit:

I tried Keras like this:

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.embeddings import Embedding
from keras.layers.recurrent import LSTM

model = Sequential()
model.regularizers = []
model(LSTM(256, 128, activation='sigmoid',
inner_activation='hard_sigmoid'))
model.compile(loss='mean_squared_error', optimizer='rmsprop')


but I'm getting this error when trying to fit it model.fit(X_train, y_train, batch_size=16, nb_epoch=10)

IndexError: index 800 is out of bounds for axis 1 with size 12


while X_train and y_train are arrays of arrays (of length 12), e.g. [[i for i in range(12)] for j in range(1000)]

• Why do you need RNN if all your sequences have the same length? Train static input vector via ANN would be easier and faster. – itdxer Jun 29 '15 at 11:52
• The vectors come from timeseries. So RNN is necessary I guess? – kotrfa Jun 29 '15 at 13:05
• RNN is a greate for tasks when you don't know input or output vector size. For example you want build network which will describe in English what you can see on picture, so your input can be static image, but output will vary dependently on what you can see on picture. Another example when you want get text translation, there your input and output are both unknown. – itdxer Jun 29 '15 at 13:40
• Thank you for clarification. So there is no reason to use RNN in this task. Ok. – kotrfa Jun 29 '15 at 13:41
• @itdxer's comment is misleading. Even if your data has the same length everywhere, using an RNN can be beneficial. An RNN introduces an assumption of the data, mainly that it is of sequential nature. E.g. translation along the time axis is handled gracefully by RNNs, but not by feed forward methods--they need much more training data to realise that and have many more parameters to estimate. There are many more cases where an ANN will just crash and burn if used instead of an RNN. – bayerj Jul 3 '15 at 8:16

I finally found a way and documented it on my blog here.

There is comparison of several frameworks and then also one implementation in Keras.

I would suggest the following:

0) Theano is really powerful but yes the cod can be diffucult sometimes to start with

1) I would suggest you to check out breze: https://github.com/breze-no-salt/breze/blob/master/notebooks/recurrent-networks/RNNs%20for%20Piano%20music.ipynb which is slightly easier to be understood and has an LSTM module as well. Furthermore, an intresting choice is autograd by Harvards, which does automatic symbolic differentiation of numpy functions https://github.com/HIPS/autograd/blob/master/examples/lstm.py and therefore you can easily understand whats going on.

2) I'm a python fan but this is my personal preference. Have you considered using Torch7 is the most user-friendly framework for neural networks and is also used by Google Deepmind and Facebook AI? You can check this very intresting blog post about RNNs http://karpathy.github.io/2015/05/21/rnn-effectiveness/. Additionally, an LSTM implementation is available in the github repo of the post, while an alternative is the rnn package https://github.com/Element-Research/rnn.

• I have successfully used Karpathy's work for last few weeks. Unfortunately, I'm not able to tweak his algorithm to make prediction of vectors and not sequences of characters. It's also because I'm not very familiar with Lua. Hence, I've passively used Torch7 also, but I don't really find it very friendly. Thank you – kotrfa Jun 29 '15 at 12:25
• Perfect. So let me confirm that you have some $S$ sequences with $N$ linear values, and you want to predict a new sequence sized $N$? Are the N and S constant all the time? – Yannis Assael Jun 29 '15 at 13:05
• I have vector of 12 elements from measurement for every second. I would like to train the net, then prime it e.g. by 5 vectors (of length 12) and let it predict the following vector. Nothing more. I have updated my question with my try using Keras. – kotrfa Jun 29 '15 at 13:08
• you could also use a simple feed forward for that even if your vectors come from a time-series. Just make sure that the number of sequences is constant. – Yannis Assael Jun 29 '15 at 13:25
• From the few articles I've read, like the Karpathy's, I understand that the LSTM is the best choice for sequences, no? Does simple feed forward NN has the "memory feature"? – kotrfa Jun 29 '15 at 13:29

I have tested LSTM predicting some time sequence with Theano. I found that for some smooth curve, it can be predicted properly. However for some zigzag curve . It's hard to predict. The detailed article are as below: Predict Time Sequence with LSTM

The predicted result can be shown as follow: