Using RNN (LSTM) for predicting the timeseries vectors (Theano)

Question

I have very simple problem but I cannot find a right tool to solve it.

I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and then make it to predict new sequence of vectors of length $n$ based on several priming vectors.

I cannot find simple implementation which would done this. My base language is Python, but anything what doesn't install for days will hold.

I tried to use Lasagne, but implementation of RNN is not ready yet and it's in separated package nntools. Anyway, I tried the latter one but I can't figure out how to train it, then prime it by some test vectors and let it predict the newone(s). Blocks are the same problem - no documentation is available for LSTM RNN, although it seems that there are some classes and functions which could work (e.g. blocks.bricks.recurrent).

There are several implementation of RNN LSTM in Theano, like GroundHog, theano-rnn, theano_lstm and code for some papers, but non of those have tutorial or guide how to do what I want.

The only usable solution I've found was using Pybrain. But unfortunately it lacks the features of Theano (mainly GPU computation) and is orphaned (no new features and support).

Does anyone know where I could find what I'm asking for? Easy to work with RNN LSTM for predicting sequences of vectors?

Edit:

I tried Keras like this:

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.embeddings import Embedding
from keras.layers.recurrent import LSTM

model = Sequential()
model.add(Embedding(12, 256))
model.regularizers = []
model(LSTM(256, 128, activation='sigmoid', 
               inner_activation='hard_sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(128, 12))
model.add(Activation('sigmoid'))
model.compile(loss='mean_squared_error', optimizer='rmsprop')

but I'm getting this error when trying to fit it model.fit(X_train, y_train, batch_size=16, nb_epoch=10)

IndexError: index 800 is out of bounds for axis 1 with size 12

while X_train and y_train are arrays of arrays (of length 12), e.g. [[i for i in range(12)] for j in range(1000)]

Answer 1

I finally found a way and documented it on my blog here.

There is comparison of several frameworks and then also one implementation in Keras.

Answer 2

I would suggest the following:

0. Theano is really powerful but yes the cod can be difficult sometimes to start with

1. I would suggest you to check out breze: https://github.com/breze-no-salt/breze/blob/master/notebooks/recurrent-networks/RNNs%20for%20Piano%20music.ipynb which is slightly easier to be understood and has an LSTM module as well. Furthermore, an intresting choice is autograd by Harvards, which does automatic symbolic differentiation of numpy functions https://github.com/HIPS/autograd/blob/master/examples/lstm.py and therefore you can easily understand whats going on.

2. I'm a python fan but this is my personal preference. Have you considered using Torch7 is the most user-friendly framework for neural networks and is also used by Google Deepmind and Facebook AI? You can check this very interesting blog post about RNNs http://karpathy.github.io/2015/05/21/rnn-effectiveness/. Additionally, an LSTM implementation is available in the github repo of the post, while an alternative is the rnn package https://github.com/Element-Research/rnn.

Answer 3

I have tested LSTM predicting some time sequence with Theano. I found that for some smooth curve, it can be predicted properly. However for some zigzag curve . It's hard to predict. The detailed article are as below: Predict Time Sequence with LSTM

The predicted result can be shown as follow: