# Time series analysis of hybrid data with RNN?

I have a time series $$x_t\in\mathbb{R}^n$$ where $$t=1,\dots T$$. For some $$t$$, I have also a message which is a piece of text, i.e. sequence of characters of unspecified length. I assume that the messages provide some information about the time series.

My goal is to obtain a precise model of $$x$$, taking advantage of both past $$x$$ values and messages.

How to do it? What is the best model?

I am considering recurrent neural networks in tensorflow. How would you model and train this problem?

My thoughts so far (brute force a little bit): to define a RNN with $$n$$ inputs, primarily for $$x$$. I intend to use it for "reading" all the history. That means when a message is received at time $$t$$, I split it into $$n$$-tuples of characters and I use them as input to the network. All inputs normalized, last $$n$$-tuple may be prolonged by blank characters if needed.

• Can you say a little more about the structure (if any) of the messages? E.g. are there a small number of possible, pre-determined messages, a fixed format with minor individual variations, or is each message completely unique? Do they have a constant, or at least known maximum length? Do you need to parse the whole message, or is there a pre-determined way to extract out the relevant, unique pieces? – user20160 Jul 13 '17 at 8:12
• The messages contain general comments in English. They may contain some quantitative and qualitative information, some sentiment but there's nothing obviously systematic about them. – Karel Macek Jul 18 '17 at 12:54
• Can you give a toy example of your data? – user0 Jul 18 '17 at 20:47
• What do your targets look like? Is there one at each time step t, or are you trying to process a sequence of x and then get an output? Also about how long are your messages? I have an idea but it may struggle on longer messages. – Frobot Jul 18 '17 at 21:59
• The message is typically (97%) shorter than 300 characters. The maximum length is about 1000. The target is to predict next 10 steps of $x$. – Karel Macek Jul 19 '17 at 10:55

Here is an idea, you could have two RNNs, one that models the sequence of $x$ (RNN A) and another that models the sequence of tokens (words, n-grams) of the message (RNN B).

Then for RNN A, before outputting the next value of $x$ you concatenate its output with the output RNN B and pass them both as features to a fully connected layer.

Here is a crude drawing of what I mean: This way the next predicted value of $x$ will take into consideration both previous values of $x$ and the message received at that time. If you don't receive a message at all times I would just have a message with the blank token as input for those times. I don't know if this has been done before, if it works or how computationally feasible it is, but it sounds like a fun architecture to implement in tensorflow.

Here is some tensorflow pseudo code inspired by this tutorial that uses LSTMs (an extension of the RNN):

rnn_a = tf.contrib.rnn.BasicLSTMCell(lstm_size)
rnn_b = tf.contrib.rnn.BasicLSTMCell(lstm_size)
# Initial state of the LSTM memory.
state_a = tf.zeros([batch_size, lstm.state_size])
state_b = tf.zeros([batch_size, lstm.state_size])
probabilities = []
loss = 0.0
# also here you should initialize the weights and biases of the fully connected layer
for x in time_series:
for token in message:
output_b, state_b = rnn_b(token, state_b)
output_a, state_a = rnn_a(x, state_a)
#fully connected layer
predicted_x = tf.concat(output_a, output_b) * weights + biases
# target_x should be the next x in the time series
loss += loss_function(predicted_x, target_x)


Note that this is just pseudo code and won't work on its own and there are probably better ways of doing it, but you can get the general idea. I strongly recommend you read through the tutorial I mentioned, and if you've never implemented an RNN or used tensorflow before maybe try something simpler first to get acquainted with the technology.

• Thank you. Could you please add a little bit more information about the way how to implement it in tensorflow? – Karel Macek Jul 20 '17 at 5:09
• I added some tensorflow pseudo code to the answer, I hope it helps! – Miguel Jul 20 '17 at 8:57
• This was the idea I had also. Probably your best bet – Frobot Jul 23 '17 at 19:36