Time series analysis of hybrid data with RNN?

Question

I have a time series $x_t\in\mathbb{R}^n$ where $t=1,\dots T$. For some $t$, I have also a message which is a piece of text, i.e. sequence of characters of unspecified length. I assume that the messages provide some information about the time series.

My goal is to obtain a precise model of $x$, taking advantage of both past $x$ values and messages.

How to do it? What is the best model?

I am considering recurrent neural networks in tensorflow. How would you model and train this problem?

My thoughts so far (brute force a little bit): to define a RNN with $n$ inputs, primarily for $x$. I intend to use it for "reading" all the history. That means when a message is received at time $t$, I split it into $n$-tuples of characters and I use them as input to the network. All inputs normalized, last $n$-tuple may be prolonged by blank characters if needed.

Answer 1

Here is an idea, you could have two RNNs, one that models the sequence of $x$ (RNN A) and another that models the sequence of tokens (words, n-grams) of the message (RNN B).

Then for RNN A, before outputting the next value of $x$ you concatenate its output with the output RNN B and pass them both as features to a fully connected layer.

Here is a crude drawing of what I mean: This way the next predicted value of $x$ will take into consideration both previous values of $x$ and the message received at that time. If you don't receive a message at all times I would just have a message with the blank token as input for those times. I don't know if this has been done before, if it works or how computationally feasible it is, but it sounds like a fun architecture to implement in tensorflow.

Here is some tensorflow pseudo code inspired by this tutorial that uses LSTMs (an extension of the RNN):

rnn_a = tf.contrib.rnn.BasicLSTMCell(lstm_size)
rnn_b = tf.contrib.rnn.BasicLSTMCell(lstm_size)
# Initial state of the LSTM memory.
state_a = tf.zeros([batch_size, lstm.state_size])
state_b = tf.zeros([batch_size, lstm.state_size])
probabilities = []
loss = 0.0
# also here you should initialize the weights and biases of the fully connected layer
for x in time_series:
    for token in message:
        output_b, state_b = rnn_b(token, state_b)
    output_a, state_a = rnn_a(x, state_a)
    #fully connected layer
    predicted_x = tf.concat(output_a, output_b) * weights + biases
    # target_x should be the next x in the time series
    loss += loss_function(predicted_x, target_x)

Note that this is just pseudo code and won't work on its own and there are probably better ways of doing it, but you can get the general idea. I strongly recommend you read through the tutorial I mentioned, and if you've never implemented an RNN or used tensorflow before maybe try something simpler first to get acquainted with the technology.