I have a time series $x_t\in\mathbb{R}^n$ where $t=1,\dots T$. For some $t$, I have also a message which is a piece of text, i.e. sequence of characters of unspecified length. I assume that the messages provide some information about the time series.

My goal is to obtain a precise model of $x$, taking advantage of both past $x$ values and messages.

How to do it? What is the best model?

I am considering recurrent neural networks in tensorflow. How would you model and train this problem?

My thoughts so far (brute force a little bit): to define a RNN with $n$ inputs, primarily for $x$. I intend to use it for "reading" all the history. That means when a message is received at time $t$, I split it into $n$-tuples of characters and I use them as input to the network. All inputs normalized, last $n$-tuple may be prolonged by blank characters if needed.

  • $\begingroup$ Can you say a little more about the structure (if any) of the messages? E.g. are there a small number of possible, pre-determined messages, a fixed format with minor individual variations, or is each message completely unique? Do they have a constant, or at least known maximum length? Do you need to parse the whole message, or is there a pre-determined way to extract out the relevant, unique pieces? $\endgroup$ – user20160 Jul 13 '17 at 8:12
  • $\begingroup$ The messages contain general comments in English. They may contain some quantitative and qualitative information, some sentiment but there's nothing obviously systematic about them. $\endgroup$ – Karel Macek Jul 18 '17 at 12:54
  • $\begingroup$ Can you give a toy example of your data? $\endgroup$ – user0 Jul 18 '17 at 20:47
  • $\begingroup$ What do your targets look like? Is there one at each time step t, or are you trying to process a sequence of x and then get an output? Also about how long are your messages? I have an idea but it may struggle on longer messages. $\endgroup$ – Frobot Jul 18 '17 at 21:59
  • $\begingroup$ The message is typically (97%) shorter than 300 characters. The maximum length is about 1000. The target is to predict next 10 steps of $x$. $\endgroup$ – Karel Macek Jul 19 '17 at 10:55

Here is an idea, you could have two RNNs, one that models the sequence of $x$ (RNN A) and another that models the sequence of tokens (words, n-grams) of the message (RNN B).

Then for RNN A, before outputting the next value of $x$ you concatenate its output with the output RNN B and pass them both as features to a fully connected layer.

Here is a crude drawing of what I mean: enter image description here This way the next predicted value of $x$ will take into consideration both previous values of $x$ and the message received at that time. If you don't receive a message at all times I would just have a message with the blank token as input for those times. I don't know if this has been done before, if it works or how computationally feasible it is, but it sounds like a fun architecture to implement in tensorflow.

Here is some tensorflow pseudo code inspired by this tutorial that uses LSTMs (an extension of the RNN):

rnn_a = tf.contrib.rnn.BasicLSTMCell(lstm_size)
rnn_b = tf.contrib.rnn.BasicLSTMCell(lstm_size)
# Initial state of the LSTM memory.
state_a = tf.zeros([batch_size, lstm.state_size])
state_b = tf.zeros([batch_size, lstm.state_size])
probabilities = []
loss = 0.0
# also here you should initialize the weights and biases of the fully connected layer
for x in time_series:
    for token in message:
        output_b, state_b = rnn_b(token, state_b)
    output_a, state_a = rnn_a(x, state_a)
    #fully connected layer
    predicted_x = tf.concat(output_a, output_b) * weights + biases
    # target_x should be the next x in the time series
    loss += loss_function(predicted_x, target_x)

Note that this is just pseudo code and won't work on its own and there are probably better ways of doing it, but you can get the general idea. I strongly recommend you read through the tutorial I mentioned, and if you've never implemented an RNN or used tensorflow before maybe try something simpler first to get acquainted with the technology.

  • $\begingroup$ Thank you. Could you please add a little bit more information about the way how to implement it in tensorflow? $\endgroup$ – Karel Macek Jul 20 '17 at 5:09
  • $\begingroup$ I added some tensorflow pseudo code to the answer, I hope it helps! $\endgroup$ – Miguel Jul 20 '17 at 8:57
  • $\begingroup$ This was the idea I had also. Probably your best bet $\endgroup$ – Frobot Jul 23 '17 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.