Here is an idea, you could have two RNNs, one that models the sequence of $x$ (RNN A) and another that models the sequence of tokens (words, n-grams) of the message (RNN B).
Then for RNN A, before outputting the next value of $x$ you concatenate its output with the output RNN B and pass them both as features to a fully connected layer.
Here is a crude drawing of what I mean:
This way the next predicted value of $x$ will take into consideration both previous values of $x$ and the message received at that time. If you don't receive a message at all times I would just have a message with the blank token as input for those times.
I don't know if this has been done before, if it works or how computationally feasible it is, but it sounds like a fun architecture to implement in tensorflow.
Here is some tensorflow pseudo code inspired by this tutorial that uses LSTMs (an extension of the RNN):
rnn_a = tf.contrib.rnn.BasicLSTMCell(lstm_size)
rnn_b = tf.contrib.rnn.BasicLSTMCell(lstm_size)
# Initial state of the LSTM memory.
state_a = tf.zeros([batch_size, lstm.state_size])
state_b = tf.zeros([batch_size, lstm.state_size])
probabilities = 
loss = 0.0
# also here you should initialize the weights and biases of the fully connected layer
for x in time_series:
for token in message:
output_b, state_b = rnn_b(token, state_b)
output_a, state_a = rnn_a(x, state_a)
#fully connected layer
predicted_x = tf.concat(output_a, output_b) * weights + biases
# target_x should be the next x in the time series
loss += loss_function(predicted_x, target_x)
Note that this is just pseudo code and won't work on its own and there are probably better ways of doing it, but you can get the general idea.
I strongly recommend you read through the tutorial I mentioned, and if you've never implemented an RNN or used tensorflow before maybe try something simpler first to get acquainted with the technology.