1
$\begingroup$

I have to predict time series that has unequal time steps using RNNs. Just to see if the approach is correct, I'm trying to predict (sine curve) using RNN with input as time and trying to predict f(t).
The important to notice is that time steps intervals are not equal.

0.006 0.224 0.322 0.347 0.418 0.45 0.462 0.471 0.499 0.501

Corresponding input and output are as follows:

0.006,0.0059999640000648
0.224,0.22213145663397
0.322,0.316464401053724
0.347,0.340078150505108
0.418,0.405933460875763
0.45,0.43496553411123
0.462,0.445739322869916
0.471,0.453777627075545
0.499,0.478547716475827
0.501,0.48030288130708

Actually, as you can see in this image, the function is sine: enter image description here

Now my question is, if I feed the following type of input to LSTM RNN,

0.006
0.224
0.322
0.347
0.418
0.45
0.462
0.471
0.499
0.501

And output as:

0.48030288130708

is it correct? That is, I'm feeding 10 timesteps as inputs, and 1 output value (output at 10th step as output of the series) and this forms one example. Next example will start from 2nd value (0.224) and will be 10 time steps, and output will be sine value of 10th step.. and so on. And then I'm applying padding and masking and then trying to predict.

Is my approach correct? Any pointers will be helpful. And broader question is can we use time steps as input to predict unequal step time series using RNNs. Thank you.

$\endgroup$
2
$\begingroup$

First of all, you should give the RNN all the outputs over time. In my example I train the model with just one sample but each timestamp is a new training example.

Second, you need to think about what useful information the earlier time points contain. If you had to guess the value of the unknown function at 0.224, knowing that the previous value of x (0.006) contains absolutely nothing useful to predict the correct output of 0.22213145663397.

A common approach with time series is to use the previous value of the output variable as one of the input features. So in practice you could also use the previous value of 0.0059999640000648 to predict the second output 0.22213145663397. In addition to this an useful feature could be to use the difference to the previous x instead of just the absolute value of x.

You can try to start from this example implemented with Keras:

import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, LSTM, TimeDistributed

diff = np.random.random_sample(10000)*0.005
x = np.cumsum(diff)
y = np.sin(x)
plt.scatter(x,y)
plt.show()

enter image description here

# Create output and input matrixes X and Y of shape (samples, time steps, features)
# Add a new feature of the previous value of the target function.
previous_y = np.roll(y,1)
previous_y[0] = 0

X = np.column_stack((x, diff, previous_y))
X = np.expand_dims(X, axis=0)

Y = y.reshape((1, -1, 1))

train_size = X.shape[1]*2/3

model = Sequential()
model.add(LSTM(50, return_sequences=True, input_shape=(None, X.shape[2])))
model.add(LSTM(50, return_sequences=True))
model.add(TimeDistributed(Dense(1, activation='tanh')))      # The outputs are expected to be between -1 and 1 so tanh is a possible activation function here.
model.compile(loss='mean_squared_error', optimizer='adam')   # Using MSE because this is a regression task

model.fit(X[:,0:train_size,:], Y[:,0:train_size,:], verbose=1, nb_epoch=10)

prediction = model.predict(X)
prediction = prediction.reshape(-1)
plt.plot(x,prediction)
plt.plot(x,y)
plt.legend(("Prediction","Actual"))
plt.show()

enter image description here

As you can see, the predictions make some kind of sense in the training set, but the model does not really work at all towards the end, which was not used to train the model. More training could solve this, or also by removing the x from the features by changing line 7 to

X = np.column_stack((diff, previous_y))

Improved results

Even though the model looks nice, there is a lot to do. A baseline model which just returns the previous value of the target function would probably give a better RMSE. RNNs require a lot of data and training, so even to get this simple example to work properly will require a lot of work. Keep up with learning new!

$\endgroup$
  • $\begingroup$ Thank you! I'm kind of new to machine learning in python.. it'll take me some time to digest your code.. $\endgroup$ – tired and bored dev Dec 30 '17 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.