LSTM time series forecasting accuracy [closed]

Question

EDIT3: [Solved] I experimented with the LSTM hyperparameters and tried to reshape or simplify my data, but that barely changed the outcome. So I stepped back from LSTM and tried a simpler approach, as originally suggested by @naive. I still converted my data set, to introduce a time lag (best results were with 3 time steps) as suggested here. I fitted the data into a random forest classifier, and got much better results (accuracy up to 90% so far, with simplified data)

It looks like that either my dataset (~200k samples) was still too small, or the time windows I'm looking at are too short for the LSTM to shine. Or, I was simply to impatient and inexperienced with LSTMs.

So, I'm trying to perform time series forcasting using Keras. So far I get an accuracy of about 45%, and I'd like to know what I could try to improve that. I've read through quite some LSTM examples on time series, and have done some tutorials on it, but now I have my own dataset and I think what I need is somewhat in between of those two examples:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/ https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py

The former is about multivariate time series forecasting, but it's regression, and I want to do classification. The latter is a text generation example, where the character (class) is predicted based on the previous x characters.

What I want to do is predict the type (class) for the next sample, based on multiple features (including the type) of the past samples. I have multiple tables that look like this:

EDIT2 For clarification:

It's keylogging and eye tracking data from a person who translated a text from language A to language B. Now, the type of each activity (row) states whether the person is looking at text A or B, and if he is typing. The assumption we have is, that there can be some kind of pattern behind the process (e.g. "read A" - "read B" - "type some" - repeat). That's why I think the time series is relevant.

I normalized all data, except the time, which I don't use right now, and the Type, which I converted to 1-hot, so that it can be used for the classification. Based on the text generation example, I converted my table into overlapping windows of e.g. 5 rows, and the corresponding label is the Type of the next row (again, 1-hot).

My model looks somewhat like this (tried with different LSTM dimensions, window widths and used features):

model = Sequential()
model.add(LSTM(100, 
      input_shape=(window_width, num_feats)))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
      optimizer="adam",metrics=['acc'])
model.fit(feats, labs,
      batch_size=batch_size,
      epochs=20,
      validation_data=(test_feats,test_labs))

Now, for the results I achieved this way, the accuracy, both training and validation, is around 45%. As you can see in this plot:

Simple guessing, would give a chance of 16% (6 classes). I kind of hoped to reach a better accuracy, and I wonder if/how I could tune my LSTM to achieve improvements. I've read about under/overfitting, and how to improve in both cases, but I'm unsure what is applicable to mine, as the training and validation losses look somewhat strange:

I checked what happens for more epochs, and seems like the training loss keeps decreasing slightly, but the validation loss increases.

Also, I'm currently working on a set of 77 tables with a total of about 46000 samples. I could acquire more data, but do you think that could improve my model?

Probably it was a bad idea to use the sliding windows? Should I reshape my data differently? As you can see in my data example, each row is one event, with a certain duration. I think I've seen an example where the events were sampled at a fixed interval, could that make a model better, or should including the duration (as I do currently) perform similarly?

Alternatively, are there any simpler machine learning algorithms, applicable for that kind of problem? So far, most I found on time series forecasting was about LSTM.

Any thoughts on how to improve my accuracy would be appreciated :)

EDIT1: Oh, also, how can I interpret that the validation loss is kind of oscillating? It keeps jumping up and down between epochs.

Answer 1

Probably it was a bad idea to use the sliding windows?

I don't think so. Because giving time lags to LSTM is like giving it a chance to learn over a sequence, which is your objective. I bet it is a good idea.

So you have built a model. Looking at the loss curves it seems there is a room for improvement of the model. Now what you are looking for is hyperparameter tuning. You want to do a search over the hyperparameter space. You can implement grid search or random search over the hyperparameters.

Tuning is an iterative process that takes a lot of effort and time, so you need to be patient with that. I would recommend some things that have worked for me:

1) Try different optimizers. Give Nadam a shot.

2) Try regularizers.

3) Try different weight initializers.

4) Try different network architectures. Change the units i.e. the dimensionality of the output space. I would recommend decreasing it from 100.

5) Try dropout.

6) Try recurrent dropout.

7) Try different batch sizes and different epochs after tuning the above mentioned parameters.