2
$\begingroup$

EDIT3: [Solved] I experimented with the LSTM hyperparameters and tried to reshape or simplify my data, but that barely changed the outcome. So I stepped back from LSTM and tried a simpler approach, as originally suggested by @naive. I still converted my data set, to introduce a time lag (best results were with 3 time steps) as suggested here. I fitted the data into a random forest classifier, and got much better results (accuracy up to 90% so far, with simplified data)

It looks like that either my dataset (~200k samples) was still too small, or the time windows I'm looking at are too short for the LSTM to shine. Or, I was simply to impatient and inexperienced with LSTMs.

So, I'm trying to perform time series forcasting using Keras. So far I get an accuracy of about 45%, and I'd like to know what I could try to improve that. I've read through quite some LSTM examples on time series, and have done some tutorials on it, but now I have my own dataset and I think what I need is somewhat in between of those two examples:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/ https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py

The former is about multivariate time series forecasting, but it's regression, and I want to do classification. The latter is a text generation example, where the character (class) is predicted based on the previous x characters.

What I want to do is predict the type (class) for the next sample, based on multiple features (including the type) of the past samples. I have multiple tables that look like this:Example data

EDIT2 For clarification:

It's keylogging and eye tracking data from a person who translated a text from language A to language B. Now, the type of each activity (row) states whether the person is looking at text A or B, and if he is typing. The assumption we have is, that there can be some kind of pattern behind the process (e.g. "read A" - "read B" - "type some" - repeat). That's why I think the time series is relevant.

I normalized all data, except the time, which I don't use right now, and the Type, which I converted to 1-hot, so that it can be used for the classification. Based on the text generation example, I converted my table into overlapping windows of e.g. 5 rows, and the corresponding label is the Type of the next row (again, 1-hot).

My model looks somewhat like this (tried with different LSTM dimensions, window widths and used features):

model = Sequential()
model.add(LSTM(100, 
      input_shape=(window_width, num_feats)))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
      optimizer="adam",metrics=['acc'])
model.fit(feats, labs,
      batch_size=batch_size,
      epochs=20,
      validation_data=(test_feats,test_labs))

Now, for the results I achieved this way, the accuracy, both training and validation, is around 45%. As you can see in this plot:

enter image description here

Simple guessing, would give a chance of 16% (6 classes). I kind of hoped to reach a better accuracy, and I wonder if/how I could tune my LSTM to achieve improvements. I've read about under/overfitting, and how to improve in both cases, but I'm unsure what is applicable to mine, as the training and validation losses look somewhat strange:

enter image description here

I checked what happens for more epochs, and seems like the training loss keeps decreasing slightly, but the validation loss increases.

Also, I'm currently working on a set of 77 tables with a total of about 46000 samples. I could acquire more data, but do you think that could improve my model?

Probably it was a bad idea to use the sliding windows? Should I reshape my data differently? As you can see in my data example, each row is one event, with a certain duration. I think I've seen an example where the events were sampled at a fixed interval, could that make a model better, or should including the duration (as I do currently) perform similarly?

Alternatively, are there any simpler machine learning algorithms, applicable for that kind of problem? So far, most I found on time series forecasting was about LSTM.

Any thoughts on how to improve my accuracy would be appreciated :)

EDIT1: Oh, also, how can I interpret that the validation loss is kind of oscillating? It keeps jumping up and down between epochs.

$\endgroup$
1
  • $\begingroup$ I don't know if its okay to say it here but it's great that you acknowledged that one of the ideas worked which I had removed from the answer shortly after your edit. Awesome! $\endgroup$
    – naive
    Jun 20, 2018 at 13:13

1 Answer 1

3
$\begingroup$

Probably it was a bad idea to use the sliding windows?

I don't think so. Because giving time lags to LSTM is like giving it a chance to learn over a sequence, which is your objective. I bet it is a good idea.

So you have built a model. Looking at the loss curves it seems there is a room for improvement of the model. Now what you are looking for is hyperparameter tuning. You want to do a search over the hyperparameter space. You can implement grid search or random search over the hyperparameters.

Tuning is an iterative process that takes a lot of effort and time, so you need to be patient with that. I would recommend some things that have worked for me:

1) Try different optimizers. Give Nadam a shot.

2) Try regularizers.

3) Try different weight initializers.

4) Try different network architectures. Change the units i.e. the dimensionality of the output space. I would recommend decreasing it from 100.

5) Try dropout.

6) Try recurrent dropout.

7) Try different batch sizes and different epochs after tuning the above mentioned parameters.

$\endgroup$
5
  • $\begingroup$ Do I have a reason to believe that there is a dependence of a feature's value at say time t on time t-1? Yes, let me explain my data a bit more It's keylogging and eye tracking data from a person who translated a text from language A to language B Now, the type of each activity states whether the person is looking at text A or B, and if he is typing. The assumption we have is, that there can be some kind of pattern behind the process (e.g. "read A" - "read B" - "type some" - repeat) Which in my opinion couldn't be recognized if we only look at one sample, to predict the next $\endgroup$
    – KlausB
    Jun 17, 2018 at 18:36
  • $\begingroup$ And, thanks for the list. I'm gonna check all those things out and see if I can make improvements :) $\endgroup$
    – KlausB
    Jun 17, 2018 at 18:52
  • $\begingroup$ @KlausB have edited the answer accordingly. Thanks $\endgroup$
    – naive
    Jun 17, 2018 at 19:04
  • $\begingroup$ Okay, I have to revise my old comment. Looks like you were right, I should have tried simpler approaches. I tried to tune different gears, and almost nothing changed. Now I adapted my dataset to feed it into a random forest classifier, while still using time lags (but only up to 5 or so). And I almost immediately reached 65% accuracy for the 6 classes. I tried simplifying the classes (reduce to 3), which changed almost nothing for the LSTM, but boosted my random forest to almost 90% :) $\endgroup$
    – KlausB
    Jun 20, 2018 at 11:46
  • $\begingroup$ That's great news! $\endgroup$
    – naive
    Jun 20, 2018 at 11:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.