1
$\begingroup$

EDIT3: [Solved] I experimented with the LSTM hyperparameters and tried to reshape or simplify my data, but that barely changed the outcome. So I stepped back from LSTM and tried a simpler approach, as originally suggested by @naive. I still converted my data set, to introduce a time lag (best results were with 3 time steps) as suggested here. I fitted the data into a random forest classifier, and got much better results (accuracy up to 90% so far, with simplified data)

It looks like that either my dataset (~200k samples) was still too small, or the time windows I'm looking at are too short for the LSTM to shine. Or, I was simply to impatient and inexperienced with LSTMs.

So, I'm trying to perform time series forcasting using Keras. So far I get an accuracy of about 45%, and I'd like to know what I could try to improve that. I've read through quite some LSTM examples on time series, and have done some tutorials on it, but now I have my own dataset and I think what I need is somewhat in between of those two examples:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/ https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py

The former is about multivariate time series forecasting, but it's regression, and I want to do classification. The latter is a text generation example, where the character (class) is predicted based on the previous x characters.

What I want to do is predict the type (class) for the next sample, based on multiple features (including the type) of the past samples. I have multiple tables that look like this:Example data

EDIT2 For clarification:

It's keylogging and eye tracking data from a person who translated a text from language A to language B. Now, the type of each activity (row) states whether the person is looking at text A or B, and if he is typing. The assumption we have is, that there can be some kind of pattern behind the process (e.g. "read A" - "read B" - "type some" - repeat). That's why I think the time series is relevant.

I normalized all data, except the time, which I don't use right now, and the Type, which I converted to 1-hot, so that it can be used for the classification. Based on the text generation example, I converted my table into overlapping windows of e.g. 5 rows, and the corresponding label is the Type of the next row (again, 1-hot).

My model looks somewhat like this (tried with different LSTM dimensions, window widths and used features):

model = Sequential()
model.add(LSTM(100, 
      input_shape=(window_width, num_feats)))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
      optimizer="adam",metrics=['acc'])
model.fit(feats, labs,
      batch_size=batch_size,
      epochs=20,
      validation_data=(test_feats,test_labs))

Now, for the results I achieved this way, the accuracy, both training and validation, is around 45%. As you can see in this plot:

enter image description here

Simple guessing, would give a chance of 16% (6 classes). I kind of hoped to reach a better accuracy, and I wonder if/how I could tune my LSTM to achieve improvements. I've read about under/overfitting, and how to improve in both cases, but I'm unsure what is applicable to mine, as the training and validation losses look somewhat strange:

enter image description here

I checked what happens for more epochs, and seems like the training loss keeps decreasing slightly, but the validation loss increases.

Also, I'm currently working on a set of 77 tables with a total of about 46000 samples. I could acquire more data, but do you think that could improve my model?

Probably it was a bad idea to use the sliding windows? Should I reshape my data differently? As you can see in my data example, each row is one event, with a certain duration. I think I've seen an example where the events were sampled at a fixed interval, could that make a model better, or should including the duration (as I do currently) perform similarly?

Alternatively, are there any simpler machine learning algorithms, applicable for that kind of problem? So far, most I found on time series forecasting was about LSTM.

Any thoughts on how to improve my accuracy would be appreciated :)

EDIT1: Oh, also, how can I interpret that the validation loss is kind of oscillating? It keeps jumping up and down between epochs.

$\endgroup$

closed as too broad by Sycorax, kjetil b halvorsen, Bernhard, Stephan Kolassa, gung Jun 29 '18 at 13:49

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ I don't know if its okay to say it here but it's great that you acknowledged that one of the ideas worked which I had removed from the answer shortly after your edit. Awesome! $\endgroup$ – naive Jun 20 '18 at 13:13
1
$\begingroup$

Probably it was a bad idea to use the sliding windows?

I don't think so. Because giving time lags to LSTM is like giving it a chance to learn over a sequence, which is your objective. I bet it is a good idea.

So you have built a model. Looking at the loss curves it seems there is a room for improvement of the model. Now what you are looking for is hyperparameter tuning. You want to do a search over the hyperparameter space. You can implement grid search or random search over the hyperparameters.

Tuning is an iterative process that takes a lot of effort and time, so you need to be patient with that. I would recommend some things that have worked for me:

1) Try different optimizers. Give Nadam a shot.

2) Try regularizers.

3) Try different weight initializers.

4) Try different network architectures. Change the units i.e. the dimensionality of the output space. I would recommend decreasing it from 100.

5) Try dropout.

6) Try recurrent dropout.

7) Try different batch sizes and different epochs after tuning the above mentioned parameters.

$\endgroup$
  • $\begingroup$ Do I have a reason to believe that there is a dependence of a feature's value at say time t on time t-1? Yes, let me explain my data a bit more It's keylogging and eye tracking data from a person who translated a text from language A to language B Now, the type of each activity states whether the person is looking at text A or B, and if he is typing. The assumption we have is, that there can be some kind of pattern behind the process (e.g. "read A" - "read B" - "type some" - repeat) Which in my opinion couldn't be recognized if we only look at one sample, to predict the next $\endgroup$ – KlausB Jun 17 '18 at 18:36
  • $\begingroup$ And, thanks for the list. I'm gonna check all those things out and see if I can make improvements :) $\endgroup$ – KlausB Jun 17 '18 at 18:52
  • $\begingroup$ @KlausB have edited the answer accordingly. Thanks $\endgroup$ – naive Jun 17 '18 at 19:04
  • $\begingroup$ Okay, I have to revise my old comment. Looks like you were right, I should have tried simpler approaches. I tried to tune different gears, and almost nothing changed. Now I adapted my dataset to feed it into a random forest classifier, while still using time lags (but only up to 5 or so). And I almost immediately reached 65% accuracy for the 6 classes. I tried simplifying the classes (reduce to 3), which changed almost nothing for the LSTM, but boosted my random forest to almost 90% :) $\endgroup$ – KlausB Jun 20 '18 at 11:46
  • $\begingroup$ That's great news! $\endgroup$ – naive Jun 20 '18 at 11:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.