EDIT3: [Solved] I experimented with the LSTM hyperparameters and tried to reshape or simplify my data, but that barely changed the outcome. So I stepped back from LSTM and tried a simpler approach, as originally suggested by @naive. I still converted my data set, to introduce a time lag (best results were with 3 time steps) as suggested here. I fitted the data into a random forest classifier, and got much better results (accuracy up to 90% so far, with simplified data)
It looks like that either my dataset (~200k samples) was still too small, or the time windows I'm looking at are too short for the LSTM to shine. Or, I was simply to impatient and inexperienced with LSTMs.
So, I'm trying to perform time series forcasting using Keras. So far I get an accuracy of about 45%, and I'd like to know what I could try to improve that. I've read through quite some LSTM examples on time series, and have done some tutorials on it, but now I have my own dataset and I think what I need is somewhat in between of those two examples:
The former is about multivariate time series forecasting, but it's regression, and I want to do classification. The latter is a text generation example, where the character (class) is predicted based on the previous x characters.
EDIT2 For clarification:
It's keylogging and eye tracking data from a person who translated a text from language A to language B. Now, the type of each activity (row) states whether the person is looking at text A or B, and if he is typing. The assumption we have is, that there can be some kind of pattern behind the process (e.g. "read A" - "read B" - "type some" - repeat). That's why I think the time series is relevant.
I normalized all data, except the time, which I don't use right now, and the Type, which I converted to 1-hot, so that it can be used for the classification. Based on the text generation example, I converted my table into overlapping windows of e.g. 5 rows, and the corresponding label is the Type of the next row (again, 1-hot).
My model looks somewhat like this (tried with different LSTM dimensions, window widths and used features):
model = Sequential() model.add(LSTM(100, input_shape=(window_width, num_feats))) model.add(Dense(num_classes)) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', optimizer="adam",metrics=['acc']) model.fit(feats, labs, batch_size=batch_size, epochs=20, validation_data=(test_feats,test_labs))
Now, for the results I achieved this way, the accuracy, both training and validation, is around 45%. As you can see in this plot:
Simple guessing, would give a chance of 16% (6 classes). I kind of hoped to reach a better accuracy, and I wonder if/how I could tune my LSTM to achieve improvements. I've read about under/overfitting, and how to improve in both cases, but I'm unsure what is applicable to mine, as the training and validation losses look somewhat strange:
I checked what happens for more epochs, and seems like the training loss keeps decreasing slightly, but the validation loss increases.
Also, I'm currently working on a set of 77 tables with a total of about 46000 samples. I could acquire more data, but do you think that could improve my model?
Probably it was a bad idea to use the sliding windows? Should I reshape my data differently? As you can see in my data example, each row is one event, with a certain duration. I think I've seen an example where the events were sampled at a fixed interval, could that make a model better, or should including the duration (as I do currently) perform similarly?
Alternatively, are there any simpler machine learning algorithms, applicable for that kind of problem? So far, most I found on time series forecasting was about LSTM.
Any thoughts on how to improve my accuracy would be appreciated :)
EDIT1: Oh, also, how can I interpret that the validation loss is kind of oscillating? It keeps jumping up and down between epochs.