I'm trying to train a model with data that looks like this:

Binary time series data

There are theoretical reasons to believe that the Y row can be predicted using the X rows, with a window of about 2 seconds. Note that this plot is just one example. There are around 100 time series like that in my dataset for training/validation.

I have a basic knowledge in ML, but never done anything with time series before, so I'm looking for some advice. I know that I can rearrange the data so that each sample will be a long vector containing window_size * features values, and pass it through a logistic regression, but I'm not sure that this is the best approach for a few reasons:

  • The large feature vector (let's say ~200 values) might cause overfitting, isn't it?
  • I saw examples that use LSTM and CRF for time series predictions. So there are probably reasons to use specialized algorithms for the tasks, but I don't know them properly to fully understand them. Will you suggest investing my time in these or stay with the a simpler logistic regression model for this type of data?
  • Using logistic regression won't really take time into account. For example, trying to generate predictions for time series with different sampling period will be impossible without resampling. Are the models that are built specifically for time series take it into account automatically?

Also, this looks like a general problem to me, that probably have a general solution. So, hints to python libraries that are built to process such time series directly (without me having to rearrange the data first) will be great!

  • 1
    $\begingroup$ The classic book on categorical time series regression is this $\endgroup$
    – bdeonovic
    Jul 3, 2017 at 12:16

1 Answer 1


Keras is a high-level API that allows implementation of recurrent neural networks (RNNs). RNNs like LSTMs might be good models for your data, depending on how much there is. Here is a good introductory tutorial.

What you suggest with the time window and logistic regression may perform better or it may perform much worse. Sometimes aggregating past time events is very effective, sometimes not so much. In my experience, it's more effective when there is less data and the function from previous time steps to the response variable is too complicated to learn given the small amount of data. Aggregating might allow you to kind of guide this learning with some domain expertise. (For me, this once outperformed a self-written vanilla RNN, but it may not have outperformed an LSTM or GRU). With a large amount of data, RNNs should outperform this method in terms of predictive performance. Another option is a Hidden Markov Model. One huge benefit of the time-window method however is that the aggregated features are interpretable. I would try both methods and see which is best.

In terms of overfitting, you can use shrinkage with logistic regression or dropout, shrinkage, or decreased nodes with an RNN. If you make sure to check error on a validation set, you can be vigilant about preventing overfitting.

You would have to resample for the RNN to take time into account (they naturally process sequences and it is up to you to make sure that the events in the sequence are evenly spaced). Pandas has a resampling functionality that may come in handy, although it's also possible that processing raw sequences rather than resampled sequences might give better results---again, would not hurt to try both ways.

A good general introduction and more to RNNs is Andrej Karpathy's blog post. Aslo, see Goodfellow et als text Chapter 10.

  • 3
    $\begingroup$ Thanks for the answer @user99889, it significantly helped me to solve the problem! I tried a few different approached but now I'm settled down on K nearest neighbors with a window size of 20 samples. I reduced the amount of features manually, and the model still generate accurate predictions. I also read the tutorial and Karpathy's blog, and tried Keras, but the extra effort doesn't worth it for me. $\endgroup$
    – Nagasaki45
    Jul 5, 2017 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.