3
$\begingroup$

This is rather a conceptual question, than technical. I am interested in performing a rolling (sliding) window analysis, where I aim to predict a label ('0' or '1') of the next value of my time-series. For example, consider the time-series data and the array of labels: (I work with Python and sklearn)

ts  = array([11, 15,  3, 18,  6, 10,  9, 25,  7, 15])
lab = array([ 0,  0,  0,  1,  1,  0,  0,  1,  0,  1])

What I am doing is trying to learn a function 'F', which maps the input features (extracted from the window from the 'ts' array) to the binary labels 'lab': F(feat(ts)) -> lab. Conceptually, this problem is equivalent to the labeling of windows of size-k conditioned on the next timestamp.

F(ts[i:i+3]) -> lab[i+3]

Practically: Consider a window of size 3, then we get:

F([11,15, 3]) -> y = 1
F([15, 3,18]) -> y = 1
F([ 3,18, 6]) -> y = 0
F([18, 6,10]) -> y = 0
F([ 6,10, 9]) -> y = 1
F([10, 9,25]) -> y = 0
F([ 9,25, 7]) -> y = 1

(the very last value is unused). The real time-series is much longer.

QUESTION: For training, testing and cross-validation, may I (pretend and) use my instances as i.i.d? What I mean is: can I randomly divide the instances to training, validation and test sets?

Of course they are not i.i.d., but when I naively tried to process my data and to learn a classifier (simple logistic regression), it surprisingly worked very well and I got quite reasonable results for the classification metrics.

EDIT

I'm reading the paper A Note on the Validity of Cross-Validation for Evaluating Time Series Prediction of and authors clearly state, that: "....and CV can and should be used without modification, as in the independent case.."

$\endgroup$
4
$\begingroup$

How you divide your data set into training/test depends on the data you have available and how your model will be used. Ideally you wouldn't randomly separate the time-points, since as you say, they are not independent if there is any temporal signal at all.

If you have multiple time-series then I'd divide the time-series themselves into training and test in whatever fashion you want.

If your training data is a single time-series and you intend to predict future values of this time-series then I'd segment it accordingly. I.e. use the first 60% of the samples as your training data and the remaining 40% as your test. Of course, these sets aren't independent but given the nature of your data this is unavoidable.

If you have a single time-series for training but the actual time-series that you want to predict future values for is entirely separate then I'd still follow the procedure from the above paragraph, but bear in mind that any estimates of model fit you derive are very likely to be inaccurate.

As an aside, I would be tempted to use a Recurrent Neural Network approach to a problem like this. This would allow you to model the temporal aspect elegantly - something like an an LSTM can maintain a memory of previous values without having to explicitly specify a window size. Of course, if you wish to use a window approach then you could in theory use any classification algorithm you want.

EDIT

That technical paper looks to have covered the issue in far more depth than my answer so I'd follow their recommendations instead. The main difference from your use case is that they are dealing with standard forecasting the next sample, rather than classification

$\endgroup$
  • $\begingroup$ thanks for your thoughts and mentioning LSTM. I develop a new method for feature extraction using the rolling window and I would definitely benchmark it agains the LSTM. I can't agree more, that data in time-series are dependent, but sometimes it might be skipped and pretended as iid. $\endgroup$ – Arnold Klein Mar 20 '17 at 11:14
  • $\begingroup$ good point (about the difference). Each row in their embedded matrix is indeed \epsilon, and thus it is reasonable to assume that rows are iid. I'm trying to understand how I can use their result in my case. $\endgroup$ – Arnold Klein Mar 20 '17 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.