0
$\begingroup$

Suppose I have a time series dataset with $t=1$ to $t=100$, and I want to take 80% (80 observations) as a training set and 20% (20 observations) as a test set. This link says that we cannot randomly select the observations:

In the case of time series, the cross-validation is not trivial. We cannot choose random samples and assign them to either the test set or the train set because it makes no sense to use the values from the future to forecast values in the past. In simple word we want to avoid future-looking when we train our model. There is a temporal dependency between observations, and we must preserve that relation during testing.

To me, I do not think so (EDIT: in general). I do not think that we are using the future to predict the past. I mean, there is no prediction in the sense of time. In fact, after taking a training set, we calculate predicted values for the test set. From predicted values under different models (model of $y$ as a function of $t$), the test set "chooses the best model".

In summary, we do calculate predicted values, we do not make prediction (in the sense of time). Therefore, I do not think that there is a barrier preventing us from randomly selecting training and test sets.

What do you think?

EDIT: I think we can do a random selection if we model $y$ as function of $t$. We may not be able to do that if we are model $y$ as a function of previous $y$ and $t$.

$\endgroup$
1
$\begingroup$

If your data is time-dependant, you can’t randomly split train and test in a time-series. The ideal would be to use a threshold and use it as a way to split.

E.x. From t=1 to t=80 in train and from t=81 to t=100 in test.

If you don’t do this, you will incur in data leakage. And yes, you are using the future to predict the past.

As a example, suposse in your train end up t=60 and in your test t= 40, you are training with the future and predicting the past.

If your data has a date field but your response variable it’s not time dependant, you can drop the date and do a random split.

$\endgroup$
1
  • $\begingroup$ What if I am modeling $y$ as a function of $t$? As I mentioned above, after fitting model for the training, we do not make prediction, we just compute predicted values (value under model). So it seems to me that there is no past-future direction in time here. $\endgroup$ – TrungDung Dec 11 '20 at 21:29
1
$\begingroup$

You can't meaningfully do a random assignment to training and test samples, at least not if your model should be taken seriously.

By "taking a model seriously", I mean that it could be used in a production context. For time series data, this usually involves taking data up to a certain time point, and predicting future data. Obviously, in such a context, you can't on Friday use next Sunday's data to predict next Saturday's - because you haven't observed Sunday's data yet. But this is what you would be doing if you train your data with a random assignment.

Or put another way: how would you use your model trained on future data to actually predict tomorrow's data point? What would you use as predictors, if it calls for using next week's observations?

There are special cases where you can use "future" data, e.g., if you are interpolating in time. But even then, you should think about how exactly you assign data to the training and the test set, and a simple random assignment would likely lead you astray.

$\endgroup$
2
  • $\begingroup$ What if I am modeling $y$ as a function of $t$? As I mentioned above, after fitting model for the training, we do not make prediction, we just compute predicted values (value under model). So it seems to me that there is no past-future direction in time here. $\endgroup$ – TrungDung Dec 11 '20 at 21:28
  • $\begingroup$ If you are only modeling $y$ as a function of $t$, then of course you can randomly split the total time. The reason is that we know future $t$s. I doubt this happens often. $\endgroup$ – Stephan Kolassa Dec 12 '20 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.