3
$\begingroup$

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package).

Let's say you have 80 days of weather data. You want to use 3 prior days of data to predict the 4th day's weather. This means you in total have 77 total observations. Let's say you want to keep 20 for validation and 17 for test, leaving you with 40 for training. What do we generally do next?

Would we just randomly select 40 out of 77 and use it to train? And then randomly select 20 for validation (which will be used to tune our hyperparameters)?

Or do we usually use the first 40 observations to train, next 20 for validation, and final 17 for testing?

$\endgroup$

1 Answer 1

4
$\begingroup$

You don't randomly split in time-series datasets because it doesn't respect the temporal order and causes data-leakage, e.g. unintentionally inferring the trend of future samples.

One approach is as you suggested: first 40 for training, next 20 for validation and final 17 for testing. Another similar way to do is time-series cross-validation, e.g. fold 1: 40 train + 5 valid, fold 2: 45 train + 5 valid ... all respecting the temporal order. And, while testing, you can still partition the final 17 and as you did in cross-validation, again respecting the temporal order.

$\endgroup$
7
  • $\begingroup$ Thanks! I guess I am not seeing why randomly selecting your training set will disrespect temporal order since I am keeping 4 sequential days together, when selecting. $\endgroup$
    – confused
    Mar 7, 2020 at 23:52
  • $\begingroup$ But, you're peeking on the future when using future samples in your training set and past samples in your test set. $\endgroup$
    – gunes
    Mar 8, 2020 at 0:01
  • $\begingroup$ How so? If I believe the past 3 days predicts the 4th day, does it really matter where the 4 data points come from? I'm only using the 3 prior days to predict the 4th and am looking for the important features in the 3 days prior. I'm not seeing how any future is brought into my training data. The features are things from the 3 prior days and the response is the 4th day. As long as that order is kept for each observation, I don't see how the future is mixed in. $\endgroup$
    – confused
    Mar 8, 2020 at 0:09
  • $\begingroup$ First of all, you can't do it in real time. This should give you a heads up why you can't do it. Secondly, assume that in your test set your series' trend changes, which might mean slightly different relation between your past 3 samples and the current sample. If you respect the temporal order, you'll have no way of predicting it, but if you don't, you'll also put that information into your model and use it. So, your results will be more optimistic. $\endgroup$
    – gunes
    Mar 8, 2020 at 0:15
  • 1
    $\begingroup$ Are there any confusions about the answer @confused ? $\endgroup$
    – gunes
    Mar 26, 2020 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.