2
$\begingroup$

Since my data is a time series, I've been using an expanding window walk forward validation via Sklearn's TimeSeriesSplit() to tune the hyper-parameters of my NN. However, I've realized that certain months of my data display characteristics unique to that specific month (e.g, consistent spikes at 3 p.m only in February and April, or a massive change during the middle of November), yet I only have data going back a year and 1 month. Therefore when I train on a subset of my data, the model may not see these special patterns. Is there a better way to split my data for training and validation?

$\endgroup$

1 Answer 1

2
$\begingroup$

Since you don't have more than one year, you can't be sure if those massive changes are repeating or not. Maybe there is another external factor affecting only that year, e.g. coronavirus. Normally, train/validation splitting strategy should mimic the real world testing. That is why we use time series cross validation.

Assuming daily data available, the best way would be splitting from the mid-month , predict the other half, and move forward for each month, e.g. use data available up to mid-April, predict the other half of April. Use data available up to mid-May, predict other half of May etc. You can go with daily steps if your training is cheap.

$\endgroup$
4
  • $\begingroup$ Thanks, this is the best approach I've seen so far. What if, however, I know the changes are going to be repeating (e.g the massive change in mid November is likely because of an annual special event, but this event only happens during November every year). $\endgroup$
    – Jeff
    Sep 2, 2020 at 9:50
  • 1
    $\begingroup$ You could use external features (e.g. holiday or other possibly domain specific indicators) But, still, you'd have to train with some of the November for the algorithm to pick up these features because. If there is not, there is nothing but tuning your algorithm so that it's able to pick up the trend in the month (e.g. looking at previous days etc.) with what data available, using the validation approach laid out above. $\endgroup$
    – gunes
    Sep 2, 2020 at 9:58
  • 1
    $\begingroup$ @Jeff "What if I know something that my data doesn't know?" is one classical justification for the use of Bayesian methods. $\endgroup$
    – Chris Haug
    Sep 2, 2020 at 12:24
  • $\begingroup$ Good point. I haven't mention bayesian methods since it'd probably need significant changes in implementation/problem formulation (I assume OP uses rnns). $\endgroup$
    – gunes
    Sep 2, 2020 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.