9
$\begingroup$

I have a model to predict +1 day ahead of this time series.

enter image description here

Looking at the chart you can notice some seasonality every 5 days. I suspect using a moving window as training set could help me making a better prediction.

However I want to programmatically find the best Moving Window Size for my model. Are these approaches below valid? Should I do something different?

Approach 1. I run the model on the historical data, with any possible Window Size, I pick the window size that minimises the prediction error. This approach is simple and fast, but I am afraid it overfits the Window Size to historical that. Right?

enter image description here

Approach 2. I use cross-validation (LOOCV) to get a more realistic prediction error. Is this better/worse than Approach 1?

enter image description here

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ Window size for what model? What role does window size play in your model? What are you trying to achieve with your analysis? Give us more information about the context. $\endgroup$
    – Tim
    Commented Sep 11, 2018 at 12:44
  • $\begingroup$ I am dealing with a similar problem at the moment, that is how best to pick a window size. Which approach did you go with and what were your results?Did you encounter any problems using cross validation to tune your models for time series data at all? $\endgroup$
    – Aesir
    Commented Nov 22, 2018 at 6:47
  • $\begingroup$ @Aesir at the end I chose Approach 3 (not-listed above 😛) I simply do a walkforward analysis. At time t, I find the window size that works best on the past data points $x_0$ to $x_{t-1}$, then I use that window size to predict $x_t$. This approach resembles best what happens in reality, where I run my algorithm every day, to predict the following day. In my case, with this approach, it turned out that no window size worked. So I reverted to using no window at all, rather just using all the past data from 0 to t-1. $\endgroup$
    – elemolotiv
    Commented Nov 22, 2018 at 11:14
  • $\begingroup$ Thanks. So to be clear you took iterated through every window size possible on your data and for each window size performed a walkforward analysis. Then you computed an average accuracy for each window size and then used that to pick a final window size that worked for your case? $\endgroup$
    – Aesir
    Commented Nov 22, 2018 at 12:15
  • 1
    $\begingroup$ Exactly- everyday I iterated all possible windowsizes and picked the one that worked best that day. The next day all over again $\endgroup$
    – elemolotiv
    Commented Nov 22, 2018 at 13:26

1 Answer 1

Reset to default
2
$\begingroup$

I would say that your first approach seems like a good start, it seems better to me than your second one. Your assessment of the possible risks, are correct as you could interpret this as tweaking hyperparameters on the test set come with the risk with a performance estimate is too optimistic. It could be an idea to tweak your first approach to include a validation set on which you can tweak the window sizes and then only use the test set to obtain a performance estimate. In case you are unfamiliar with this, I quite like what is discussed in this thread: What is the difference between test set and validation set?

Hope this helps!

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ thanks! Sure I could include one single random validation set in the 1st approach, but the leave-one-out crossvalidation in the 2nd approach is more general because it does not depend on how I randomly choose the validation set. So why you don't like the 2nd approach? :) $\endgroup$
    – elemolotiv
    Commented Sep 11, 2018 at 14:27
  • $\begingroup$ while CV - if checking the residuals for the absence of serial correlation - then 2nd approach is admissible $\endgroup$
    – JeeyCi
    Commented Dec 30, 2022 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.