I've been working with ML for sometime now, especially Deep Learning, but I haven't work with Time Series before, and now I started working in a project for Demand Forecasting. I'm studying the statistical / auto-regressive methods and also trying to understand how CNN and LSTM can be used to tackle the problem. But I'm having a hard time sorting some stuff in my head, mainly about how to split the dataset and put the model into production. So, here are my two main doubts:
I started using Time Series Nested Cross-Validation. Alright, I understand that it's not the only option, but I think it fits great to tune my model hyperparameter's and guarantee that it doesn't overfit. Since in production I'll have to forecast the next 90 days, my test set is always 90 days. But here is the question: with statistical / auto-regressive models (like ARIMA), when I finish tuning the parameters, what should I do? Should I use the model with the largest training set to put into production? But wouldn't I be missing 90 days of recent data? Is it safe to retrain it using the whole data and the same parameters so that it doesn't miss this data?
After a lot of research to understand how to use LSTM and other Machine Learning models for Time Series, I understood that the training dataset needs to be transformed into samples with a rolling window. I mean, I pass a window through the dataset with N elements as input and M elements as output with the window going one by one. Alright, but then, how do I split the training dataset into training and validation (to use ModelCheckpoint and EarlyStopping)? I've seen some tutorials using a random split of these generated samples. But I feel that it creates a data leakage between the training and validation set. The other option seems to be splitting in a temporal way before the rolling window process (eg. having 90 days of validation set). It sounds better for me, since no data would be leaked but then, how would I put it into production? If I simply pick the model trained with the largest dataset, it would be missing 90 days from the test set plus 90 days from the validation set. So, it wouldn't pick recent trends. And I don't think it's safe to simply retrain the model with the whole dataset and the same hyperparameters since I wouldn't have a way to early stop the training process.
I understand that I need to retrain my model constantly because the world is changing and it needs to pick new trends of the data. So, after finding the best hyperparameters, I expect the model to be automatically trained with them within a given schedule (every week, for example). But I can't wrap up my head around those doubts. Am I training a model to predict the next 90 days using data of 90 days ago (with the statistical models) or 180 days ago (with ML}?