Multiple time series forecasting: How to split the data for training of a neural network [duplicate]

Question

Use case: I have sales of 90 products during the first 180 days since the product launch. I want to train an LSTM network to predict sales 4 weeks ahead given the last 7 days of sales. The model should be able to predict sales on brand new products (that are not a part of the 90) when they are released.

What is the best way to structure and split the data? Should I simply concatenate the datasets for all the 90 products into one? I have to somehow separate datasets for different products during training so the model doesn't train on the last data points from one dataset and the first data points from the next. I am using Tensorflow, keras in Python.

Answer 1

Your [cross] validation and testing should respect the temporal order in the dataset. This means, at any given date, you should not have access to data at a future date for any of the products. A simple example is given in this post. Therefore, concatenating end to end is not a viable approach.

Since you have different series for different products, this can be formulated as a multivariate time series problem, where you have a 90-dimensional vector each element having the sales amount for that product. There might be some difficulties though if you have missing data for certain products for certain dates, in which you may need to impute or use a different approach. Also, the scales of these variables might need time series normalization/standardization.

If you have irregularities/missing data in your products, you may also try fitting separate models for each. Or, group them into similar groups and fit models to each group. This may help increase your data (i.e. number of time steps) that the model is learning from. Again, you need to respect the temporal order.

Apart from LSTM, I'd suggest trying some classical time series for forming a baseline.