Time series forecasting with similar days - train/test overlap

Question

I've developed a model that performs electrical load forecasting by taking as input the following, in 1 hour resolution:

• Future 24 hours of temperature
• Historical load from 1 year ago with best matching temperature
• Historical load from 2 years ago with best matching temperature
• Historical load from 3 years ago with best matching temperature
• Historical load from 4 years ago with best matching temperature

The output is the predicted future 24 hours of electrical load, in 1 hour resolution. This is using an LSTM sequence to sequence model, though I don't believe the model matters.

I have 10 years of data, split so that the train and test data are 5 years each. Because I'm using historical data, the 'historical load' inputs will sometimes be from a future year if the time being forecast is early in the period of available data. For example, if the train dataset is from 01-jan-2008 to 01-jan-2013 and a forecast is being performed at 01-june-2010 then the historical loads will be from 2008, 2009, 2011, and 2012.

Now, when this model is used in production I want it to be able to pull the four historical days from as large a pool as possible. So instead of picking historical load from each of the past four years, it will pick the best historical load from each of the past ten or so years, then pick the best four out of those.

1. Is it valid for the model to use historical data from the training set when running in inference mode?
2. What about doing the same when evaluating the model on the test dataset?

I can imagine that by using the same inputs in the test and train situations the network may have over-fit those inputs and will produce an output that it has somewhat memorized. But, I'm not quite sure if this is a big issue or if there's anything else I haven't thought of - hence my qustions :)

Answer 1

If your LSTM uses "future" data in training, then this is not necessarily overfitting, but it is a textbook example of "data leakage" in a time series context. This is of course invalid, and if at all possible, you should try to avoid this.

Whether this degrades your LSTM's performance compared to a clean training setup we can't tell. Try both and see whether it does. I would be more concerned that it might make you believe your predictions would be more accurate than they truly will be.

Also, have you included the fact that in production, you will need temperature forecasts? This adds a source of uncertainty and error that will propagate to your power demand predictions. (It's often surprisingly hard and expensive to get historical forecasts - historical actuals are easier to get. Also, you would need to use the correct "vintage" of temperature forecasts, e.g., three-day-ahead temperature forecasts for forecasting load three days ahead, and five days for five days etc.)