1
$\begingroup$

I've developed a model that performs electrical load forecasting by taking as input the following, in 1 hour resolution:

  • Future 24 hours of temperature
  • Historical load from 1 year ago with best matching temperature
  • Historical load from 2 years ago with best matching temperature
  • Historical load from 3 years ago with best matching temperature
  • Historical load from 4 years ago with best matching temperature

The output is the predicted future 24 hours of electrical load, in 1 hour resolution. This is using an LSTM sequence to sequence model, though I don't believe the model matters.

I have 10 years of data, split so that the train and test data are 5 years each. Because I'm using historical data, the 'historical load' inputs will sometimes be from a future year if the time being forecast is early in the period of available data. For example, if the train dataset is from 01-jan-2008 to 01-jan-2013 and a forecast is being performed at 01-june-2010 then the historical loads will be from 2008, 2009, 2011, and 2012.

Now, when this model is used in production I want it to be able to pull the four historical days from as large a pool as possible. So instead of picking historical load from each of the past four years, it will pick the best historical load from each of the past ten or so years, then pick the best four out of those.

  1. Is it valid for the model to use historical data from the training set when running in inference mode?
  2. What about doing the same when evaluating the model on the test dataset?

I can imagine that by using the same inputs in the test and train situations the network may have over-fit those inputs and will produce an output that it has somewhat memorized. But, I'm not quite sure if this is a big issue or if there's anything else I haven't thought of - hence my qustions :)

$\endgroup$
1
$\begingroup$

If your LSTM uses "future" data in training, then this is not necessarily overfitting, but it is a textbook example of "data leakage" in a time series context. This is of course invalid, and if at all possible, you should try to avoid this.

Whether this degrades your LSTM's performance compared to a clean training setup we can't tell. Try both and see whether it does. I would be more concerned that it might make you believe your predictions would be more accurate than they truly will be.

Also, have you included the fact that in production, you will need temperature forecasts? This adds a source of uncertainty and error that will propagate to your power demand predictions. (It's often surprisingly hard and expensive to get historical forecasts - historical actuals are easier to get. Also, you would need to use the correct "vintage" of temperature forecasts, e.g., three-day-ahead temperature forecasts for forecasting load three days ahead, and five days for five days etc.)

$\endgroup$
  • $\begingroup$ I've certainly thought about the fact that that providing future information may be giving the forecaster a "crystal ball" - to rationalise this I have made the assumption that trends and patterns are consistent over the full 10 years of available data. So, learning patterns from the future should not help it to forecast the past. Temp forecasts is likely not an issue as the historical data is quite inaccurate - and you are right past forecasts are very difficult to get! The system is currently running live and performing very well, so that's a positive sign at least. Thanks for your points $\endgroup$ – fishstix44 Jul 26 '18 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.