Problem statement

Consider this hypothetical but hopefully practical example:

  • You have a dataset consisting of home electricity usage for 1,000 homes in a city.
  • For each home, you have a time series measured hourly.
  • Each time series has a different start/stop date and time. Some are as short as 12 hours, some are as long as 336 hours (a full 2 weeks).
  • You are now given a time series for a new that isn't already in the data, and you need to forecast the time series out by several hours. This time series might be as short as 12 hours.
  • Assume, for simplicity, that the data is "clean" -- no missing data and no obvious outliers.

Hopefully you can see that this problem is analogous to many problems that appear in social and natural science.

What kind of model do you build to accomplish this task? I would expect that there is a lot of "common information" shared across all of the time series that can be used to make better forecasts when given a new time series, especially with respect to cyclical trends throughout the day and week.

My experience with time series is limited to 2 undergrad econometrics courses, the contents of FPP2, and a handful of ad-hoc problem solving I've done at work, which has had mixed results.

The question

Is there a recommended best technique for this problem?

I can think of a few possible approaches:

  1. Use some kind of time series decomposition method to learn seasonal/cyclical components from the dataset. Then see if there is a linear trend in the new time series, then apply seasonal component from the training data on top of the linear trend from the new data.
  2. A regression model trained on hour-of-day and day-of-week converted to polar + any other relevant exogenous features that might be available - this could be linear regression, XGBoost, whatever. Facebook's "Prophet" forecasting model seems to be a sophisticated variation on this idea.
  3. Some kind of Bayesian hierarchical time series model to benefit from partial pooling across homes.
  4. Something else entirely.

Related questions on Stackexchange

Note that there are several related questions on this site, but we do not appear to have any authoritative thread on this topic. So this question is not only for my benefit, but hopefully for the benefit of the community.

Related threads that I found:

Related content elsewhere

As per MaximilianAigner in the comments, this statement is similar to the M5 Forecasting Accuracy competition on Kaggle: https://www.kaggle.com/c/m5-forecasting-accuracy

Stephen G in the comments suggested looking at blog posts like this one: https://petolau.github.io/Forecast-electricity-consumption-with-similar-day-approach-in-R/

  • 1
    $\begingroup$ You might find it interesting to look at the M5 forecasting competition on kaggle, which involved a similar kind of dataset. (Or perhaps that’s why you’re here? 😊) $\endgroup$ Aug 11, 2020 at 17:30
  • 1
    $\begingroup$ What is the unit of measurement? What other information do you have access to, and what external information can you use in the new prediction? This would be something like weather or socioeconomic information about each household. If you can only predict on a generic household, you will likely need to aggregate and predict on the average usage. This blog by Peter Laurinec has a lot of examples of time series analysis with electricity usage that you might find useful. $\endgroup$
    – Stephen G
    Aug 11, 2020 at 18:10
  • $\begingroup$ @MaximilianAigner it's not, I've actually had this problem in my own work before. I will definitely take a look. $\endgroup$ Aug 11, 2020 at 19:57
  • 1
    $\begingroup$ Take a look at Taieb, Huser, Hyndman and Genton, Forecasting Uncertainty in Electricity Smart Meter Data by Boosting Additive Quantile Regression, robjhyndman.com/papers/smart-meter-quantiles.pdf Their algorithm won the Kaggle meter competition, I think. $\endgroup$ Aug 12, 2020 at 18:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.