Large dataset of many short time series: what model to use for forecasting a new time series not in the data?

Question

Problem statement

Consider this hypothetical but hopefully practical example:

• You have a dataset consisting of home electricity usage for 1,000 homes in a city.
• For each home, you have a time series measured hourly.
• Each time series has a different start/stop date and time. Some are as short as 12 hours, some are as long as 336 hours (a full 2 weeks).
• You are now given a time series for a new that isn't already in the data, and you need to forecast the time series out by several hours. This time series might be as short as 12 hours.
• Assume, for simplicity, that the data is "clean" -- no missing data and no obvious outliers.

Hopefully you can see that this problem is analogous to many problems that appear in social and natural science.

What kind of model do you build to accomplish this task? I would expect that there is a lot of "common information" shared across all of the time series that can be used to make better forecasts when given a new time series, especially with respect to cyclical trends throughout the day and week.

My experience with time series is limited to 2 undergrad econometrics courses, the contents of FPP2, and a handful of ad-hoc problem solving I've done at work, which has had mixed results.

The question

Is there a recommended best technique for this problem?

I can think of a few possible approaches:

1. Use some kind of time series decomposition method to learn seasonal/cyclical components from the dataset. Then see if there is a linear trend in the new time series, then apply seasonal component from the training data on top of the linear trend from the new data.
2. A regression model trained on hour-of-day and day-of-week converted to polar + any other relevant exogenous features that might be available - this could be linear regression, XGBoost, whatever. Facebook's "Prophet" forecasting model seems to be a sophisticated variation on this idea.
3. Some kind of Bayesian hierarchical time series model to benefit from partial pooling across homes.
4. Something else entirely.

Related questions on Stackexchange

Note that there are several related questions on this site, but we do not appear to have any authoritative thread on this topic. So this question is not only for my benefit, but hopefully for the benefit of the community.

Related threads that I found:

• https://datascience.stackexchange.com/q/80127/1156
  • Brand new and unanswered.
  • I might have spoken with this person on Discord in the past, and suggested that they ask their question here.
• https://datascience.stackexchange.com/q/51939/1156
  • Some helpful ideas, not necessarily RNN-specific, but not very comprehensive
• https://datascience.stackexchange.com/q/34200/1156
  • Some helpful ideas, but not very comprehensive
  • Maybe not related?
• Incorrect Approach to Multiple Time Series Forecasting and Is it possible to automate time series forecasting? and prediction on short time series with seasonality and data correlations and https://stats.stackexchange.com/search?q=peeling+an+onion
  • Community member IrishStat has posted a lot of helpful information about automated time series modeling workflows. However I'd be concerned about the fact that this "auto-ARIMAX" technique would fall over on the very short time series, and doesn't do any kind of pooling across series.
• How to test superior predictive ability over multiple time series?
  • Kind of related?
• Fitting a single time series forecasting model for multiple series
  • Possibly related, maybe just an abandoned question about vector autoregression.

Related content elsewhere

As per MaximilianAigner in the comments, this statement is similar to the M5 Forecasting Accuracy competition on Kaggle: https://www.kaggle.com/c/m5-forecasting-accuracy

Stephen G in the comments suggested looking at blog posts like this one: https://petolau.github.io/Forecast-electricity-consumption-with-similar-day-approach-in-R/