8
$\begingroup$

Background

Often I am forecasting possibly one up to a few dozen variables in a project, but I have an upcoming project that will involve forecasting thousands of variables. I have some ideas of my own, but as always it is wise to try to get some guidance when you're trying something new.

Question

I would like some reference(s) or guidance on how to perform forecasting on a large number of variables. My main concerns are about ensuring quality of the forecasts.

Some Thoughts So Far

  • Some of the variables are related to each other in a hierarchy, so consider taking advantage of partial pooling by having hierarchical models over those sets of variables should improve statistical precision.
  • The problem is a conventional statistical forecast. No mention of making interventions.
  • Many steps (e.g. train/test split, training, evaluation, prediction etc) will be the same as when I do them on a small scale and are quite automatable.
  • Main challenges I see are around things that I would normally do manually, such as visually inspecting EDA and diagnostic plots or summaries. A lot of value can be obtained from that moment, "hmm, that looks weird".
  • Perhaps some data mining or clustering could come in handy. E.g. clustering time series and then focusing on viewing samples from the clusters along with paying attention to time series that do not nicely fit into clusters might help me triage where to put my time/attention/energy.
$\endgroup$

1 Answer 1

9
$\begingroup$

A few thoughts:

If your data is partially hierarchical, then do consider hierarchical forecasting with the optimal reconciliation approach. This very often improves forecasts on all levels. The vanilla version only works for expectation forecasts (there are some recent papers on probabilistic hierarchical forecasting, though), and it can't very well handle larger or more complex hierarchies, but it is absolutely a thing to look at.

I absolutely agree with paneling your data to pool information for related series.

Both these points depend very much on what kind of hierarchy/hierarchies you have. For instance, you might have sales of different products in different locations, and that gives you two separate hierarchical or panel dimensions, both of which you can and should leverage.

As you write, plots are often very helpful. Plot aggregates along your hierarchies, per above. Or plot random series (or aggregates). I will often just take 200 series, which is about the limit of what I can look at, then get a cup of coffee and flip book through the plots. You can quickly identify "strange" behavior in single series or aggregates that way, stuff you didn't even think about looking for, and then investigate more deeply and/or treat issues automatically.

What is helpful here will very much depend on your dimensionality. If you have 200 products sold in 20 stores each, then you can look at 200 plots of product sales that each summarize the 20 stores in some smart way. For instance, if you have fine granular data, like daily, you might want to not plot total sales, but just have 20 lines of dots corresponding to the 20 stores, with a dot on a day on which this product was sold at this particular store. That once helped me detect markdown sales that were not noted in the data.

Alternatively, you might be able to quantify issues during data cleansing, and explicitly look at the "dirtier" time series. For instance, if you detect outliers, you could label each series $(y_t)$ by $\max\{|y_t|\}/\text{mean}\{|y_t|\}$, look at the top 20 series and think about whether you can do something about these outliers. (Alternatively, model the series and look at residuals rather than series values.)

You can also do seasonal plots by plotting beanplots, or boxplots, or raw data, together with some summary statistics.

I personally rarely use clustering, to be honest. The feature space is just too large, or put differently, there are just so many things that can go wrong in time series... I find that "flip book" approach above more useful. And when it comes to focusing attention, I usually try to get a handle on the business importance of the different series, and look at the ones that are more important.

Here is one thing I always do: when evaluating my forecasts on a holdout set, don't just look at average accuracy. In addition, look at the very bad forecasts. These are (a) the ones that erode your user's trust in your forecasts if they see them, and (b) the ones where you can learn a lot about problems in your data. For instance, you might have a very bad forecast because a series is zero in the complete evaluation period, so you need to figure out whether this series just stopped at some point, and if so, whether you can get this information in some way, or need to forecast it. Of course, this very much depends on getting a scale invariant notion of badness or forecastability, which is not easy... scaled error measures help here, but are not a panacea.

$\endgroup$
2
  • $\begingroup$ For the flip book approach, do you just open one of the series which has a key to easy toggle to the next image? I am amused by the thought of someone printing an actual flip book of plots! 😂 $\endgroup$
    – Galen
    Commented Feb 8 at 15:33
  • 1
    $\begingroup$ I just plot everything into a separate directory, then use any image viewer that lets me move to the next picture with the arrow key. In German, flip books are called Daumenkino, "thumb cinema", which makes even more sense here, except that I use my index finger... I don't quite yet print everything on paper to physically flip through... $\endgroup$ Commented Feb 8 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.