14
$\begingroup$

Suppose i have a little over 20.000 monthly time series spanning from Jan'05 to Dec'11. Each of these representing global sales data for a different product. What if, instead of computing forecasts for each and every one of them, I wanted to focus only on a small number of products that "actually matter"?

I could rank those products by total annual revenue and trim down the list using classical Pareto. Still it seems to me that, although they do not contribute much to bottom line, some products are so easy to forecast that leaving them out would be bad judjement. A product that sold 50$ worth each month for the past 10 years might not sound like much, but it requires so little effort to generate predictions about future sales that I might as well do it.

So let's say I divide my products in four categories: high revenue/easy to forecast - low revenue/easy to forecast - high revenue/hard to forecast - low revenue/hard to forecast.

I think it would be reasonable to leave behind only those time series belonging to the fourth group. But how exactly can I evaluate "forecastability"?

Coefficient of variation seems like a good starting point (I also remember seeing some paper about it a while ago). But what if my time series exhibit seasonality/level shifts/calendar effects/strong trends?

I would imagine I should base my evaluation only on variability of the random component and not the one of the "raw" data. Or am I missing something?

Has anybody stumbled upon a similar problem before? How would you guys go about it?

As always, any help is greatly appreciated!

Improve this question
$\endgroup$

4 Answers 4

Reset to default
13
$\begingroup$

Here's a second idea based on stl.

You could fit an stl decomposition to each series, and then compare the standard error of the remainder component to the mean of the original data ignoring any partial years. Series that are easy to forecast should have a small ratio of se(remainder) to mean(data).

The reason I suggest ignoring partial years is that seasonality will affect the mean of the data otherwise. In the example in the question, all series have seven complete years, so it is not an issue. But if the series extended part way into 2012, I suggest the mean is computed only up to the end of 2011 to avoid seasonal contamination of the mean.

This idea assumes that mean(data) makes sense -- that is that the data are mean stationary (apart from seasonality). It probably wouldn't work well for data with strong trends or unit roots.

It also assumes that a good stl fit translates into good forecasts, but I can't think of an example where that wouldn't be true so it is probably an ok assumption.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Rob, thanks for getting back to me. I like you idea so I'll give it a try and see if it provides the desired level of filtering. Just one more thing, is there any particular reason for using mean(data) over mean(remainder)? I'm afraid some of my time series might have a somewhat strong trend. STL decomposed series, instead, should not. Also do you think that the approach we outlined so far for assessing forecastability/spotting outliers is good enough to be implemented in a real business enviroment? Or is it too "amateurial"? Would you normally do thing much differently? $\endgroup$
    – Bruder
    Commented Feb 20, 2012 at 13:51
  • $\begingroup$ mean(remainder) will be close to zero. You want to compare noise to scale of the data, so mean(data) should be ok. Not sure how to deal with your trends. I would test the approach carefully on a range of data before believing the results. $\endgroup$
    – Rob Hyndman
    Commented Feb 20, 2012 at 21:09
9
$\begingroup$

You might be interested in ForeCA: Forecastable Component Analysis (disclaimer: I am the author). As the name suggests it is a dimension reduction / blind source separation (BSS) technique to find most forecastable signals from many multivariate - more or less stationary - time series. For your particular case of 20,000 time series it might not be the fastest thing to do (the solution involves multivariate power spectra and iterative, analytic updating of the best weightvector; furthermore I guess it might run into the $p \gg n$ problem.)

There is also an R package ForeCA available at CRAN (again: I am the author) which implements basic functionality; right now it supports the functionality to estimate forecastability measure $\Omega(x_t)$ for univariate time series and it has some good wrapper functions for multivariate spectra (again 20,000 time series is probably too much to handle at once).

But maybe you can try to use the MASE measure proposed by Rob to make a coarse grid separation of the 20,000 in several sub-groups and then apply ForeCA to each separately.

Improve this answer
$\endgroup$
9
$\begingroup$

This is a fairly common problem in forecasting. The traditional solution is to compute mean absolute percentage errors (MAPEs) on each item. The lower the MAPE, the more easily forecasted is the item.

One problem with that is many series contain zero values and then MAPE is undefined.

I proposed a solution in Hyndman and Koehler (IJF 2006) [Preprint version] using mean absolute scaled errors (MASEs). For monthly time series, the scaling would be based on in-sample seasonal naive forecasts. That is if $y_t$ is an observation at time $t$, data are available from times 1 to $T$ and $$ Q = \frac{1}{T-12}\sum_{t=13}^T |y_t-y_{t-12}|, $$ then a scaled error is $q_t = (y_t-\hat{y}_t)/Q$, where $\hat{y}_t$ is a forecast of $y_t$ using whatever forecasting method you are implementing for that item. Take the mean absolute value of the scaled errors to get the MASE. For example, you might use a rolling origin (aka time series cross-validation) and take the mean absolute value of the resulting one-step (or $h$-step) errors.

Series that are easy to forecast should have low values of MASE. Here "easy to forecast" is interpreted relative to the seasonal naive forecast. In some circumstances, it may make more sense to use an alternative base measure to scale the results.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Hi Rob, thank you for your kind reply. As always your approach is very neat, straightforward and reasonable. I already evaluate forecast value added (FTV) against a seasonal naive model so your idea of assessing forecastability using the same "base measure" sounds very appealing. The only problem is that, in order to calculate MASE, I need to choose a forecasting method and run simulations for each of my 20000 time-series. I was hoping I could spot easy-to-forecast series beforehand, so that I can save computational time. $\endgroup$
    – Bruder
    Commented Feb 17, 2012 at 13:31
  • $\begingroup$ For some reason I thought that time-series with a lower relative variability (i.e. CV) would necessarily result in easier & more accurate forecasts. Calculating forecasts and then, and only then measuring errors, kind of defies, I think, my purpouse. I guess what I'm trying to say is that I look at MASE more like a measure of forecast accuracy than a measure of forecastability. But I might be wrong... :) $\endgroup$
    – Bruder
    Commented Feb 17, 2012 at 13:37
  • 1
    $\begingroup$ @Bruder: 2 thoughts: 1. You could look at a simple naive forecast, rather than a seasonal forecast. A simple naive forecast just uses the previous value of the time series, and will pick up a strong trend (with a 1-period lag). 2. STL decomposition is a good idea. If the residuals are very small compared to the seasonal and trend components, then you can probably easily forecast the series. $\endgroup$
    – Zach
    Commented Feb 17, 2012 at 16:43
  • 1
    $\begingroup$ @Rob - what about STL decomposition? Can I get two birds with one stone (i.e. spotting outliers and assessing forecastability, therefore assessing "true" forecastability)? It amazes me how many thing I can accomplish with just STL and a seasonal naive model. But you know what happens when things are too good to be true... $\endgroup$
    – Bruder
    Commented Feb 18, 2012 at 10:31
  • 1
    $\begingroup$ I believe that the index of the sum should be $t$ instead of $i$. $\endgroup$
    – blakeoft
    Commented Nov 4, 2019 at 19:54
1
$\begingroup$

This answer is very late, but for those who are still looking for an appropriate measure of forecastability for product demand time series, I highly suggest looking at approximate entropy.

The presence of repetitive patterns of fluctuation in a time series renders it more predictable than a time series in which such patterns are absent. ApEn reflects the likelihood that similar patterns of observations will not be followed by additional similar observations.[7] A time series containing many repetitive patterns has a relatively small ApEn; a less predictable process has a higher ApEn.

Product demand tend to have a very strong seasonal component, making the coefficient of variation (CV) inappropriate. ApEn(m, r) is able to correctly handle this. In my case, since my data tends to have a strong weekly seasonality, I set parameters m=7 and r=0.2*std as recommended here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ In the wikipedia article, what does $u^*$ mean? $\endgroup$
    – blakeoft
    Commented Nov 4, 2019 at 20:38
  • $\begingroup$ I see now. I thought $u$ and $u^*$ were related, but they are in fact not. $\endgroup$
    – blakeoft
    Commented Nov 4, 2019 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.