Assessing forecastability of time series

Suppose i have a little over 20.000 monthly time series spanning from Jan'05 to Dec'11. Each of these representing global sales data for a different product. What if, instead of computing forecasts for each and every one of them, I wanted to focus only on a small number of products that "actually matter"?

I could rank those products by total annual revenue and trim down the list using classical Pareto. Still it seems to me that, although they do not contribute much to bottom line, some products are so easy to forecast that leaving them out would be bad judjement. A product that sold 50$worth each month for the past 10 years might not sound like much, but it requires so little effort to generate predictions about future sales that I might as well do it. So let's say I divide my products in four categories: high revenue/easy to forecast - low revenue/easy to forecast - high revenue/hard to forecast - low revenue/hard to forecast. I think it would be reasonable to leave behind only those time series belonging to the fourth group. But how exactly can I evaluate "forecastability"? Coefficient of variation seems like a good starting point (I also remember seeing some paper about it a while ago). But what if my time series exhibit seasonality/level shifts/calendar effects/strong trends? I would imagine I should base my evaluation only on variability of the random component and not the one of the "raw" data. Or am I missing something? Has anybody stumbled upon a similar problem before? How would you guys go about it? As always, any help is greatly appreciated! - 3 Answers Here's a second idea based on stl. You could fit an stl decomposition to each series, and then compare the standard error of the remainder component to the mean of the original data ignoring any partial years. Series that are easy to forecast should have a small ratio of se(remainder) to mean(data). The reason I suggest ignoring partial years is that seasonality will affect the mean of the data otherwise. In the example in the question, all series have seven complete years, so it is not an issue. But if the series extended part way into 2012, I suggest the mean is computed only up to the end of 2011 to avoid seasonal contamination of the mean. This idea assumes that mean(data) makes sense -- that is that the data are mean stationary (apart from seasonality). It probably wouldn't work well for data with strong trends or unit roots. It also assumes that a good stl fit translates into good forecasts, but I can't think of an example where that wouldn't be true so it is probably an ok assumption. -  Hi Rob, thanks for getting back to me. I like you idea so I'll give it a try and see if it provides the desired level of filtering. Just one more thing, is there any particular reason for using mean(data) over mean(remainder)? I'm afraid some of my time series might have a somewhat strong trend. STL decomposed series, instead, should not. Also do you think that the approach we outlined so far for assessing forecastability/spotting outliers is good enough to be implemented in a real business enviroment? Or is it too "amateurial"? Would you normally do thing much differently? – Bruder Feb 20 '12 at 13:51 mean(remainder) will be close to zero. You want to compare noise to scale of the data, so mean(data) should be ok. Not sure how to deal with your trends. I would test the approach carefully on a range of data before believing the results. – Rob Hyndman Feb 20 '12 at 21:09 This is a fairly common problem in forecasting. The traditional solution is to compute mean absolute percentage errors (MAPEs) on each item. The lower the MAPE, the more easily forecasted is the item. One problem with that is many series contain zero values and then MAPE is undefined. I proposed a solution in Hyndman and Koehler (IJF 2006) [Preprint version] using mean absolute scaled errors (MASEs). For monthly time series, the scaling would be based on in-sample seasonal naive forecasts. That is if$y_t$is an observation at time$t$, data are available from times 1 to$T$and $$Q = \frac{1}{T-12}\sum_{i=13}^T |y_t-y_{t-12}|,$$ then a scaled error is$q_t = (y_t-\hat{y}_t)/Q$, where$\hat{y}_t$is a forecast of$y_t$using whatever forecasting method you are implementing for that item. Take the mean absolute value of the scaled errors to get the MASE. For example, you might use a rolling origin (aka time series cross-validation) and take the mean absolute value of the resulting one-step (or$h$-step) errors. Series that are easy to forecast should have low values of MASE. Here "easy to forecast" is interpreted relative to the seasonal naive forecast. In some circumstances, it may make more sense to use an alternative base measure to scale the results. - Hi Rob, thank you for your kind reply. As always your approach is very neat, straightforward and reasonable. I already evaluate forecast value added (FTV) against a seasonal naive model so your idea of assessing forecastability using the same "base measure" sounds very appealing. The only problem is that, in order to calculate MASE, I need to choose a forecasting method and run simulations for each of my 20000 time-series. I was hoping I could spot easy-to-forecast series beforehand, so that I can save computational time. – Bruder Feb 17 '12 at 13:31 For some reason I thought that time-series with a lower relative variability (i.e. CV) would necessarily result in easier & more accurate forecasts. Calculating forecasts and then, and only then measuring errors, kind of defies, I think, my purpouse. I guess what I'm trying to say is that I look at MASE more like a measure of forecast accuracy than a measure of forecastability. But I might be wrong... :) – Bruder Feb 17 '12 at 13:37 @Bruder You could just calculate Q, which should be relatively straightforward. Time series with lower Q's should be easier to forecast. You could also look at the error for a non-seasonal naive model, which will also be relatively easy to compute (compare each observation to the one immediately prior.) – Zach Feb 17 '12 at 14:05 @Zach - Thank you for your reply. Yours is not a bad idea, but I fear it might be biased by factors such as a strong trend. Two years might differ a lot (high Q) if there is a large increase in sales from one period to the other, altough the pattern of demand (the relative increase from one month to the other) actually remains the same (I hope you can see what I mean). On the other hand, a time-serie with a flat trend but a more random pattern might appear easier to forecast (if the magnitude of the randomness is small compared to overall level). – Bruder Feb 17 '12 at 16:08 @Bruder: 2 thoughts: 1. You could look at a simple naive forecast, rather than a seasonal forecast. A simple naive forecast just uses the previous value of the time series, and will pick up a strong trend (with a 1-period lag). 2. STL decomposition is a good idea. If the residuals are very small compared to the seasonal and trend components, then you can probably easily forecast the series. – Zach Feb 17 '12 at 16:43 show 5 more comments You might be interested in ForeCA: Forecastable Component Analysis (disclaimer: I am the author). As the name suggests it is a dimension reduction / blind source separation (BSS) technique to find most forecastable signals from many multivariate - more or less stationary - time series. For your particular case of 20,000 time series it might not be the fastest thing to do (the solution involves multivariate power spectra and iterative, analytic updating of the best weightvector; furthermore I guess it might run into the$p \gg n$problem.) There is also an R package ForeCA available at CRAN (again: I am the author) which implements basic functionality; right now it supports the functionality to estimate forecastability measure$\Omega(x_t)\$ for univariate time series and it has some good wrapper functions for multivariate spectra (again 20,000 time series is probably too much to handle at once).

But maybe you can try to use the MASE measure proposed by Rob to make a coarse grid separation of the 20,000 in several sub-groups and then apply ForeCA to each separately.

-