Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I need to automate time-series forecasting, and I don't know in advance the features of those series (seasonality, trend, noise, etc).

My aim is not to get the best possible model for each series, but to avoid pretty bad models. In other words, to get small errors every time is not a problem, but to get big errors once in a while is.

I thought I could achieve this by combining models calculated with different techniques.

That is, although ARIMA would be the best approach for a specific series, it may not be the best for another series; the same for exponential smoothing.

However, if I combine one model from each technique, even if one model isn't so good, the other will bring the estimate closer to the real value.

It is well-known that ARIMA works better for long-term well-behaved series, while exponential smoothing stands out with short-term noisy series.

  • My idea is to combine models generated from both techniques in order to get more robust forecasts, does it make sense?

There might be many ways to combine those models.

  • If this is a good approach, how should I combine them?

A simple mean of forecasts is an option, but maybe I could get better predictions if I weight the mean according to some goodness measure of the model.

  • What would be the treatment of the variance when combining models?
share|improve this question
    
Your ideas sound great, but I'm not so sure about using automatically fit ARIMA models. For univariate series perhaps ... Conventional wisdom is that Holt-Winters is pretty robust used automatically, so that could be your baseline for out-of-sample comparisons between methods. –  Scortchi Jan 17 '13 at 15:36
    
@Scortchi I forgot to mention that all series are univariate! ;) I agree that Holt-Winters performs really good when used automatically, but I intend to get one more opinion from another model, to avoid cases where forecasts aren't so good. Sometimes HW shows strange trend behaviour. –  João Daniel Jan 17 '13 at 17:35
1  
Even in the univariate case I struggle to imagine an automatic procedure - there's trend (stochastic or deterministic), possible transformations, seasonality (multiplicative or additive) to think about, & I find that to get to a model I use a lot of prior knowledge about what would be sensible for what a particular series represents in reality. Still, the proof of the pudding is in the eating - I really just wanted to say not to forget to do out-of-sample comparisons with simple techniques- so good luck with it. –  Scortchi Jan 17 '13 at 18:16
add comment

3 Answers

up vote 5 down vote accepted

Combining forecasts is an excellent idea. (I think it is not an exaggeration to say that this is one of the few things academic forecasters agree on.)

I happen to have written a paper a while back looking at different ways to weight forecasts in combining them: http://www.sciencedirect.com/science/article/pii/S0169207010001032 Basically, using (Akaike) weights did not consistently improve combinations over simple or trimmed/winsorized means or medians, so I personally would think twice before implementing a complex procedure that may not yield a definite benefit (recall, though, that combinations consistently outperformed selection single methods by information criteria). This may depend on the particular time series you have, of course.

I looked at combining prediction intervals in the paper above, but not at combining variance as such. I seem to recall a paper not long back in the IJF with this focus, so you may want to search for "combining" or "combination" through back issues of the IJF.

A few other papers that have looked at combining forecasts are here (from 1989, but a review) and here and here (also looks at densities) and here and here. Many of these note that it is still poorly understood why forecast combinations frequently outperform single selected models. The second-to-last paper is on the M3 forecasting competition; one of their main findings was (number (3) on p. 458) that "The accuracy of the combination of various methods outperforms, on average, the specific methods being combined and does well in comparison with other methods." The last of these papers finds that combinations do not necessarily perform better than single models, but that they can considerably reduce the risk of catastrophic failure (which is one of your goals). More literature should readily be found in the International Journal of Forecasting, the Journal of Forecasting and for more specific applications in the econometrics or supply chain literature.

share|improve this answer
    
Great point of view about combining models! Your answer was very constructive! –  João Daniel Jan 18 '13 at 19:18
add comment

Why not specify it further? I don't think that any one model you would produce could be better or good enough than a specific choice.

With that said, if you can narrow down your choices a bit to those you can test for, and the data input can be standardized, then why not write an automated testing procedure in R?

Say you decide your data will fall within a range to be estimated by five models as well as one "fallback". Say you can characterize the input by different tests. Then just go ahead and write an R (or a program like that) algorithm that runs this for you. This works if you could produce a flowchart of which model to run based on test data, that is if any point of the decision tree is binary.

If this is not an option because the decision may not be binary, I suggest you implement a rating system based on applicable tests and run some "extreme cases" simulated data through your grid to see if the results are what you are looking for.

You can combine these things obviously, for example testing for non-stationarity may be give a definitve yes-no, while other attributes may fall into a range such as multicollinearity.
You can draw this out on paper first, then build it, simulate it with known distributions you expect to have.

Then just run the R program everytime new data arrives. I see no need to combine several models with the computational capabilities you most likely have at hand.

share|improve this answer
    
Narrowing the choices down is a good idea, like not using non-seasonal methods if the data are obviously seasonal. But even then, I would argue that averaging multiple seasonal models (additive vs. multiplicative seasonality, with or without trend etc.) will on average improve forecast accuracy. At least that is the impression I get from quite a bit of exposure to the forecasting community as well as to the M3 and similar forecasting competitions. –  Stephan Kolassa Jan 17 '13 at 15:59
    
Do you have additional papers on this? I mean this would be a straight-forward yet relevant research subject. Very interesting idea, though just intuitively I don't agree that it necessarily would be better than a dynamic grid of models. –  IMA Jan 18 '13 at 8:02
    
Good point. I edited my answer to include an additional paragraph with more literature pointers. I agree that this is straightforward and relevant, and it is still poorly understood why forecast averaging usually improves accuracy. –  Stephan Kolassa Jan 18 '13 at 8:19
    
Yeah I mean you could model all sorts of distributional problems and attack it computationally and fundamentally. Thanks for the papers, very interesting. –  IMA Jan 18 '13 at 8:39
add comment

There is a nice and simple formulae for combining two forecasting methods, you just weight them multipling the first by a and the other by (1 - a), where a is found by minimizing the variance of this combined forecast. As you know the errors of both forecasting methods, you can calculate the errors of the combination wich will depend on "a". The calculation is simple when the mean of each method is = 0. For combining more than 2 methods the formulae is still "simple" in the sense that you can calculate it analytically "by hand", or also use the Solver option from EXCEL

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.