Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 8 down vote favorite
2

From reading posts on this site I know there is an R function auto.arima (in the forecast package). I also know that IrishStat, a member of of this site built the commercial package autobox in the early 1980s. As these two packages exist today and automatically select arima models for given data sets what do they do differently? Will they possibly produce different models for the same data set?

share|improve this question
Thanks for the edit @Wayne. I am not familar with the R forecast package but I am sure that is what I mean to compare with autobox. – Michael Chernick Jul 21 '12 at 15:14
(I just made a second little change of "auto-arima" to "auto.arima".) There may be other auto.arima functions out there in other packages, but there definitely is one in forecast, whose description is: "Returns best ARIMA model according to either AIC, AICc or BIC value. The function conducts a search over possible model within the order constraints provided." – Wayne Jul 21 '12 at 15:35
AUTOBOX treats automatic identification in a holistic way by iterating though the automatic identification by actually estimating and then doing diagnostic step-up and step-down procedures to render a model that only has statistically significant parameters while having an error process that is free of identifiable structure.In this way it follows the script of iteration. Early versions of AUTOBOX circa 1975 tried to use the "one statistic approach" but this was found wanting as identified models either had redundant or silly structure (5,1,2 for example ) or evidented insufficent structure. – IrishStat Jul 23 '12 at 10:33
1  
@IrishStat That sounds like a good approach. What do you do if you find two competing models that meef your requirements. It seems possbile. Do you recommend an "optimal" model based on specific criteria? I realize the picking of a model with only "statistically significant parameters" may tend to favor parsimony But isn't possible to have a low paramter AR process and another low order AEMA model where all the parameters are statistically significant and the residuals look like white noise? – Michael Chernick Jul 23 '12 at 10:58
1  
@IriehStat. I agree with you. In the end what do you do for the user. Do you provide just one model or might you give an ordered list of competing acceptable models? If not the latter maybe that would be a good option to add where you limit the list to a small number. – Michael Chernick Jul 23 '12 at 11:40
show 3 more comments

3 Answers

up vote 3 down vote accepted

michael/wayne AUTOBOX would definitely deliver/identify a different model if one or more of the following conditions is met 1) there are pulses in the data 2) there is 1 or more level/step shift in the data 3) if there are seasonal pulses in the data 4) there are 1 or more local time trends in the data that are not simply remedied, 5) if the parameters of the model change over time 6) if the variance of the errors change over time and no power transformation is adequate. In terms of a specific example, I would suggest that both of you select/make a time series and post both of them to the web. I will use AUTOBOX to analyse the data in an unattended mode and I will post the models to the list. You then run the R program and then each of you make a separate objective analysis of both results, pointing out similarities and differences. Send those two models complete with all available supporting material including the final error terms to me for my comments. Summarize and presents these results to the list and then ask readers of the list to VOTE for which procedure seems best to them.

share|improve this answer
Do you mean a contest like this one? – whuber♦ Jul 23 '12 at 13:01
@whuber Yes. Perhaps even using some "unknown/coded text book example" which could be used as a backdrop. – IrishStat Jul 23 '12 at 14:00
up vote 8 down vote

They represent two different approaches to two similar but different problems. I wrote auto.arima and @IrishStat is the author of Autobox.

auto.arima() fits (seasonal) ARIMA models including drift terms. Autobox fits transfer function models to handle level shifts and outliers. An ARIMA model is a special case of a transfer function model.

Even if you turned off the level shifts and outlier detection in Autobox, you would get a different ARIMA model from auto.arima() due to different choices in how to identify the ARIMA parameters.

In my testing on the M3 and M-competition data, auto.arima() produces more accurate forecasts than Autobox for these data. However, Autobox will do better with data containing major outliers and level shifts.

share|improve this answer
I believe that you were referring to a version of AUTOBOX from many, many years ago. AUTOBOX has changed signiifcantly oh these many years. If I am not wrong you only compared accuracies from 1 origin which I am sure you will agree is a sample of 1. Accuracies need to be evaluated from a number of origins. – IrishStat Jul 22 '12 at 0:58
3  
I am referring to published comparisons across thousands of series. As Editor-in-Chief of the International Journal of Forecasting, I think that I have some idea about how to evaluate forecasts. – Rob Hyndman Jul 22 '12 at 3:41
1  
I didn't intend for this question to bring out arguments for about who has the best forecasting algorithm. I think both autobox and auto.arima are probably very good packages. A head to head comparison might not be fair for many reasons. 1) The user may not be expert enough to know how to judge them. 2) Forecast accuracy on a single time series is a crap shot. One might have a lower mean square error in prediction, but whenever randomness is involved it must be taken into account. You need to look at several series and as IrishStat suggests you should look at different starting points. – Michael Chernick Jul 22 '12 at 16:31
Also different points to initiate forecasting would be useful. 3) In the ARIMA world there are multiple representations for the same time series model, finite AR processes have infinite moving average representations and vice versa. So a low order AR could be nearly the same as a high order moving average or an ARMA. Box always suggested following the principle of parsimony. But if you have a lot of data you can get good estimates of the parameters and the high order model may generate nearlt the same forecasts as the parsimonious one. 4) The two packages have different objectives. – Michael Chernick Jul 22 '12 at 16:38
If I understood Rob correctly auto.arima lets the user select from a series of criteria including the well known information criteria like AIC and BIC. I think autobox uses one criterion. There may be options for the user but I don't think the user specifies the selection method. If I am worng on this we have the developers here to correct me. – Michael Chernick Jul 22 '12 at 16:42
show 2 more comments
up vote 7 down vote

EDIT: Per your comment, I believe that if you turn off many of autobox's options, you'd probably get a similar answer to auto.arima. But if you do not, and in the presence of outliers there will definitely be a difference: auto.arima doesn't care about outliers, while autobox will detect them and handle them appropriately, which would give a better model. There may be other differences as well, and I"m sure IrishStat can describe those.

I believe autobox detects outliers and other things beyond just searching for the best AR, I, and MA coefficients. If that's correct, it would require more analysis and a couple of other R functions to have similar functionality. And IrishStats is a valuable member of this community, and quite friendly.

Of course, R is free and can do a bazillion things beyond ARIMA.

Another choice that's free for economics-style ARIMA is X13-ARIMA SEATS, from the US Census Bureau, which is open source. There are binaries for Windows and Linux, but it compiled straightforwardly on my Mac, given that I'd already loaded gnu's gfortran compiler. It's the successor to X12-ARIMA, and was just released in the last few days, after years of development and testing. (It updates X12 and also adds in SEATS/TRAMO features. X12 is the official US tool, while SEATS/TRAMO is from the Bank of Spain and is the "European tool".)

I really like X12 (and now X13) a lot. If you output a fair amount of diagnostics and read through them and learn what they mean, they are actually a fairly good education in ARIMA and time series. I've developed my own workflow, but there's an R package x12 for doing most work from within R (you still have to create the input model (".spc") file for X12).

I say X12 is good at "economics style" ARIMA to mean monthly data with more than 3 years of data. (You need 5+ years of data to use some diagnostic features.) It has an outlier identification feature, can handle all kinds of outlier specifications, and can handle holidays, floating holidays, trading day effects, and a host of economic things. It's the tool that the US government uses to create seasonally-adjusted data.

share|improve this answer
My question was really given a data set will the two algorithms possibly produce different model selections. It is really the automatic slection that I am interested in and not qny other diagnostic features that one may hve that the other does not. It is known that the family of ARMA models and two models in the family can be exact or nearly exact alternative representations of the same model. So if there are minor differences in the selection proceudres I would think they could give different model choices. – Michael Chernick Jul 21 '12 at 15:29
3  
@MichaelChernick: Ah. My guess would be that if you turn off all of the auto-stuff in autobox you'd get the same answer. But one of the points of using autobox is that it will detect outliers and handle them as such, so the model returned would be different if there are outliers. – Wayne Jul 21 '12 at 15:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.