From reading posts on this site I know there is an R function auto.arima (in the forecast package). I also know that IrishStat, a member of of this site built the commercial package autobox in the early 1980s. As these two packages exist today and automatically select arima models for given data sets what do they do differently? Will they possibly produce different models for the same data set?
$\begingroup$
$\endgroup$
8
asked Jul 21, 2012 at 15:03
3 Answers
$\begingroup$
$\endgroup$
2
michael/wayne
AUTOBOX would definitely deliver/identify a different model if one or more of the following conditions is met
1) there are pulses in the data
2) there is 1 or more level/step shift in the data
3) if there are seasonal pulses in the data
4) there are 1 or more local time trends in the data that are not simply remedied
5) if the parameters of the model change over time
6) if the variance of the errors change over time and no power transformation is adequate.
In terms of a specific example, I would suggest that both of you select/make a time series and post both of them to the web. I will use AUTOBOX to analyse the data in an unattended mode and I will post the models to the list. You then run the R program and then each of you make a separate objective analysis of both results, pointing out similarities and differences. Send those two models complete with all available supporting material including the final error terms to me for my comments. Summarize and presents these results to the list and then ask readers of the list to VOTE for which procedure seems best to them.
answered Jul 21, 2012 at 23:59
-
-
$\begingroup$ @whuber Yes. Perhaps even using some "unknown/coded text book example" which could be used as a backdrop. $\endgroup$ Jul 23, 2012 at 14:00
$\begingroup$
$\endgroup$
7
They represent two different approaches to two similar but different problems. I wrote auto.arima and @IrishStat is the author of Autobox.
auto.arima() fits (seasonal) ARIMA models including drift terms. Autobox fits transfer function models to handle level shifts and outliers. An ARIMA model is a special case of a transfer function model.
Even if you turned off the level shifts and outlier detection in Autobox, you would get a different ARIMA model from auto.arima() due to different choices in how to identify the ARIMA parameters.
In my testing on the M3 and M-competition data, auto.arima() produces more accurate forecasts than Autobox for these data. However, Autobox will do better with data containing major outliers and level shifts.
answered Jul 22, 2012 at 0:01
-
3$\begingroup$ I believe that you were referring to a version of AUTOBOX from many, many years ago. AUTOBOX has changed signiifcantly oh these many years. If I am not wrong you only compared accuracies from 1 origin which I am sure you will agree is a sample of 1. Accuracies need to be evaluated from a number of origins. $\endgroup$ Jul 22, 2012 at 0:58
-
18$\begingroup$ I am referring to published comparisons across thousands of series. As Editor-in-Chief of the International Journal of Forecasting, I think that I have some idea about how to evaluate forecasts. $\endgroup$ Jul 22, 2012 at 3:41
-
2$\begingroup$ I didn't intend for this question to bring out arguments for about who has the best forecasting algorithm. I think both autobox and auto.arima are probably very good packages. A head to head comparison might not be fair for many reasons. 1) The user may not be expert enough to know how to judge them. 2) Forecast accuracy on a single time series is a crap shot. One might have a lower mean square error in prediction, but whenever randomness is involved it must be taken into account. You need to look at several series and as IrishStat suggests you should look at different starting points. $\endgroup$ Jul 22, 2012 at 16:31
-
$\begingroup$ Also different points to initiate forecasting would be useful. 3) In the ARIMA world there are multiple representations for the same time series model, finite AR processes have infinite moving average representations and vice versa. So a low order AR could be nearly the same as a high order moving average or an ARMA. Box always suggested following the principle of parsimony. But if you have a lot of data you can get good estimates of the parameters and the high order model may generate nearlt the same forecasts as the parsimonious one. 4) The two packages have different objectives. $\endgroup$ Jul 22, 2012 at 16:38
-
3$\begingroup$ The method has evolved over time. Dave Reilly is very active on this site as IrishStat and he has been very open about explaining how it works in general terms. It is an essential aspect of business to have trade secrets and proprietary algorithms. From his point of view R is hurting his business just like it is for SPlus. But he does not show bitterness and is very willing to demonstrate his software as you can see he did today. He is also willing to run tests against competitors and I believe he has entered time series forecasting competitions. $\endgroup$ Jul 25, 2012 at 3:52
$\begingroup$
$\endgroup$
4
EDIT: Per your comment, I believe that if you turn off many of autobox's options, you'd probably get a similar answer to auto.arima. But if you do not, and in the presence of outliers there will definitely be a difference: auto.arima doesn't care about outliers, while autobox will detect them and handle them appropriately, which would give a better model. There may be other differences as well, and I"m sure IrishStat can describe those.
I believe autobox detects outliers and other things beyond just searching for the best AR, I, and MA coefficients. If that's correct, it would require more analysis and a couple of other R functions to have similar functionality. And IrishStats is a valuable member of this community, and quite friendly.
Of course, R is free and can do a bazillion things beyond ARIMA.
Another choice that's free for economics-style ARIMA is X13-ARIMA SEATS, from the US Census Bureau, which is open source. There are binaries for Windows and Linux, but it compiled straightforwardly on my Mac, given that I'd already loaded gnu's gfortran compiler. It's the successor to X12-ARIMA, and was just released in the last few days, after years of development and testing. (It updates X12 and also adds in SEATS/TRAMO features. X12 is the official US tool, while SEATS/TRAMO is from the Bank of Spain and is the "European tool".)
I really like X12 (and now X13) a lot. If you output a fair amount of diagnostics and read through them and learn what they mean, they are actually a fairly good education in ARIMA and time series. I've developed my own workflow, but there's an R package x12 for doing most work from within R (you still have to create the input model (".spc") file for X12).
I say X12 is good at "economics style" ARIMA to mean monthly data with more than 3 years of data. (You need 5+ years of data to use some diagnostic features.) It has an outlier identification feature, can handle all kinds of outlier specifications, and can handle holidays, floating holidays, trading day effects, and a host of economic things. It's the tool that the US government uses to create seasonally-adjusted data.
-
$\begingroup$ My question was really given a data set will the two algorithms possibly produce different model selections. It is really the automatic slection that I am interested in and not qny other diagnostic features that one may hve that the other does not. It is known that the family of ARMA models and two models in the family can be exact or nearly exact alternative representations of the same model. So if there are minor differences in the selection proceudres I would think they could give different model choices. $\endgroup$ Jul 21, 2012 at 15:29
-
3$\begingroup$ @MichaelChernick: Ah. My guess would be that if you turn off all of the auto-stuff in
autoboxyou'd get the same answer. But one of the points of usingautoboxis that it will detect outliers and handle them as such, so the model returned would be different if there are outliers. $\endgroup$– WayneJul 21, 2012 at 15:41 -
$\begingroup$ @Wayne +1 for the extra information about X13-ARIMA SEATS and SEATS/TRAMO. $\endgroup$ May 23, 2013 at 22:59
-
$\begingroup$ @Wayne By the way, another "European Tool" is DEMETRA+. $\endgroup$ May 23, 2013 at 23:22
auto.arimafunctions out there in other packages, but there definitely is one inforecast, whose description is: "Returns best ARIMA model according to either AIC, AICc or BIC value. The function conducts a search over possible model within the order constraints provided." $\endgroup$