Selecting ARIMA orders by ACF/PACF vs. by information criteria

Question

We keep on getting questions here about selecting ARIMA model orders based on ACF/PACF plots. This is the older methodology proposed by Box and Jenkins.

More modern tools like the auto.arima() function in the forecast package for R or ARIMA() in the fable package instead use a grid search over different orders and select the one that optimizes an information criterion. I highly respect the creators and maintainers of these packages and consider them among the best informed forecasters in the world.

Per Hyndman & Athanasopoulos (Rob Hyndman is the maintainer of the packages above), ACF/PACF plots can't be used to select orders if both the AR and the MA order are nonzero.

In the comments, Richard Hardy points to Shumway & Stoffer, Time Series Analysis and Its Application with R examples. This is a prime example of the way I often see the Box-Jenkins approach taught in textbooks: the process seems to involve a lot of decisions that are based on expertise (or, uncharitably described, arbitrariness), like what it means for a (P)ACF to decay "quickly" or not, whether variance "is changing" or not, and this seems to be very hard to automate, especially for larger sets of multiple time series. Overall, this advice seems to be most problematic for people who are just starting out with time series analysis, and who therefore IMO would be best served with an automated approach as the one referenced above.

Thus, I am confused why people would still use, teach and propagate the older Box-Jenkins method, rather than use a grid search using information criteria. I suspect this is just due to people perpetuating old and superseded advice.

Question: is there any published research on any benefits of a Box-Jenkins approach (however automated) over optimizing information criteria?

Related: Selecting ARIMA orders based on ACF-PACF vs. auto.arima, and actually almost all threads with the box-jenkins tag.

Answer 1

While there is limited published research directly comparing the Box-Jenkins method based on ACF/PACF plots and information criteria-based methods in ARIMA model selection, most researchers and practitioners have gravitated towards using information criteria methods, such as AIC or BIC, due to their increased efficiency, automation, and out-of-sample forecasting performance.

One study that provides a comparison between the two methods is:

Makridakis, S., & Hibon, M. (2000). The M3-competition: results, conclusions and implications. International Journal of Forecasting, 16(4), 451-476.

The M3 competition involved researchers and practitioners submitting their models for forecasting various time series. While the study does not focus specifically on comparing Box-Jenkins and information criteria-based methods, it does provide insights into which methods performed better in real-world situations. The results of the M3 competition indicate that the methods based on optimizing information criteria tend to perform better in terms of out-of-sample forecasting accuracy.

Given the limited direct comparison of the two approaches in the literature, the preference for information criteria-based methods can be attributed to the following reasons:

1. Automation: Information criteria-based methods automate the model selection process, reducing the need for subjective decisions and interpretation of ACF/PACF plots.

2. Efficiency: Grid searching over possible model orders and selecting the one that optimizes an information criterion is generally faster and more efficient than manually identifying the optimal order from ACF/PACF plots.

3. Out-of-sample forecasting performance: In practice, methods based on optimizing information criteria tend to yield better out-of-sample forecasting accuracy than the Box-Jenkins method.

While the Box-Jenkins method may still have educational value and could be useful in specific cases, for most practical applications, using an automated approach like the ones provided by the forecast or fable packages in R is recommended.

Answer 2

From my experience, finding ARIMA (p) and (q) parameters from the ACF/PACF plots yields better results than auto.arima, measured as lower RSME and/or lower MAPE errors. Note that ARIMA is a statistical model, and the output of the auto-correlation plots relate directly to the auto-regression (AR) and moving average (MA) parts of the model. I acknowledge that it's not very practical though, especially when one needs to develop several ARIMA models at once (eg. to predict sales of several products, etc).

This is where auto.arima comes in handy, as it automatically finds a decent set of parameters. It's not always the best parameters (ie. not necessarily the lowest forecast error metrics), but typically decent parameters yielding not so different error metrics as compared to manual search. These automatic processes for picking the best model are based on some Information Criterion, such as AIC or BIC, which increasingly penalizes the inclusion of more parameters in the model. So these methods tend to find a balance between a good performance and a simple model - is to say, they find the best performance with a simpler model. They do so as typically the inclusion of more parameters tends to yield lower error metrics, even though the model may be running into overfitting - it is thus a way to avoid overfitting.

In conclusion:

• If I am developing 1 (or very few) models, I'll search parameters manually through ACF/PACF for the best performance.
• If I am developing several ARIMA models, I'll conduct the search automatically by auto.arima for the sake of practicality, without fearing a major loss in performance.