9
$\begingroup$

We keep on getting questions here about selecting ARIMA model orders based on ACF/PACF plots. This is the older methodology proposed by Box and Jenkins.

More modern tools like the auto.arima() function in the forecast package for R or ARIMA() in the fable package instead use a grid search over different orders and select the one that optimizes an information criterion. I highly respect the creators and maintainers of these packages and consider them among the best informed forecasters in the world.

Per Hyndman & Athanasopoulos (Rob Hyndman is the maintainer of the packages above), ACF/PACF plots can't be used to select orders if both the AR and the MA order are nonzero.

In the comments, Richard Hardy points to Shumway & Stoffer, Time Series Analysis and Its Application with R examples. This is a prime example of the way I often see the Box-Jenkins approach taught in textbooks: the process seems to involve a lot of decisions that are based on expertise (or, uncharitably described, arbitrariness), like what it means for a (P)ACF to decay "quickly" or not, whether variance "is changing" or not, and this seems to be very hard to automate, especially for larger sets of multiple time series. Overall, this advice seems to be most problematic for people who are just starting out with time series analysis, and who therefore IMO would be best served with an automated approach as the one referenced above.

Thus, I am confused why people would still use, teach and propagate the older Box-Jenkins method, rather than use a grid search using information criteria. I suspect this is just due to people perpetuating old and superseded advice.

Question: is there any published research on any benefits of a Box-Jenkins approach (however automated) over optimizing information criteria?

Related: Selecting ARIMA orders based on ACF-PACF vs. auto.arima, and actually almost all threads with the tag.

$\endgroup$
8
  • 2
    $\begingroup$ I think Shumway (as in Shumway & Stoffer "Time Series Analysis and Its Application with R examples" (4th ed., 2017)) was not very happy with auto.arima. Somewhere on his website (or perhaps also in the textbook) he had an example where a more Box-Jenkins-like approach does a better job than auto.arima on the popular air passengers dataset. Sorry for not remembering where exactly I read that, but here is a starting point. (This does not really answer the question, so I am posting it as a comment.) $\endgroup$ Nov 9, 2022 at 12:09
  • 1
    $\begingroup$ @RichardHardy: thanks a lot! I just searched through the book, and the only point where the AirPassengers dataset is mentioned is Example 3.49, which illustrates using (P)ACF to narrow the model space down to two possible ARIMA models on differenced-and-seasonally-differenced-logged data, among which they choose using information criteria. Nothing about auto.arima() here. (What I am unhappy about is that they discuss forecasting the logged series, but nowhere in the book do they address the back-transformation and the bias correction, at least not that I see.) $\endgroup$ Nov 9, 2022 at 12:23
  • $\begingroup$ It is somewhere out there :) In any case it is merely an anecdote, a single case. $\endgroup$ Nov 9, 2022 at 13:42
  • 1
    $\begingroup$ Just for the record, I find the description in section 3.7 "Building an ARIMA model" singularly unhelpful, especially for the non-specialist. We are told to plot the data and "inspect the graph for any anomalies". If the variability grows with time, we are told to Box-Cox-transform - but with what $\lambda$, and how would a non-specialist assess whether this is indicated? (Besides, a far simpler situation might be multiplicative seasonality, which can be modeled using exponential smoothing, moving away from ARIMA.) ... $\endgroup$ Nov 14, 2022 at 7:44
  • 1
    $\begingroup$ ... Next, "A time plot of the data will typically suggest whether any differencing is needed." This may be true for blatant trends, but will leave anyone who has not already seen a lot of time series hanging. Later, we are told to check whether the sample ACF decays "fast" - but what is "fast" decay, rather than "slow" decay? $\endgroup$ Nov 14, 2022 at 7:46

2 Answers 2

1
+200
$\begingroup$

While there is limited published research directly comparing the Box-Jenkins method based on ACF/PACF plots and information criteria-based methods in ARIMA model selection, most researchers and practitioners have gravitated towards using information criteria methods, such as AIC or BIC, due to their increased efficiency, automation, and out-of-sample forecasting performance.

One study that provides a comparison between the two methods is:

Makridakis, S., & Hibon, M. (2000). The M3-competition: results, conclusions and implications. International Journal of Forecasting, 16(4), 451-476.

The M3 competition involved researchers and practitioners submitting their models for forecasting various time series. While the study does not focus specifically on comparing Box-Jenkins and information criteria-based methods, it does provide insights into which methods performed better in real-world situations. The results of the M3 competition indicate that the methods based on optimizing information criteria tend to perform better in terms of out-of-sample forecasting accuracy.

Given the limited direct comparison of the two approaches in the literature, the preference for information criteria-based methods can be attributed to the following reasons:

  1. Automation: Information criteria-based methods automate the model selection process, reducing the need for subjective decisions and interpretation of ACF/PACF plots.

  2. Efficiency: Grid searching over possible model orders and selecting the one that optimizes an information criterion is generally faster and more efficient than manually identifying the optimal order from ACF/PACF plots.

  3. Out-of-sample forecasting performance: In practice, methods based on optimizing information criteria tend to yield better out-of-sample forecasting accuracy than the Box-Jenkins method.

While the Box-Jenkins method may still have educational value and could be useful in specific cases, for most practical applications, using an automated approach like the ones provided by the forecast or fable packages in R is recommended.

$\endgroup$
2
  • $\begingroup$ Thank you. This answer is useful and actually provides a reference. It might be added that most subsequent academic research about time series forecasting also used information criteria. I would have loved to see something more recent, to be honest... but it may simply be that there is nothing. $\endgroup$ Mar 29, 2023 at 16:25
  • $\begingroup$ The information-theoretic based arguments by Akaike 1973 seem to me to be strong arguments against the rather heuristic procedure of BJ. $\endgroup$
    – Math-fun
    Apr 3 at 16:00
0
$\begingroup$

From my experience, finding ARIMA (p) and (q) parameters from the ACF/PACF plots yields better results than auto.arima, measured as lower RSME and/or lower MAPE errors. Note that ARIMA is a statistical model, and the output of the auto-correlation plots relate directly to the auto-regression (AR) and moving average (MA) parts of the model. I acknowledge that it's not very practical though, especially when one needs to develop several ARIMA models at once (eg. to predict sales of several products, etc).

This is where auto.arima comes in handy, as it automatically finds a decent set of parameters. It's not always the best parameters (ie. not necessarily the lowest forecast error metrics), but typically decent parameters yielding not so different error metrics as compared to manual search. These automatic processes for picking the best model are based on some Information Criterion, such as AIC or BIC, which increasingly penalizes the inclusion of more parameters in the model. So these methods tend to find a balance between a good performance and a simple model - is to say, they find the best performance with a simpler model. They do so as typically the inclusion of more parameters tends to yield lower error metrics, even though the model may be running into overfitting - it is thus a way to avoid overfitting.

In conclusion:

  • If I am developing 1 (or very few) models, I'll search parameters manually through ACF/PACF for the best performance.
  • If I am developing several ARIMA models, I'll conduct the search automatically by auto.arima for the sake of practicality, without fearing a major loss in performance.
$\endgroup$
4
  • 1
    $\begingroup$ Thank you. Can you explain just how you use (P)ACF plots to determine ARMA orders? From what I understand, they cannot be used if both orders are nonzero. Also, if you see lower MAPEs, that does set off alarm bells for me, because ARIMA aims for unbiased expectation forecasts, and the MAPE elicits forecasts that are systematically lower than the expectation. $\endgroup$ Mar 23, 2023 at 15:52
  • $\begingroup$ I use check auto-correlation for the MA(p) parameter and the remaining partial auto-correlation for the AR(p) parameter. In both cases the result is reached by visual inspection of the plot. I'm commenting only from my (limited) experience, in which I usually achieve lower error metrics when searching parameters manually. Also I usually tend to priorize RMSE over MAPE when choosing the best manual model. I have never came across that suggestion that this method cannot be used if both orders are nonzero - I'll dig it further. Thanks for pointing that out. $\endgroup$
    – jma.alves
    Mar 23, 2023 at 16:30
  • 1
    $\begingroup$ ACF and PACF can be a guide if we have one order (either AR or MA) because we can see the lag where ACF or PACF cuts off. In the case of both orders, both plots gradually decline and we can't point out the exact lags we need for AR and MA orders. I haven't come across any source that explains how to choose lags in the case of both orders. Personally, I only use ACF/PACF as a mere guide to narrow down my search space for choosing the ARIMA structure with information criteria. $\endgroup$ Mar 24, 2023 at 4:45
  • 1
    $\begingroup$ I am still rather unclear as to how you use ACF/PACF to narrow down the search space. I see lots of questions here on interpreting ACF/PACF plots and am always at a loss as to what to answer, beyond wondering why people do this in the first place, rather than using information criteria. I would also appreciate published comparisons of different approaches. $\endgroup$ Mar 29, 2023 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.