How does ACF & PACF identify the order of MA and AR terms?

It's been more than 2 years that I am working on different time series. I have read on many articles that ACF is used to identify order of MA term, and PACF for AR. There is a thumb rule that for MA, the lag where ACF shuts off suddenly is the order of MA and similarly for PACF and AR.

Here is one of the articles I followed from PennState Eberly College of Science.

My Question is why is it so? For me even ACF can give AR term. I need explanation of thumb rule mentioned above. I am not able to understand thumb rule intuitively/mathematically that why -

Identification of an AR model is often best done with the PACF.
Identification of an MA model is often best done with the ACF rather than the PACF

Please note:- I don't need how but "WHY". :)

The quotes are from the link in the OP:

Identification of an AR model is often best done with the PACF.

For an AR model, the theoretical PACF “shuts off” past the order of the model. The phrase “shuts off” means that in theory the partial autocorrelations are equal to 0 beyond that point. Put another way, the number of non-zero partial autocorrelations gives the order of the AR model. By the “order of the model” we mean the most extreme lag of x that is used as a predictor.

... a $k^{\text{th}}$ order autoregression, written as AR(k), is a multiple linear regression in which the value of the series at any time t is a (linear) function of the values at times $t-1,t-2,\ldots,t-k:$

$$\begin{equation*} y_{t}=\beta_{0}+\beta_{1}y_{t-1}+\beta_{2}y_{t-2}+\cdots+\beta_{2}y_{t-k}+\epsilon_{t}. \end{equation*}$$

This equation looks like a regression model, as indicated on the linked paged... So what is a possible intuition of what we are doing...

In Chinese whispers or the telephone game as illustrated here

the message gets distorted as it is whispered from person to person, and all traces of resemblance (any truthful words, if you will) are lost after the red participant (with the exception of the article 'a'). PACF would tell us that the coefficients for the blue and the yellow participants are non-contributory once the effect of the brown and red participants are accounted for (the green participant at the end of the line doesn't distort the message).

It is not difficult to come very close to the actual output of the R function by actually obtaining consecutive OLS regressions through the origin of farther lagged sequences, and collecting the coefficients into a vector. Schematically,

a very similar process to the telephone game - it'll come a point, when there won't be any variability in the signal of the actual initial time series found in progressively more distant snippets of itself.

For an MA model, the theoretical PACF does not shut off, but instead tapers toward 0 in some manner. A clearer pattern for an MA model is in the ACF. The ACF will have non-zero autocorrelations only at lags involved in the model.

A moving average term in a time series model is a past error (multiplied by a coefficient).

The $q^{\text{th}}$-order moving average model, denoted by MA(q) is

$$x_t = \mu + w_t +\theta_1w_{t-1}+\theta_2w_{t-2}+\dots + \theta_qw_{t-q}$$

with $w_t \overset{iid}{\sim} N(0, \sigma^2_w).$

Here, it is not the message resemblance across time points that is searched backwards in time step-by-step, but rather the contribution of the noise, which I picture as the often massive deviations that a random walk can lead along the time line:

Here there are multiple, progressively offset sequences that are correlated, discarding any contribution of the intermediate steps. This would be the graph of the operations involved:

In this regard, "CV is cool!" is not completely different than "Naomi has a pool". From the noise point of view, the rhymes are still there all the way to the beginning of the game.

• This is a really cool answer, good job (+1) – Firebug Nov 15 '17 at 12:25
• Rob Hyndman suggests this strategy for ARIMA, which uses both pacf and acf to determine the orders. Do we need to know beforehand what series we have to use the strategy described in your answer? thanks! – stucash May 8 '19 at 11:51
• Please take my answer as a didactic exercise. I am no expert in the topic. – Antoni Parellada May 8 '19 at 11:53

Robert Nau from Duke's Fuqua School of Business gives a detailed and somewhat intuitive explanation of how ACF and PACF plots can be used to choose AR and MA orders here and here. I give a brief summary of his arguments below.

A simple explanation of why PACF identifies the AR order

The partial autocorrelations can be computed by fitting a sequence of AR models starting with the first lag only and progressively adding more lags. The coefficient of lag $k$ in an AR($k$) model gives the partial autocorrelation at lag $k$. Given this, if the partial autocorrelation "cuts off"/ceases to be significant at a certain lag (as seen in an ACF plot) this indicates that that lag does not add explanatory power to a model and therefore that the AR order should be the previous lag.

A more complete explanation which also addresses the use of ACF to identify the MA order

Time series can have AR or MA signatures:

• An AR signature corresponds to a PACF plot displaying a sharp cut-off and a more slowly decaying ACF;
• An MA signature corresponds to an ACF plot displaying a sharp cut-off and a PACF plot that decays more slowly.

AR signatures are often associated with positive autocorrelation at lag 1, suggesting that the series is slightly "underdifferenced" (this means that further differencing is necessary to completely eliminate autocorrelation). Since AR terms achieve partial differencing (see below), this can be fixed by adding an AR term to the model (hence the name of this signature). Therefore a PACF plot with a sharp cut-off (accompanied by a slowly decaying ACF plot with a positive first lag) can indicate the order of the AR term. Nau puts it as folows:

If the PACF of the differenced series displays a sharp cutoff and/or the lag-1 autocorrelation is positive--i.e., if the series appears slightly "underdifferenced"--then consider adding an AR term to the model. The lag at which the PACF cuts off is the indicated number of AR terms.

MA signatures, on the other hand, are commonly associated with negative first lags, suggesting that the series is "overdifferenced" (i.e. it is necessary to partially cancel out the differencing to obtain a stationary series). Since MA terms can cancel an order of differencing (see below), the ACF plot of a series with an MA signature indicates the necessary MA order:

If the ACF of the differenced series displays a sharp cutoff and/or the lag-1 autocorrelation is negative--i.e., if the series appears slightly "overdifferenced"--then consider adding an MA term to the model. The lag at which the ACF cuts off is the indicated number of MA terms.

Why AR terms achieve partial differencing and MA terms partially cancel previous differencing

Take a basic ARIMA(1,1,1) model, presented without the constant for simplicity:

$y_t = Y_t - Y_{t-1}$

$y_t = \phi y_{t-1} + e_t - \theta e_{t-1}$

Defining $B$ as the lag/backshift operator, this can be written as follows:

$y_t = (1-B)Y_t$

$y_t = \phi B y_t + e_t - \theta B e_t$

which can be further simplified to give:

$(1-\phi B) y_t = (1-\theta B) e_t$

or equivalently:

$(1-\phi B)(1-B) Y_t = (1-\theta B)e_t$.

We can see that the AR(1) term gave us the $(1-\phi B)$ term, thus partially (if $\phi \in (0,1)$) increasing the order of differencing. Moreover, if we manipulate $B$ as a numeric variable (which we can do because it is a linear operator), we can see that the MA(1) term gave us the $(1-\theta B)$ term, thus partially cancelling out the original differencing term—$(1-B)$—in the left-hand-side.

On a higher level, here is how to understand it. (If you need a more mathematical approach, I can gladly go after some of my notes on time series analysis)

ACF and PACF are theoretical statistical constructs just like an expected value or variance, but on different domains. The same way that Expected values come up when studying random variables, ACF and PACF come up when studying time series.

When studying random variables, there is the question of how to estimate their parameters, which is where the method of moments, MLE and other procedures and constructs come in, as well as inspecting the estimates, their standard errors and etc.

Inspecting the estimated ACF and PACF come is from the same idea, estimating the parameters of a random time series process. Get the idea?

If you think you need a more mathematically inclined answer, please let me know, and I'll try and see if I can craft something by the end of the day.