The Box-Jenkins model selection procedure in time series analysis begins by looking at the autocorrelation and partial autocorrelation functions of the series. These plots can suggest the appropriate $p$ and $q$ in an ARMA$(p,q)$ model. The procedure continues by asking the user to apply the AIC/BIC criteria to select the most parsimonious model among those that produce a model with a white noise error term.

I was wondering how these steps of visual inspection and criterion-based model selection impact the estimated standard errors of the final model. I know that many search procedures in a cross-sectional domain can bias standard errors downward, for example.

On the first step, how does selecting the appropriate number of lags by looking at the data (ACF/PACF) impact the standard errors for time series models?

I would guess that selecting the model based upon AIC/BIC scores would have an impact analogous to that for cross-sectional methods. I actually don't know much about this area either, so any comments would be appreciated on this point as well.

Lastly, if you wrote down the precise criterion used for each step, could you bootstrap the entire process to estimate the standard errors and eliminate these concerns?

  • $\begingroup$ is bias in standard errors (of parameters?) so important in a-theoretic ARMA models? ARMA models ASFAIK are used for short-run forecasts mostly. The problems with parameters interpretation and their properties are less (least?) important. Of course if you are not meaning the characteristics of an innovation process (error term), planing to produce relevant prediction intervals. $\endgroup$ – Dmitrij Celov Jul 5 '11 at 8:13
  • $\begingroup$ @Dmitrij, There are two main reasons why I am concerned about bias in the standard errors of the coefficients. The first, as you alluded to, is the creation of prediction intervals. The second is testing for structural breaks in the model, a common question that an economist would be interested in answering. The standard errors generated using a selection procedure should be too small, giving prediction intervals that are too narrow and test statistics that are too large. $\endgroup$ – Charlie Jul 6 '11 at 14:04
  • $\begingroup$ but in a-theoretic models (meaning there is no theory, no structure), structural breaks are little to do with the parameters it would be some general tests, regarding the behavior of the residuals of the model. Well in this case unbiased estimates of models parameters are less important, ARMA simply does not have structural models interpretation. Thus parsimonious models are indeed better predictors, since they well balance the usually poor properties of small sample estimators and the accuracy of prediction. $\endgroup$ – Dmitrij Celov Jul 6 '11 at 14:25
  • $\begingroup$ Note, that even if you know the data generating process that has a lot of parameters, in small samples simpler model will probably do better predictions, but in structural context the parameters of such a model will be very biased (omitted variable bias)! $\endgroup$ – Dmitrij Celov Jul 6 '11 at 14:30

Any model selection procedure will affect the standard errors and this is hardly ever accounted for. For example, prediction intervals are computed conditionally on the estimated model and the parameter estimation and model selection are usually ignored.

It should be possible to bootstrap the whole procedure in order to estimate the effect of the model selection process. But remember that time series bootstrapping is trickier than normal bootstrapping because you have to preserve the serial correlation. The block bootstrap is one possible approach although it loses some serial correlation due to the block structure.

| cite | improve this answer | |

In my opinion selecting the appropriate number of lags is no different than selecting the number of input series in a stepwise forward regression procedure. The incremental importance of lags or a specific input series is the basis for the tentative model specification.

Since you have asserted that the acf/pacf is the only basis for Box-Jenkins model selection, let me tell you what some experience has taught me. If a series exhibits an acf that doesn't decay, the Box-Jenkins approach (circa 1965) suggests differencing the data. But if a series has a level shift, like the Nile data, then the "visually apparent" non-stationarity is a symptom of needed structure but differencing is not the remedy. This Nile dataset can be modeled without differencing by simply identifying the need for a level shift first. In a similar vein we are taught using 1960 concepts that if the acf exhibits a seasonal structure (i.e. significant values at lags of s,2s,3s,...) then we should incorporate a seasonal ARIMA component. For discussion purposes, consider a series that is stationary around a mean and at fixed intervals, say every June there is a "high value". This series is properly treated by incorporating an "old-fashioned" dummy series of 0 and 1's (at June) in order to treat the seasonal structure. A seasonal ARIMA model would incorrectly use memory instead of an unspecified but waiting-to-be-found X variable. These two concepts of identifying/incorporating unspecified deterministic structure are direct applications of the work of I. Chang, William Bell, George Tiao, R.Tsay, Chen et al (starting in 1978) under the general concept of Intervention Detection.

Even today some analysts are mindlessly performing memory maximization strategies, calling them Automatic ARIMA, without recognizing that "mindless memory modeling" assumes that deterministic structure such as pulses, level shifts, seasonal pulses and local time trends are non-existent or worse yet play no role in model identification. This is akin to putting one's head in the sand, IMHO.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the excellent advice on model selection, but I was interested in how this process impacts our inference afterwards. $\endgroup$ – Charlie Jul 8 '11 at 19:57
  • $\begingroup$ Charlie: I don't believe that up-front data diagnistics such as reviewing autocorrelations or cross-correlations or any other Model Identification Procedure like graphical presentations has any effect on the statistical significance of any estimated parameters. That is my opinion and "here I stand I can do no other". Thanks for the kudos and if I can help young Charlie in any way please get in touch with me as I love to be listed as a supporting (unpaid !) resource in Phd dissertations. $\endgroup$ – IrishStat Jul 8 '11 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.