Lately, I have been interested in phenomenons related to omission of variables. For example, it can be shown that the expected value of the sample variance under the inclusion of one variable $x_1$ but omission of one variable $x_2$ is $\mathbb{E}(s^2|x_1,x_2)= \sigma^2 + \frac{\sigma^2}{n-1} RSS_{x_1, \beta_2 x_2}$ where $\beta_2$ is the true coefficient of $x_2$ and $RSS_{x_1, \beta_2 x_2}$ is the residual sum of squares when running a regression with $x_1$ as predictor and $\beta_2x_2$ as outcome. Now, I am interested in getting a better expression for this when we have an $AR(2)$ process.

Let an $AR(2)$-process be given by $x_t = ax_{t-1}+bx_{t-2} + \epsilon_t$ (with the $\epsilon_i$ being independent and normal with standard deviation $\sigma$).

If we run a regression with $(x_2,\ldots, x_T)$ as the response vector and $(x_1,\ldots, x_{T-1})$ as the predicting variable, what can be said about the residual sum of squares? That is, if we (falsely) think that the process is an $AR(1)$-process, what can be said about the expected residual sum of squares?

Since I am quite unschooled in statistics (my only academic background is in mathematics), I would be interested in references (articles and books) as well as answers, even if they are only tangentially related to this question.


Nice introductory book on the topic related to different aspects of time series models could be Introduction to Time Series and Forecasting by Brockwell and Davis among many others. Roughly speaking, the characteristic of the autoregressive process of order $p$ is linked to the partial autocorrelation function. Estimating the $AR(p)$ process: $$X_t = \sum_{j=1}^p\phi_jX_{t-j} + \varepsilon_t$$

one common solution is to apply the Durbin-Levinson (wiki on the math of Levinson recursion) method, where the residual sum of squares of $AR(p) $

$$RSS_p = \mathbb{E} \varepsilon^2= \mathbb{E}(X_t - \sum_{j=1}^p\phi_jX_{t-j})^2$$

is linked to the $RSS_{p-1}$ as:

$$RSS_p = RSS_{p-1}(1-\varphi_{pp}^2),$$

with $\varphi_{pp}$ being the partial autocorrelations or the last component of

$$\Gamma_p^{-1}\gamma_p = {([\gamma(i-j)]}_{i,j=1}^p)^{-1}[\gamma(1),\gamma(2),\dots,\gamma(p)]^\prime,$$

and $\gamma(.)$ being autocorrelation function. Thus if you apply wrong order autoregression it will cost you in theory the $(1 - \varphi_{pp}^2))$, note that in practice the estimation error also adds here. In small samples it may happen that a smaller model $(AR(1))$ is a better predictor than the true model $AR(2)$ (as the parameters has to be estimated and they are not known!). This is also known as the parsimony property of a smaller model.

  • $\begingroup$ When you are saying $RSS_p = RSS_{p-1}(1-\phi^2_{pp})$, do you mean $\mathbb{E}(RSS_p) = \mathbb{E}(RSS_{p-1}(1-\phi^2_{pp}))$? It seems to me that the only sensible thing we can talk about is the expected value of the $RSS$. $\endgroup$ – Har Jul 20 '11 at 10:59
  • $\begingroup$ @Har, I have updated the answer with the definitions of $RSS_p$ and $AR(p)$, thanks for noting to that, but note that here I use theoretical model and errors with known parameters, in practice you have to estimate everything, so the conclusion on the prediction accuracy remains. Note also that $ARIMA$ models are a-theoretical usually and lacks the interpretation, so the biases in parameters due to "wrong order" are of least concern! Primary goal is the accurate prediction. $\endgroup$ – Dmitrij Celov Jul 21 '11 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.