8
$\begingroup$

I am doing auto-regress by usual linear regression package.
e.g. $y_t=φx+ε_t$ with $x =y_{t-1}$

My reason is that,
Auto-regression does assumes iid errors, same for linear regression. Linear Regression doesn't have assumption on independent variables. What's different is merely that the independent variables is replaced with lagged dependent variable y.

But I see AR always is fitted by specific methods, I am afraid I'm doing wrong.

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ Actually, Hamilton "Time Series Analysis" says explicitly on p. 123 that OLS is the common method of estimation of AR processes. $\endgroup$
    – Richard Hardy
    Commented Nov 18, 2015 at 10:37

2 Answers 2

Reset to default
7
$\begingroup$

To answer the title question, fitting an AR($p$) model using OLS will yield biased estimates. The reason is that for unbiasedness, the model errors should be uncorrelated with past, current and future values of regressors, which is not the case in autoregressive models. For example, in case of AR(1)

$$ y_t=\varphi y_{t-1}+\varepsilon_t $$

(assuming zero mean for simplicity). Lag this by 1 to obtain

$$ y_{t-1}=\varphi y_{t-2}+\varepsilon_{t-1}. $$

Note that $\varepsilon_{t-1}$ enters the model of $y_{t-1}$; hence, the regressor $y_{t-1}$ will be correlated with lagged error $\varepsilon_{t-1}$. The argument is given (without proof) e.g. in this lecture note, p. 5-6.

On a positive note, OLS gives consistent estimators for an autoregressive model (see the same lecture note, p. 4-5)

Also, in my experience OLS is quite popular for fitting AR models, and is pretty standard for fitting multivariate AR, i.e. VAR, models.

Update (year 2024): biasedness is not really the main issue, as I believe all of the popular estimators of AR($p$) coefficients are biased. E.g. the other popular estimator, maximum likelihood, is also biased, but it uses the information in the data more efficiently than OLS does. See Glen_b's answer for that. I think that should be the accepted answer.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Nice answer, upvoted. Unfortunately the link appears to be broken? $\endgroup$
    – DanLewis3264
    Commented Dec 4, 2022 at 2:33
  • $\begingroup$ @DanLewis3264, I should have included a full title back then... Now it is gone. $\endgroup$
    – Richard Hardy
    Commented Dec 4, 2022 at 9:37
  • $\begingroup$ Conditional on the first $p$ observations the MLE and least squares estimate are equivalent. Could it be that the MLE is also biased? $\endgroup$
    – Sextus Empiricus
    Commented Apr 15 at 10:25
  • $\begingroup$ @SextusEmpiricus, I think you are right and Glen_b's answer is the right one. Seeing that I have learned something since 2015 offers a bit of consolation :) $\endgroup$
    – Richard Hardy
    Commented Apr 15 at 11:14
3
$\begingroup$

If you fit by regressing $\mathbf{y}_{p+1:n}=(y_{p+1},...,y_n)^\top$ on its lags $X=[\mathbf{y}_{p:n-1},\mathbf{y}_{p-1:n-2},...,\mathbf{y}_{1:n-p}]$ the lags of that on you're going to be conditioning on the first $p$ values (for an AR(p)). If you fit by say maximum likelihood you're able to incorporate the likelihood for the first $p$ values.

If $n$ is not very large relative to $p$ is can sometimes make a substantial difference.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I thought that estimating by OLS would amount to effectively cutting the sample by $p$ observations so that lagged regressors could be obtained -- instead of appending the data with some arbitrary values for the negative lags. As such there would be no conditioning on the first $p$ values. $\endgroup$
    – Richard Hardy
    Commented Nov 17, 2015 at 18:34
  • $\begingroup$ @RIchardHardy you just exactly described why it is conditioning on the first $p$ values. They're removed from $y$, but they're all still in $X$ in the regression (and in regression, $y$ is conditioned on $X$). ergo the likelihood you maximize is one conditioning on the first $p$ y-values. (You can also show it algebraically by decomposing the likelihood, it's quite straightforward. The likelihood for the AR can be split via the prediction error decomposition into two components, that for $y_{p+1},...,y_n$ and $y_1,...,y_p$, ...ctd $\endgroup$
    – Glen_b
    Commented Nov 17, 2015 at 21:18
  • $\begingroup$ ctd... if you write the likelihood conditional on $y_1,...,y_p$ the second term drops out and you're left with the regression model). $\endgroup$
    – Glen_b
    Commented Nov 17, 2015 at 21:22
  • $\begingroup$ Thanks for the explanation! Apparently, my confusion is in terminology. Also, I did not make the connection between OLS and maximum likelihood (I only thought in terms of OLS), which apparently explains conditioning. It still gives me some thought to digest. $\endgroup$
    – Richard Hardy
    Commented Nov 17, 2015 at 21:22
  • 1
    $\begingroup$ A question where the likelihood for an AR1 process is completely spelled out is: How to estimate maximum liklihood of a custom log likelihood function?. If it wasn't for the first term, then the likelihood would be entirely equivalent to OLS. $\endgroup$
    – Sextus Empiricus
    Commented Apr 15 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.