Having decided the orders (p,q,d) of an ARIMA model - why do we use Maximum Likelihood Estimation instead of least squares to determine the coefficients?
An ARIMA model is after all very similar to a multivariable regression.
5
$\begingroup$
$\endgroup$
-
1$\begingroup$ I think this has been asked before, so I am only answering this in the comment: when $q>0$, OLS cannot be applied since the necessary ingredient are missing; you do not have the design matrix $X$ since you do not observe the lagged errors. If you had observable lagged errors, you could surely use OLS, but you don't. Meanwhile, for the case $q=0$, maximum likelihood and OLS are almost the same. $\endgroup$ – Richard Hardy Jan 16 '18 at 13:54
-
1$\begingroup$ Possible duplicate of What is/are the "mechanical" difference between multiple linear regression with lags and time series?. The question is different, but the answer answers it, so not really sure if it should be a duplicate or not. $\endgroup$ – Richard Hardy Jan 16 '18 at 14:00
-
$\begingroup$ @RichardHardy Interesting point, personally I have voted to leave this one open. In fact, if anything, some of the answers there really would be a better fit here! (Within its scope, this seems to be a "better" question - very clear and to the point.) Though I don't think the other question could be closed as a duplicate of this one either, because its scope seems to spread wider. $\endgroup$ – Silverfish Jan 16 '18 at 14:55
-
$\begingroup$ @Silverfish, you are probably right. Should the relevant part of the answer be just copied here? $\endgroup$ – Richard Hardy Jan 16 '18 at 15:11
-
$\begingroup$ @RichardHardy That seems perfectly appropriate to me! $\endgroup$ – Silverfish Jan 16 '18 at 19:06
3
$\begingroup$
$\endgroup$
OLS only works when all regressors are observable, because it employs the design matrix $X$ in estimating the model coefficients ($\beta=(X'X)^{-1}X'y$). Meanwhile, the ARIMA model specification contains unobservable variables in the moving average part (the lagged errors); hence, the OLS estimator is infeasible when the moving-average order $q>0$. On the other hand, an AR model can be estimated with OLS, and this is in fact quite a common approach. For normally distributed errors, it is also very close to the Maximum Likelihood (ML) solution. (There are subtle differences on how to treat the initial values in the ML estimation, but other than that it coincides with OLS for AR models.)
answered Jan 16 '18 at 19:57
-
$\begingroup$ Since I first posted this question, I've done further research and I came across this post from Rob Hyndman - in bullet point 3 he mentions that another package besides the R package called Eviews does use least squares for ARIMA model estimation. How do I reconcile that with what you mention about unobservable variables? (Note that the specific example mentioned in the post has order 0 for the MA components - but I assume that Eviews uses least squares even when q > 0 for MA(q).) $\endgroup$ – Reinstate Monica Jan 22 '18 at 19:57
-
$\begingroup$ Interesting. It is quite obvious that a direct OLS estimation of MA(q) and ARMA(p,q) with $q>0$ models is impossible due to the unobservable regressors. I would therefore not assume that Eviews uses OLS even when $q>0$ for MA(q). The closest one can get to OLS estimation of these models is a two-stage procedure where in stage 1 an AR(r) model is fit with a relatively large $r$, and then the residuals of this model are used as proxies for the unobservable lagged error terms in stage 2 where OLS is used. $\endgroup$ – Richard Hardy Jan 22 '18 at 20:07
-
$\begingroup$ Oh, and that post you refer to actually says Eviews uses nonlinear least squares (NLS) which is quite a different matter. It might be true that NLS is used for models with unobservable regressors. $\endgroup$ – Richard Hardy Jan 22 '18 at 20:10
-
$\begingroup$ How would non-linearity effect the issue? Either way you need observed values of the parameter to calculate least squares - no? $\endgroup$ – Reinstate Monica Jan 22 '18 at 20:59
-
$\begingroup$ @Alex, I do not know how NLS works exactly, but if it needs all regressors to be observable, then it cannot be applied directly. Perhaps there is a way related to the two-stage procedure I described above. But there OLS suffices, so I don't know. Maybe the EViews manual has an explanation. $\endgroup$ – Richard Hardy Jan 23 '18 at 6:23
