What is/are the "mechanical" difference between multiple linear regression with lags and time series?

Question

I'm a graduate from business and economics who's currently studying for a master's degree in data engineering. While studying linear regression (LR) and then time series analysis (TS), a question popped into my mind. Why create a whole new method, i.e., time series (ARIMA), instead of using multiple linear regression and adding lagged variables to it (with the order of lags determined using ACF and PACF)? So the teacher suggested that I write a little essay about the issue. I wouldn't come to look for help empty-handed, so I did my research on the topic.

I knew already that when using LR, if the Gauss-Markov assumptions are violated, the OLS regression is incorrect, and that this happens when using time series data (autocorrelation, etc). (another question on this, one G-M assumption is that the independent variables should be normally distributed? or just the dependent variable conditional to the independent ones?)

I also know that when using a distributed lag regression, which is what I think I'm proposing here, and using OLS to estimate parameters, multicollinearity between variables may (obviously) arise, so estimates would be wrong.

In a similar post about TS and LR here, @IrishStat said:

... a regression model is a particular case of a Transfer Function Model also known as a dynamic regression model or an XARMAX model. The salient point is that model identification in time series i.e. the appropriate differences, the appropriate lags of the X's , the appropriate ARIMA structure, the appropriate identification of unspecified deterministic structure such as Pulses, level Shifts,Local time trends, Seasonal Pulses, and incorporation of changes in parameters or error variance must be considered.

(I also read his paper in Autobox about Box Jenkins vs LR.) But this still does not resolve my question (or at least it doesn't clarify the different mechanics of RL and TS for me).

It is obvious that even with lagged variables OLS problems arise and it is not efficient nor correct, but when using maximum likelihood, do these problems persist? I have read that ARIMA is estimated through maximum likelihood,so if the LR with lags is estimated with ML instead of OLS, does it yield the "correct" coefficients (lets assume that we include lagged error terms as well, like an MA of order q).

In short, is the problem OLS? Is the problem solved applying ML?

Answer 1

Why create a whole new method, i.e., time series (ARIMA), instead of using multiple linear regression and adding lagged variables to it (with the order of lags determined using ACF and PACF)?

One immediate point is that a linear regression only works with observed variables while ARIMA incorporates unobserved variables in the moving average part; thus, ARIMA is more flexible, or more general, in a way. AR model can be seen as a linear regression model and its coefficients can be estimated using OLS; $\hat\beta_{OLS}=(X'X)^{-1}X'y$ where $X$ consists of lags of the dependent variable that are observed. Meanwhile, MA or ARMA models do not fit into the OLS framework since some of the variables, namely the lagged error terms, are unobserved, and hence the OLS estimator is infeasible.

one G-M assumption is that the independent variables should be normally distributed? or just the dependent variable conditional to the independent ones?

The normality assumption is sometimes invoked for model errors, not for the independent variables. However, normality is required neither for the consistency and efficiency of the OLS estimator nor for the Gauss-Markov theorem to hold. Wikipedia article on the Gauss-Markov theorem states explicitly that "The errors do not need to be normal".

multicollinearity between variables may (obviously) arise, so estimates would be wrong.

A high degree of multicollinearity means inflated variance of the OLS estimator. However, the OLS estimator is still BLUE as long as the multicollinearity is not perfect. Thus your statement does not look right.

It is obvious that even with lagged variables OLS problems arise and it is not efficient nor correct, but when using maximum likelihood, do these problems persist?

An AR model can be estimated using both OLS and ML; both of these methods give consistent estimators. MA and ARMA models cannot be estimated by OLS, so ML is the main choice; again, it is consistent. The other interesting property is efficiency, and here I am not completely sure (but clearly the information should be available somewhere as the question is pretty standard). I would try commenting on "correctness", but I am not sure what you mean by that.

Answer 2

That's a great question. The real difference between ARIMA models and multiple linear regression lies in your error structure. You can manipulate the independent variables in a multiple linear regression model so that they fit your time series data, which is what @IrishStat is saying. However, after that, you need to incorporate ARIMA errors into your multiple regression model to get correct coefficient and test results. A great free book on this is: https://www.otexts.org/fpp/9/1. I've linked the section that discusses combining ARIMA and multiple regression models.

Answer 3

Good question, I actually have built both in my day job as a Data Scientist. Time series models are easy to build (forecast package in R lets you build one in less in 5 seconds), the same or more accurate than regression models, etc. Generally, one should always build time series, then regression. There are Philosophical implications of time series as well, if you can predict without knowing anything, then what does that mean?

My take on Darlington. 1) "Regression is far more flexible and powerful, producing better models. This point is developed in numerous spots throughout the work."

No, quite the opposite. Regression models make far more assumptions than time series models. The fewer the assumptions, the more likely the ability to withstand the earthquake (regime change). Furthermore, time series models respond faster to sudden shifts.

2) "Regression is far easier to master than ARIMA, at least for those already familiar with the use of regression in other areas." This is circular reasoning.

3) "Regression uses a "closed" computational algorithm that is essentially guaranteed to yield results if at all possible, while ARIMA and many other methods use iterative algorithms that often fail to reach a solution. I have often seen the ARIMA method "hang up" on data that gave the regression method no problem."

Regression gives you an answer, but is it the right answer? If I build linear regression and machine learning models and they all come to the same conclusion, what does it mean?

So in summary, yes regression and time series can both answer the same question and technically, time series is technically regression (albeit auto-regression). Time series models are less complex and therefore more robust than regression models. If you think about specialization, then TS models specialize in forecasting whereas regression specialize in understanding. It boils down to whether you want to explain or predict.

Answer 4

In think the deepest difference between transfer functions and multipe linear regression (in its usual use) lies in their objectives, multiple regressions is oriented to find the main causal observable determinants of the dependent variable while transfer functions just want to forecasts the effect on a dependent variable of the variation of a specific exogenous variable...In summary, multiple regression is oriented to exhaustive explanation and transfer function to forecasting very specific effects...