What is/are the “mechanical” difference between multiple linear regression with lags and time series?

I'm a graduate from business and economics who's currently studying for a master's degree in data engineering. While studying linear regression (LR) and then time series analysis (TS), a question popped into my mind. Why create a whole new method, i.e., time series (ARIMA), instead of using multiple linear regression and adding lagged variables to it (with the order of lags determined using ACF and PACF)? So the teacher suggested that I write a little essay about the issue. I wouldn't come to look for help empty-handed, so I did my research on the topic.

I knew already that when using LR, if the Gauss-Markov assumptions are violated, the OLS regression is incorrect, and that this happens when using time series data (autocorrelation, etc). (another question on this, one G-M assumption is that the independent variables should be normally distributed? or just the dependent variable conditional to the independent ones?)

I also know that when using a distributed lag regression, which is what I think I'm proposing here, and using OLS to estimate parameters, multicollinearity between variables may (obviously) arise, so estimates would be wrong.

In a similar post about TS and LR here, @IrishStat said:

... a regression model is a particular case of a Transfer Function Model also known as a dynamic regression model or an XARMAX model. The salient point is that model identification in time series i.e. the appropriate differences, the appropriate lags of the X's , the appropriate ARIMA structure, the appropriate identification of unspecified deterministic structure such as Pulses, level Shifts,Local time trends, Seasonal Pulses, and incorporation of changes in parameters or error variance must be considered.

(I also read his paper in Autobox about Box Jenkins vs LR.) But this still does not resolve my question (or at least it doesn't clarify the different mechanics of RL and TS for me).

It is obvious that even with lagged variables OLS problems arise and it is not efficient nor correct, but when using maximum likelihood, do these problems persist? I have read that ARIMA is estimated through maximum likelihood,so if the LR with lags is estimated with ML instead of OLS, does it yield the "correct" coefficients (lets assume that we include lagged error terms as well, like an MA of order q).

In short, is the problem OLS? Is the problem solved applying ML?

• Uncanny resemblance there to John Maynard Keynes. – Nick Cox Dec 31 '15 at 13:41
• Hi @NickCox , yes, he is my fav economist, I think he was an amazing man and extremely talented in many ways... any help on my question? What I'm trying to figure out is why wont the lagged model work with OLS estimation, and if it would estimate correctly with maximum likelyhood estimation. I understand that the best model is a transfer function, and am studying it at the moment. But the theoretical question still remains there about OLS. If no autocorrelation was present cause the lags eliminate it (assume also that multicoll. is not present), would it work? or is there still and underlying – Miguel M. Dec 31 '15 at 19:38
• @NickCox ...effect/violation of gaussian assumptions that OLS cannot work with and that cannot be fitted with this method? As you can see I am a bit lost with this, if its too long to answer, please if you can provide some lecture that might enlighten I'd appreciate too – Miguel M. Dec 31 '15 at 19:40
• In terms of mechanics let me suggest that the ARMA model for the user suggested (appropriately differenced) X variable reflects non-stationarity.If that filter is applied to BOTH appropriately differenced series the resultant pair of series can often be studied via cross-correlation procedures yielding a suggested lag structure (understanding). This lag structure can then be applied to the appropriately differenced original series to yield a suggestion about the unspecified/background series ( the tentative error process.). This error process can then be studied to yield the appropriate ARMA. – IrishStat Dec 31 '15 at 20:01
• @IrishStat so please let me rephrase what you just said. Let us have dependent variable Yt and independent variable Xt, we difference both Yt and Xt until we have stationarity in both, and then we can apply the cross correlation function to find out the lag structure. Afterwards we regress Yt to Xt and we study the error term. If we find ARMA structure in the error term, we apply it in the model until we have white noise, correct? But, my question is still, is that last model fitted via OLS? If not, why not, and what method do we use? – Miguel M. Dec 31 '15 at 20:26

Why create a whole new method, i.e., time series (ARIMA), instead of using multiple linear regression and adding lagged variables to it (with the order of lags determined using ACF and PACF)?

One immediate point is that a linear regression only works with observed variables while ARIMA incorporates unobserved variables in the moving average part; thus, ARIMA is more flexible, or more general, in a way. AR model can be seen as a linear regression model and its coefficients can be estimated using OLS; $\hat\beta_{OLS}=(X'X)^{-1}X'y$ where $X$ consists of lags of the dependent variable that are observed. Meanwhile, MA or ARMA models do not fit into the OLS framework since some of the variables, namely the lagged error terms, are unobserved, and hence the OLS estimator is infeasible.

one G-M assumption is that the independent variables should be normally distributed? or just the dependent variable conditional to the independent ones?

The normality assumption is sometimes invoked for model errors, not for the independent variables. However, normality is required neither for the consistency and efficiency of the OLS estimator nor for the Gauss-Markov theorem to hold. Wikipedia article on the Gauss-Markov theorem states explicitly that "The errors do not need to be normal".

multicollinearity between variables may (obviously) arise, so estimates would be wrong.

A high degree of multicollinearity means inflated variance of the OLS estimator. However, the OLS estimator is still BLUE as long as the multicollinearity is not perfect. Thus your statement does not look right.

It is obvious that even with lagged variables OLS problems arise and it is not efficient nor correct, but when using maximum likelihood, do these problems persist?

An AR model can be estimated using both OLS and ML; both of these methods give consistent estimators. MA and ARMA models cannot be estimated by OLS, so ML is the main choice; again, it is consistent. The other interesting property is efficiency, and here I am not completely sure (but clearly the information should be available somewhere as the question is pretty standard). I would try commenting on "correctness", but I am not sure what you mean by that.

• Hi Mr. Hardy, thank you very much for the answer. Regarding the observed vs. non-observed values , just to summarize. In ARIMA and time series (more specifically XARIMAX), we employ a "dynamic" approach, for we use the prediction error, and in linear regression we do not use them - but we could use them nonetheless. I do not understand then the issue here. Or as @IrishStat says, the only difference is the path to identification and model revision strategies? – Miguel M. Jan 2 '16 at 12:45
• And what about estimation, is OLS (again) correct when including lagged errors in the model? Regarding multicolinearity, I meant that the estimated coefficients might not be correct, for their estimation has a big variance. By correct method I meant, if using OLS gives unbiased and efficient estimates compared to ML when using the proposed lagged models. – Miguel M. Jan 2 '16 at 12:53
• @MiguelM, I am travelling now, I will try to get back later. – Richard Hardy Jan 3 '16 at 6:36
• Regarding "in linear regression we do not use them - but we could use them nonetheless": we do not observe these variables, and hence they cannot be used in the linear regression framework due to the mechanics there (as I noted in the answer, the estimator is infeasible); however, they can be used in ARIMA framework. Regarding "is OLS (again) correct when including lagged errors in the model?", yes, that should be true. Regarding "correctness", if the model is correctly specified and both OLS and ML are feasible, both should work fine. Under misspecification things tend to go wrong. – Richard Hardy Jan 3 '16 at 19:02
• I must be bad at explaining, and I find it difficult to come up with an alternative explanation in this case... Suppose you have to run a regression $y=\beta_0+\beta_1 x+\varepsilon$, and you do not observe $x$. Then there is no way you can run the regression. This is the main point. OLS does not allow having missing variables. However, certain structures with missing variables can be recovered using ML, and one example of such a structure is the MA model. (The regression $y=\beta_0+\beta_1 x+\varepsilon$ is infeasible not only for OLS but also for the ML estimation when $x$ is not observed.) – Richard Hardy Jan 3 '16 at 23:12

That's a great question. The real difference between ARIMA models and multiple linear regression lies in your error structure. You can manipulate the independent variables in a multiple linear regression model so that they fit your time series data, which is what @IrishStat is saying. However, after that, you need to incorporate ARIMA errors into your multiple regression model to get correct coefficient and test results. A great free book on this is: https://www.otexts.org/fpp/9/1. I've linked the section that discusses combining ARIMA and multiple regression models.

Good question, I actually have built both in my day job as a Data Scientist. Time series models are easy to build (forecast package in R lets you build one in less in 5 seconds), the same or more accurate than regression models, etc. Generally, one should always build time series, then regression. There are Philosophical implications of time series as well, if you can predict without knowing anything, then what does that mean?

My take on Darlington. 1) "Regression is far more flexible and powerful, producing better models. This point is developed in numerous spots throughout the work."

No, quite the opposite. Regression models make far more assumptions than time series models. The fewer the assumptions, the more likely the ability to withstand the earthquake (regime change). Furthermore, time series models respond faster to sudden shifts.

2) "Regression is far easier to master than ARIMA, at least for those already familiar with the use of regression in other areas." This is circular reasoning.

3) "Regression uses a "closed" computational algorithm that is essentially guaranteed to yield results if at all possible, while ARIMA and many other methods use iterative algorithms that often fail to reach a solution. I have often seen the ARIMA method "hang up" on data that gave the regression method no problem."

Regression gives you an answer, but is it the right answer? If I build linear regression and machine learning models and they all come to the same conclusion, what does it mean?

So in summary, yes regression and time series can both answer the same question and technically, time series is technically regression (albeit auto-regression). Time series models are less complex and therefore more robust than regression models. If you think about specialization, then TS models specialize in forecasting whereas regression specialize in understanding. It boils down to whether you want to explain or predict.

• "Time series models are less complex and therefore more robust than regression models" .... What you meant to say was "ARIMA models are less complex and therefore more robust than regression models" . Incorporating ARIMA and regression is referred to as Transfer Function Models ...which is then the wise choice thus combining both understanding (regression) and unknown/unspecified background factors (ARIMA) . – IrishStat Dec 31 '15 at 18:58
• @IrishStat Hi Mr. Reilly, I've been reading your answers to several posts here in stackexchange, and I've also read many of the papers in Autobox as well as the links for the PSU time series course, but I still do not understand why (or if) a linear regression (using OLS), with the use of lagged variables and lagged error terms if necessary wouldn't work – Miguel M. Dec 31 '15 at 20:05
• @IrishStat is it the OLS method that doesn't work? – Miguel M. Dec 31 '15 at 20:06
• IrishStat to expand upon your point, the goal would be Granger causality. For example, even if a coefficient is statistically significant, it may not be necessarily significant in improving the forecast accuracy. In my research, I have found that the regression models (linear, lasso, etc), tend to say that things are important than they actually are, while the random forest tends to downgrade them and identify the true levers. Also, random forest has the same out of sample accuracy as the linear models. The only drawback is that you can't tell what the coefficients actually are. – Hidden Markov Model Jan 1 '16 at 4:04
• @MiguelM. It certainly could work because a Transfer Function is a Polynomial Distributed Lag model perhaps including empirically detected level shifts/time trends/seasonal pulses while adjusting for pulses (one time anomalies) I think the primary difference is the path to identification and model revision strategies – IrishStat Jan 1 '16 at 17:17

In think the deepest difference between transfer functions and multipe linear regression (in its usual use) lies in their objectives, multiple regressions is oriented to find the main causal observable determinants of the dependent variable while transfer functions just want to forecasts the effect on a dependent variable of the variation of a specific exogenous variable...In summary, multiple regression is oriented to exhaustive explanation and transfer function to forecasting very specific effects...

• I don't think this is quite accurate, because both methods yield coefficients that can in fact be interpreted. Also, transfer functions DO rely heavily on causal analysis, and are actually better at distinguishing such than multiple linear regression. Also, this post asks for the mechanical/methodological differences between such two methods – Miguel M. Dec 21 '17 at 17:40