Auto-regression versus linear regression of x(t)-with-t for modelling time series

Question

What difference precisely does autoregression (for AR(p), p=1,2,...) have when compared to linear regression of that time series random variable w.r.t time axis? Explanation with diagrams clarifying the practical and conceptual differences would be very much appreciated. How does the variable being stochastic make any difference? Why can't we use regular Machine Learning techniques for time series?

Answer 1

Auto-regressive models (ARIMA) use previous values as predictors depending upon the form of the model and forecasts are adaptive in form generally responding to previous values. Models using time as a predictor can be understood as using previous values to estimate the model parameters (thus previous values do come into play ) but they are otherwise not part of the forecast equation thus being generally non-adaptive or fixed until re-estimation occurs. Models using time or time-squared or time-cubed etc. are anachronistic and generally not used/preferred except in very simple textbooks and in very simple classroom exercises. Models using time variables will generally exhibit auto-correlated residuals thus should be studiously avoided as the presumed model. However my work usually includes/investigates both procedures as tentative/possible approaches since only the data knows which approach is better or which approach delivers a more efficient model.

Response to comment @Veneeth :

I didn't say less accurate I wrote (implied) different. A time based model predicts based upon the input variable/series 1,2,3,3,...t which means that the prediction for t+2 ,t+ 3 , t+ 4 is fixed or deterministic or unchanged because when you observe y(t+1) as it was before you observed y(t+1). The new value has no effect on the prediction if you don't re-estimate parameters while a model that uses the value of y(t+1) et. al. and is ARIMA based will provide different forecasts. If you use the time predictor approach and re-estimate with y(t+1) in addition to all the previous y's the impact of the new observation will be normally minimal on the model coefficients unless the sample size is very small or the new observation is an anomaly which should be identified and neutralized.

Since @Veneeth asked for a quantitative example , I attempt here to answer that. With apologies to Charles Dickens one could entitle this as " A tale of three approaches" I selected a real world example not a trivial textbook example which emphasizes the impact of presumption when it comes to model identification . Consider 1) The time based model (the only non-automatic run ) . Here is the actual fit and forecast with equation and residual plot . Followed by 2) The ARIMA model . Now consider a hybrid model incorporating both deterministic structure (input series) and ARIMA . The variance of the errors from each of the three models reduced dramatically. The deterministic structure that was identified in the hybrid approach was a Level/Step Shift which reflects an intercept change. Visually one could make a case for a possible two-trended model using approach 1 yielding but no no avail

Answer 2

My response is more from a practical perspective. I'm specifically going to address your second part of question: why can't we use machine learning techniques for time series?

Reason #1: there is NO empirical evidence that machine learning are known to be superior than simple statistical time series models. Why bother with machine learning when there is no evidence present that it works in time series data?. I once read an article by the editor of the international journal of forecasting (a leading journal in time series forecasting) who said they very rarely if at all publish machine learning methods because machine learning can't even show superiority to naive methods like simple exponential smoothing.

Reason #2: I read the book, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics) by Hastie et, al which is considered a bible for modern machine learning. If you look hard at the index the number of times "time series/time" is mentioned guess what, it is 0(zero). Same can be said for other machine learning books, I have very rarely come across time series in a machine learning context.

Statisticians have elegantly handled time in methods that they have developed, I doubt if there exist similar methods in machine learning, may be its in infancy/ more research needed.

I'm yet to see any empirical evidence that shows any machine learning techniques have superiority of traditional methods, may be someone can share it.