What difference precisely does autoregression (for AR(p), p=1,2,...) have when compared to linear regression of that time series random variable w.r.t time axis? Explanation with diagrams clarifying the practical and conceptual differences would be very much appreciated. How does the variable being stochastic make any difference? Why can't we use regular Machine Learning techniques for time series?

  • $\begingroup$ What do "ML techniques" stand for: maximum likelihood or machine learning, or yet something else? Also, let me turn your other question around: what similarities do you see between an AR($p$) model and a linear regression with time trend as a regressor? Why are you not comparing AR($p$) to some other model? $\endgroup$ Commented Nov 1, 2015 at 11:04
  • $\begingroup$ By ML, I mean Machine Learning models. Why not treat time as a coordinate and learn? I compare mainly AR(1) with linear regression only because they are two-variable, linear fitting (but with some difference). $\endgroup$ Commented Nov 1, 2015 at 17:19

2 Answers 2


Auto-regressive models (ARIMA) use previous values as predictors depending upon the form of the model and forecasts are adaptive in form generally responding to previous values. Models using time as a predictor can be understood as using previous values to estimate the model parameters (thus previous values do come into play ) but they are otherwise not part of the forecast equation thus being generally non-adaptive or fixed until re-estimation occurs. Models using time or time-squared or time-cubed etc. are anachronistic and generally not used/preferred except in very simple textbooks and in very simple classroom exercises. Models using time variables will generally exhibit auto-correlated residuals thus should be studiously avoided as the presumed model. However my work usually includes/investigates both procedures as tentative/possible approaches since only the data knows which approach is better or which approach delivers a more efficient model.

Response to comment @Veneeth :

I didn't say less accurate I wrote (implied) different. A time based model predicts based upon the input variable/series 1,2,3,3,...t which means that the prediction for t+2 ,t+ 3 , t+ 4 is fixed or deterministic or unchanged because when you observe y(t+1) as it was before you observed y(t+1). The new value has no effect on the prediction if you don't re-estimate parameters while a model that uses the value of y(t+1) et. al. and is ARIMA based will provide different forecasts. If you use the time predictor approach and re-estimate with y(t+1) in addition to all the previous y's the impact of the new observation will be normally minimal on the model coefficients unless the sample size is very small or the new observation is an anomaly which should be identified and neutralized.

Since @Veneeth asked for a quantitative example , I attempt here to answer that. With apologies to Charles Dickens one could entitle this as " A tale of three approaches" I selected a real world example not a trivial textbook example which emphasizes the impact of presumption when it comes to model identification . Consider 1) The time based model (the only non-automatic run ) . Here is the actual fit and forecast enter image description here with equation enter image description here and residual plot enter image description here . Followed by 2) The ARIMA model . enter image description here enter image description here enter image description here Now consider a hybrid model incorporating both deterministic structure (input series) and ARIMA enter image description here enter image description here enter image description here . The variance of the errors from each of the three models reduced dramatically. The deterministic structure that was identified in the hybrid approach was a Level/Step Shift which reflects an intercept change. Visually one could make a case for a possible two-trended model using approach 1 yielding but no no avail enter image description hereenter image description here enter image description here

  • 1
    $\begingroup$ "Models using time as a predictor are deterministic in form where previous values only come into play when estimating the model parameters but are ignored in the forecast" does not describe a linear regression model in which time is a regressor. It is incorrect on both counts: it is neither "deterministic" (because it explicitly models the response probabilistically) and forecasts do depend (linearly) on responses at all previous times, as is shown explicitly by the standard formula $$\hat y_0=x_0\hat\beta=x_0(X^\prime X)^{-1}X^\prime y.$$ $\endgroup$
    – whuber
    Commented Oct 31, 2015 at 17:57
  • $\begingroup$ I attempted to modify my answer. If you can aid my explanation please do so . $\endgroup$
    – IrishStat
    Commented Oct 31, 2015 at 18:19
  • 1
    $\begingroup$ When you use linear regression, aren't you also not "taking into account" all the previous values, and infact the whole of data till that particular moment? At each moment, say, that we update our linear regression (not AR) model using all the available data until that point of time, how would such a model then be less accurate than AR models? Any quantitative/practical example/explanation would be highly appreciated. $\endgroup$ Commented Nov 1, 2015 at 17:22
  • $\begingroup$ Autoregression is a linear regression model that uses lagged variables as its input variables. This is heavily studied in signals and systems as well as system identification in control engineering. It stems from "difference equations" and eventually "stochastic difference equations". "Time dependency" is only a metaphore here. $\endgroup$
    – Ash
    Commented Apr 14, 2023 at 13:13
  • $\begingroup$ "Models using time variables will generally exhibit auto-correlated residuals thus should be studiously avoided as the presumed model." My humble guess is this statement is inaccurate. For example, when a time series is spatiotemporal nature. We can chose the lags to be non-sequential in any AR model. This minimizes the "correlation" just mentioned in the post. One should not use linear correlation-based methods in chosing time lags. In order to find the proper time lags it is necessary to use a method that can extract non-linear correlations. $\endgroup$
    – Ash
    Commented Apr 14, 2023 at 13:27

My response is more from a practical perspective. I'm specifically going to address your second part of question: why can't we use machine learning techniques for time series?

Reason #1: there is NO empirical evidence that machine learning are known to be superior than simple statistical time series models. Why bother with machine learning when there is no evidence present that it works in time series data?. I once read an article by the editor of the international journal of forecasting (a leading journal in time series forecasting) who said they very rarely if at all publish machine learning methods because machine learning can't even show superiority to naive methods like simple exponential smoothing.

Reason #2: I read the book, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics) by Hastie et, al which is considered a bible for modern machine learning. If you look hard at the index the number of times "time series/time" is mentioned guess what, it is 0(zero). Same can be said for other machine learning books, I have very rarely come across time series in a machine learning context.

Statisticians have elegantly handled time in methods that they have developed, I doubt if there exist similar methods in machine learning, may be its in infancy/ more research needed.

I'm yet to see any empirical evidence that shows any machine learning techniques have superiority of traditional methods, may be someone can share it.

  • 1
    $\begingroup$ Amen ! This is well said (written ). $\endgroup$
    – IrishStat
    Commented Nov 6, 2015 at 10:08
  • $\begingroup$ deepmind.com/blog/wavenet-generative-model-raw-audio $\endgroup$
    – CpILL
    Commented Oct 20, 2016 at 11:51
  • 3
    $\begingroup$ @forecaster In light of many new developments in machine learning in recent yrs, couldn't disagree more. Ex. recurrent neural networks. $\endgroup$
    – horaceT
    Commented May 20, 2017 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.