6
$\begingroup$

What difference precisely does autoregression (for AR(p), p=1,2,...) have when compared to linear regression of that time series random variable w.r.t time axis? Explanation with diagrams clarifying the practical and conceptual differences would be very much appreciated. How does the variable being stochastic make any difference? Why can't we use regular Machine Learning techniques for time series?

$\endgroup$
  • $\begingroup$ What do "ML techniques" stand for: maximum likelihood or machine learning, or yet something else? Also, let me turn your other question around: what similarities do you see between an AR($p$) model and a linear regression with time trend as a regressor? Why are you not comparing AR($p$) to some other model? $\endgroup$ – Richard Hardy Nov 1 '15 at 11:04
  • $\begingroup$ By ML, I mean Machine Learning models. Why not treat time as a coordinate and learn? I compare mainly AR(1) with linear regression only because they are two-variable, linear fitting (but with some difference). $\endgroup$ – Vineeth Bhaskara Nov 1 '15 at 17:19
8
$\begingroup$

Auto-regressive models (ARIMA) use previous values as predictors depending upon the form of the model and forecasts are adaptive in form generally responding to previous values. Models using time as a predictor can be understood as using previous values to estimate the model parameters (thus previous values do come into play ) but they are otherwise not part of the forecast equation thus being generally non-adaptive or fixed until re-estimation occurs. Models using time or time-squared or time-cubed etc. are anachronistic and generally not used/preferred except in very simple textbooks and in very simple classroom exercises. Models using time variables will generally exhibit auto-correlated residuals thus should be studiously avoided as the presumed model. However my work usually includes/investigates both procedures as tentative/possible approaches since only the data knows which approach is better or which approach delivers a more efficient model.

Response to comment @Veneeth :

I didn't say less accurate I wrote (implied) different. A time based model predicts based upon the input variable/series 1,2,3,3,...t which means that the prediction for t+2 ,t+ 3 , t+ 4 is fixed or deterministic or unchanged because when you observe y(t+1) as it was before you observed y(t+1). The new value has no effect on the prediction if you don't re-estimate parameters while a model that uses the value of y(t+1) et. al. and is ARIMA based will provide different forecasts. If you use the time predictor approach and re-estimate with y(t+1) in addition to all the previous y's the impact of the new observation will be normally minimal on the model coefficients unless the sample size is very small or the new observation is an anomaly which should be identified and neutralized.

Since @Veneeth asked for a quantitative example , I attempt here to answer that. With apologies to Charles Dickens one could entitle this as " A tale of three approaches" I selected a real world example not a trivial textbook example which emphasizes the impact of presumption when it comes to model identification . Consider 1) The time based model (the only non-automatic run ) . Here is the actual fit and forecast enter image description here with equation enter image description here and residual plot enter image description here . Followed by 2) The ARIMA model . enter image description here enter image description here enter image description here Now consider a hybrid model incorporating both deterministic structure (input series) and ARIMA enter image description here enter image description here enter image description here . The variance of the errors from each of the three models reduced dramatically. The deterministic structure that was identified in the hybrid approach was a Level/Step Shift which reflects an intercept change. Visually one could make a case for a possible two-trended model using approach 1 yielding but no no avail enter image description hereenter image description here enter image description here

$\endgroup$
  • $\begingroup$ "Models using time as a predictor are deterministic in form where previous values only come into play when estimating the model parameters but are ignored in the forecast" does not describe a linear regression model in which time is a regressor. It is incorrect on both counts: it is neither "deterministic" (because it explicitly models the response probabilistically) and forecasts do depend (linearly) on responses at all previous times, as is shown explicitly by the standard formula $$\hat y_0=x_0\hat\beta=x_0(X^\prime X)^{-1}X^\prime y.$$ $\endgroup$ – whuber Oct 31 '15 at 17:57
  • $\begingroup$ I attempted to modify my answer. If you can aid my explanation please do so . $\endgroup$ – IrishStat Oct 31 '15 at 18:19
  • $\begingroup$ When you use linear regression, aren't you also not "taking into account" all the previous values, and infact the whole of data till that particular moment? At each moment, say, that we update our linear regression (not AR) model using all the available data until that point of time, how would such a model then be less accurate than AR models? Any quantitative/practical example/explanation would be highly appreciated. $\endgroup$ – Vineeth Bhaskara Nov 1 '15 at 17:22
6
$\begingroup$

My response is more from a practical perspective. I'm specifically going to address your second part of question: why can't we use machine learning techniques for time series?

Reason #1: there is NO empirical evidence that machine learning are known to be superior than simple statistical time series models. Why bother with machine learning when there is no evidence present that it works in time series data?. I once read an article by the editor of the international journal of forecasting (a leading journal in time series forecasting) who said they very rarely if at all publish machine learning methods because machine learning can't even show superiority to naive methods like simple exponential smoothing.

Reason #2: I read the book, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics) by Hastie et, al which is considered a bible for modern machine learning. If you look hard at the index the number of times "time series/time" is mentioned guess what, it is 0(zero). Same can be said for other machine learning books, I have very rarely come across time series in a machine learning context.

Statisticians have elegantly handled time in methods that they have developed, I doubt if there exist similar methods in machine learning, may be its in infancy/ more research needed.

I'm yet to see any empirical evidence that shows any machine learning techniques have superiority of traditional methods, may be someone can share it.

$\endgroup$
  • 1
    $\begingroup$ Amen ! This is well said (written ). $\endgroup$ – IrishStat Nov 6 '15 at 10:08
  • $\begingroup$ deepmind.com/blog/wavenet-generative-model-raw-audio $\endgroup$ – CpILL Oct 20 '16 at 11:51
  • $\begingroup$ @forecaster In light of many new developments in machine learning in recent yrs, couldn't disagree more. Ex. recurrent neural networks. $\endgroup$ – horaceT May 20 '17 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.