I'm a beginner at machine learning and have a question regarding time series. I have a data set dependent over time, with a single feature and I am trying to predict the future value of this. This far I have used what I think is a sliding window. I model my data as:

W Value
1 10
2 40
3 60
4 80
5 110


W-2 W-1 W
10  40  60
40  60  80
60  80  110

From what I understand, this is the Sliding window with a lag of 2. Now I use this data and apply Linear Regression or LinearSVR, with W as my y-label and W-2, W-1 as my z-label. The question I ask is whether this is regarded as an Autoregressive model or am I missing something?

My reasoning behind this is that linear regression says: $$ Y= \beta_0 + \beta_1 x_1 + ... + \beta_n x_n + \epsilon $$ and applying my features gives: $$ Y = \beta_0 + \beta_1 x_{1} + \beta_2 x_{2} + \epsilon$$ (where $Y(t)=w(t)$ and $x_i(t)=w(t-i), i=1,2$, while an autoregressive model of order $n$ is formulated: $$ w_t = \beta_0 + \beta_1 w_{t-1} + ... + \beta_{n} x_{w-n} + \epsilon $$ in this case, for an AR(2): $$ w_t = \beta_0 + \beta_1 x_{t-1} + \beta_2 x_{t-2} + \epsilon$$ Is this assumption correct or is there something wrong with my reasoning?

TL;DR Is an autoregression the same as this sliding window linear regression?

  • $\begingroup$ I slightly edited your mathematics for consistency (of various kinds), but you should feel free to alter it if you feel it's necessary. $\endgroup$ – Glen_b -Reinstate Monica May 2 '17 at 12:11

Here's a conceptual description that might help you understand the difference between AR and MA.

In the AR world, the next value in a time series is a direct function of the last known values. i.e forecast = f(t1,t2,t3)

In the MA world, the next value in a time series is a function of the errors you've made in prediction in the previous instances. i.e. forecast = f(e1,e2,e3) , where e = forecast - actual


If you pad with zeros before you regress, then in general this could bias your results (typically time series don't do us the courtesy of being conveniently near 0). This would not usually be regarded as suitable -- but when it is, it would be a form of AR, but not any of the usual ones.

If instead you omitted the first two rows (the ones you padded) so that there's no extra zero's brought in, then: if you regress $w_t$ on $w_{t-1}$ and $w_{t-2}$ then that is effectively an autoregressive model, but the estimation differs from the usual way the AR(2) model is estimated in the way that the first few observations are treated. Specifically the regression method effectively conditions on the first two observations (using what's called conditional least squares estimation), while the more usual approaches to estimating AR(2) doesn't condition on those first few observations -- it incorporates them into the estimation as well (there are several different estimation methods that don't do this conditioning, of which maximum likelihood estimation would be the most widely used nowadays).

Consider for example, an AR(1) model. The conditional regression is equivalent to writing the Gaussian likelihood for the last n-1 observations (conditional on the first one) and maximizing that. Maximum likelihood estimation would multiply that likelihood by the likelihood of the first observation (which we can write unconditionally - using a regression-style parameterization - it's $w_1 \sim N(\beta_0/(1-\beta_1),\sigma^2/(1-\beta_1^2))$). Similarly likelihood for the first few terms of higher order AR models can be written by only conditioning on the prior observations that are available -- more generally you write the likelihood for $w_1$, then $w_1|w_2$ then $w_3|w_1,w_2$ and so forth until you hit the order of the AR and you're back to the usual terms as you have with errors from the conditional least squares.

In a long series the conditioning in the conditional least squares approach will typically make only a little difference. In a short series it can be more substantial.

There is a form of estimation that does "fill in" data before the first observation (to do something similar to what you attempted but in a more sophisticated way) -- it effectively uses the AR model to "backcast" the pre-sample values and them proceeds with least squares again.

  • $\begingroup$ I think I understand your answer, if I remove the first two rows in my example (where I padd with 0's), the linear regression would be equal to the autoregressive? $\endgroup$ – Math May 2 '17 at 11:20
  • $\begingroup$ No. But the way you asked that makes me realize your question wasn't quite asking what I thought it was. I'll have to modify my answer slightly $\endgroup$ – Glen_b -Reinstate Monica May 2 '17 at 11:22
  • $\begingroup$ @Math now modified. I might add a little more later. $\endgroup$ – Glen_b -Reinstate Monica May 2 '17 at 11:31
  • $\begingroup$ This clarified a bit, I now updated my question to clarify my reasoning a bit. $\endgroup$ – Math May 2 '17 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.