I have time series data recorded at multiple locations, stored in a matrix $Y$. I have fit a Vector Autoregressive Model to it which forecasts the data pretty well on a test set. However, if I plot the residuals there is definitely some time series structure left; ideally the residuals would look like noise.

I have been learning about boosted trees (package 'gbm' in R) that fits a response vector $Y$ to some input features $X$, pulls the residuals $E = Y - f(X)$, and then fits $E$ to the same input features $X$ again (repeat until the process starts to overfit). It's my belief that a tree won't be able to model time series structure well, so I am wondering if the same process can be done using an AR model as the weak learner.

After fitting my VAR model, I take the residuals and fit an AR process to it, and forecast it using the 'forecast' package in R. I then take my VAR forecasts and add them to my AR forecasts to get a final prediction. The error at some locations decrease, but for most locations the error increases.

Below are the plots of the residuals having only fit the VAR model - there is definitely some structure that can still be explained using some model. Any suggestions?

Update: The time series represent energy consumption at various locations. During the night we expect energy consumption to be very low and flat, during the day there may be high volatility so the variance is definitely not constant. My first step was to remove the daily seasonal component using STL (Seasonal Trend Decomposition Using Loess). After that, I fit my VAR model, and subsequently another AR model to the residuals. The time series after removing the Seasonal Component look like this:

enter image description here

The error terms from the VAR look like

VAR Errors

  • $\begingroup$ Can you show us the original time series? It looks like there might be an increase in variability and perhaps some other nonstationary effect. But it could be some short term blip due to an intervention also. It would help to put this into context. What do these series represent? Is there something that you would expect would happen? $\endgroup$ Commented Jun 10, 2012 at 12:16

2 Answers 2


You can have a look at the paper "Boosting multi-step autoregressive forecasts (link not working)" by Souhaib Ben Taieb and Rob J. Hyndman. Or directly at Hyndmans website Boosting multi-step autoregressive forecasts.

The main idea is to boost a traditional autoregressive (linear) model using a gradient boosting approach.


You probably need to include some nonlinear regressors into your AR formulation. Thus instead of regressing the vector $x_{t+1}$ against some $x_{t-i}$ ($i=0,1,...$) only (which gives standard AR), regress against $x_{t-i}$ and $x_{t-i}^2$ ($i=0,1,...$) or even $x_{t-i}x_{t-j}$ ($i\le j$). To get the most useful result you probably need to use subset selection to identify the most useful nonlinear terms. (All operations on vectors pointwise, but you may also consider products of different components.)

  • 4
    $\begingroup$ Without further elaboration, this is better placed as a comment. $\endgroup$
    – cardinal
    Commented Jun 10, 2012 at 16:12
  • $\begingroup$ @cardinal: I added details $\endgroup$ Commented Jun 11, 2012 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.