I am analyzing a data set of power consumption with the aim of forecasting. The times when there is consumption are rather sparse. If there is consumption then there is likely one in the next time step too. The result is a significant auto-correlation at lag 1 and as there are seasonal patterns at other lags too.

I would say that the lag 1 auto-correlation is spurious in the sense that it is not causal. Consumption at $t+1$ does not happen because of consumption at $t$ it just happens together. Thus I would not model this relation.

Furthermore forecasting such a relation seems dangerous to me. Can you point me to some reference about such a situation?

EDIT: In practice and rather high frequency data (hourly) one does n-step ahead forecasts. Thus given data up to $t$ we have to forecast up to $t+n$ e.g. $t+24$. If we model an AR coefficient, then we would plug in the most recent value and $23$ subsequent forecasts. Isn't this really dangerous if the correlation is rather spurious?

EDIT 2: This is a plot of a subsample Histogram of the data in the subsample

EDIT 3: a sample of data very similar to my data is available here.

  • $\begingroup$ Just because a relationship is non-causal doesn't mean it is not useful for forecasting. For example, the sun comes up around 30 min after my alarm clock sounds. My alarm clock obviously does not cause the sun to come up, but I can still use my alarm clock as a 'signal' to predict that the sun will come up soon. The same logic holds for forecasting. In many applications we accept the reality that are models (ARIMA, GARCH, state space, etc) are non-causal or endogenous, but the forecast $y_{t+1}=f(y_1,...,y_t)+u_{t+1}$ can still use statistical dependencies to improve predictive accuracy. $\endgroup$ Commented Jul 24, 2015 at 9:27
  • $\begingroup$ There are instances where we do care about non-causality in our time series models. One example is when we are interested not so much in predictive accuracy, but rather interpreting the coefficients being estimated by the model and the "effect" of past innovations on future outcomes. This is a big concern for those studying economic policy. Another example, is when we believe that shifts in confounding variables being neglected by the model can cause the relationship between observed variables to change leading to sub-optimal forecasts. $\endgroup$ Commented Jul 24, 2015 at 9:47
  • 1
    $\begingroup$ I am unfamiliar with your data, but energy consumption, for example, depends on factors like whether and production, so incorporating those predictors in the model would likely improve the forecast, at least in the short-term. As far as references go, I don't know anything specific off the top of my head, but doing research on the difference between Granger causality and actual causality would be informative. I know the economist Grayham Mizon does research concerning exogeneity in forecasting and failure of time series with lacking information, but that may be to much for what you want. $\endgroup$ Commented Jul 24, 2015 at 10:17
  • $\begingroup$ Thanks for your comments: please see my edit too. The example by the sun can be put this way: it gets warm and your alarm sounds. There is no causality between the clock and warmth but a common factor: the sun. If we model the sun, then we do the right thing ... $\endgroup$
    – Richi W
    Commented Jul 24, 2015 at 10:18
  • 1
    $\begingroup$ "all models are wrong, but some are useful"-George Box. Point being that in this context, it's not about getting it perfectly right as much as identifying statistical dependencies well enough to offer superior forecasts. After all I am not smart enough to model the Sun and using my clock to predict warmth is usually better than using nothing at all. For your edit, it's hard to imagine a case where a standard AR model is useful for forecasting 24 periods in the future, but in the case that it might be, the danger would be in non-stationarity or a non-ergotic time series. $\endgroup$ Commented Jul 24, 2015 at 10:50

1 Answer 1


I have had success with integrating hourly dummies into the model which effectively forecasts what is expected at hour j based upon the previous day's hour j value rather than the auto-regressive structure using the value at j-1 etc . This easily extends to incorporating day-of-the-week effects , week-of-the=year effects , monthly effects , pre and post holiday effects , long-weekend effects. This approach often is sufficient but sometimes needs/incorporates an ARMA modification


A very workable solution for this multi-frequency data set (24 hours and 7 days) possibly holiday/event/promo driven is available in a piece of software that I have helped write (AUTOBOX). I will try to offer an honest description of how the solution unfolds. One can not simply apply ARIMA to the entire series (20,040 data points) as the first observation in each day is primarily driven/related to the first observation 1 day ago and the particular day-of-the-week , week-of-the-year, week-of-the-month, month-of-the-year etc while an ARIMA model would falsely use short-term prior hourly values and miss the big picture . Additionally the underlying model for this kind of daily data is primarily deterministic not simply auto-regressive.

The first step is to build a model for the total daily series (GROUPT) enter image description here shown here enter image description here which uses monthly indicators and day-of-the=week indicators and level shifts in conjunction with outlier detection. The second step is to do this for each of the 24 periods using the daily total as possible important predictor reflecting high-level trends. I show here the results for hour10 using monthly dummies , day-of-the-week dummies , step/level shifts AND the GROUPT reflecting macro trends/effects.enter image description here using enter image description here . We now have 24 individual forecast vectors and we then introduce an option to reconcile them with the forecast of GROUPT. I present here two possible reconciliations . Top-down enter image description here and bottom-up enter image description here for the future hourly forecasts.

Notice that there is no spurious auto-correlation induced by the consecutive zeroes in this approach.


enter image description here

  • $\begingroup$ My data points are zero in 70% of the cases and accumulate at some level, say 1000 if there is consumption. I use hour dummies and weekday and month dummies (and holidays) as you say. I could fit a lm on the seasonal dummies and use arma for the errors (similar to what you propose). Just applying a log (1+x ) transform however will not account for the clumbing at zero and 1000. Do you have any experience with such climbing? $\endgroup$
    – Richi W
    Commented Jul 24, 2015 at 13:38
  • $\begingroup$ I would not "fit a lm" . Why don't you post your data (excel format) and I will try ant take a look at it. $\endgroup$
    – IrishStat
    Commented Jul 24, 2015 at 14:00
  • $\begingroup$ I have attached plots of a subperiod of the data and a histogram. Moreover I have attached a link to a sample of data that looks like my data set. I would appreciate any comments on how one could model such data. Thanks! $\endgroup$
    – Richi W
    Commented Jul 27, 2015 at 11:21
  • $\begingroup$ You posted 3 days . Please post 700 days in a linked excel format (csv file) ; 730 rows (day) ; 24 columns (hour) $\endgroup$
    – IrishStat
    Commented Jul 27, 2015 at 11:52
  • 1
    $\begingroup$ @Richard Since 24 individual models are developed , each one using GroupT as a predictor , the measure of accuracy would be by hour . I will expand mu answer to give you the results for hour 10. $\endgroup$
    – IrishStat
    Commented Jul 29, 2015 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.