0
$\begingroup$

I am analyzing a data set of power consumption with the aim of forecasting. The times when there is consumption are rather sparse. If there is consumption then there is likely one in the next time step too. The result is a significant auto-correlation at lag 1 and as there are seasonal patterns at other lags too.

I would say that the lag 1 auto-correlation is spurious in the sense that it is not causal. Consumption at $t+1$ does not happen because of consumption at $t$ it just happens together. Thus I would not model this relation.

Furthermore forecasting such a relation seems dangerous to me. Can you point me to some reference about such a situation?

EDIT: In practice and rather high frequency data (hourly) one does n-step ahead forecasts. Thus given data up to $t$ we have to forecast up to $t+n$ e.g. $t+24$. If we model an AR coefficient, then we would plug in the most recent value and $23$ subsequent forecasts. Isn't this really dangerous if the correlation is rather spurious?

EDIT 2: This is a plot of a subsample Histogram of the data in the subsample

EDIT 3: a sample of data very similar to my data is available here.

Improve this question
$\endgroup$
7
  • $\begingroup$ Just because a relationship is non-causal doesn't mean it is not useful for forecasting. For example, the sun comes up around 30 min after my alarm clock sounds. My alarm clock obviously does not cause the sun to come up, but I can still use my alarm clock as a 'signal' to predict that the sun will come up soon. The same logic holds for forecasting. In many applications we accept the reality that are models (ARIMA, GARCH, state space, etc) are non-causal or endogenous, but the forecast $y_{t+1}=f(y_1,...,y_t)+u_{t+1}$ can still use statistical dependencies to improve predictive accuracy. $\endgroup$ – Zachary Blumenfeld Jul 24 '15 at 9:27
  • $\begingroup$ There are instances where we do care about non-causality in our time series models. One example is when we are interested not so much in predictive accuracy, but rather interpreting the coefficients being estimated by the model and the "effect" of past innovations on future outcomes. This is a big concern for those studying economic policy. Another example, is when we believe that shifts in confounding variables being neglected by the model can cause the relationship between observed variables to change leading to sub-optimal forecasts. $\endgroup$ – Zachary Blumenfeld Jul 24 '15 at 9:47
  • 1
    $\begingroup$ I am unfamiliar with your data, but energy consumption, for example, depends on factors like whether and production, so incorporating those predictors in the model would likely improve the forecast, at least in the short-term. As far as references go, I don't know anything specific off the top of my head, but doing research on the difference between Granger causality and actual causality would be informative. I know the economist Grayham Mizon does research concerning exogeneity in forecasting and failure of time series with lacking information, but that may be to much for what you want. $\endgroup$ – Zachary Blumenfeld Jul 24 '15 at 10:17
  • $\begingroup$ Thanks for your comments: please see my edit too. The example by the sun can be put this way: it gets warm and your alarm sounds. There is no causality between the clock and warmth but a common factor: the sun. If we model the sun, then we do the right thing ... $\endgroup$ – Ric Jul 24 '15 at 10:18
  • 1
    $\begingroup$ "all models are wrong, but some are useful"-George Box. Point being that in this context, it's not about getting it perfectly right as much as identifying statistical dependencies well enough to offer superior forecasts. After all I am not smart enough to model the Sun and using my clock to predict warmth is usually better than using nothing at all. For your edit, it's hard to imagine a case where a standard AR model is useful for forecasting 24 periods in the future, but in the case that it might be, the danger would be in non-stationarity or a non-ergotic time series. $\endgroup$ – Zachary Blumenfeld Jul 24 '15 at 10:50
1
$\begingroup$

I have had success with integrating hourly dummies into the model which effectively forecasts what is expected at hour j based upon the previous day's hour j value rather than the auto-regressive structure using the value at j-1 etc . This easily extends to incorporating day-of-the-week effects , week-of-the=year effects , monthly effects , pre and post holiday effects , long-weekend effects. This approach often is sufficient but sometimes needs/incorporates an ARMA modification

EDIT UPON RECEIPT OF 835 DAYS OF DATA FOR 24 HOURS:

A very workable solution for this multi-frequency data set (24 hours and 7 days) possibly holiday/event/promo driven is available in a piece of software that I have helped write (AUTOBOX). I will try to offer an honest description of how the solution unfolds. One can not simply apply ARIMA to the entire series (20,040 data points) as the first observation in each day is primarily driven/related to the first observation 1 day ago and the particular day-of-the-week , week-of-the-year, week-of-the-month, month-of-the-year etc while an ARIMA model would falsely use short-term prior hourly values and miss the big picture . Additionally the underlying model for this kind of daily data is primarily deterministic not simply auto-regressive.

The first step is to build a model for the total daily series (GROUPT) enter image description here shown here enter image description here which uses monthly indicators and day-of-the=week indicators and level shifts in conjunction with outlier detection. The second step is to do this for each of the 24 periods using the daily total as possible important predictor reflecting high-level trends. I show here the results for hour10 using monthly dummies , day-of-the-week dummies , step/level shifts AND the GROUPT reflecting macro trends/effects.enter image description here using enter image description here . We now have 24 individual forecast vectors and we then introduce an option to reconcile them with the forecast of GROUPT. I present here two possible reconciliations . Top-down enter image description here and bottom-up enter image description here for the future hourly forecasts.

Notice that there is no spurious auto-correlation induced by the consecutive zeroes in this approach.

MODIFIED TO INCLUDE MODEL RESULTS FOR HOUR 10:

enter image description here

Improve this answer
$\endgroup$
7
  • $\begingroup$ My data points are zero in 70% of the cases and accumulate at some level, say 1000 if there is consumption. I use hour dummies and weekday and month dummies (and holidays) as you say. I could fit a lm on the seasonal dummies and use arma for the errors (similar to what you propose). Just applying a log (1+x ) transform however will not account for the clumbing at zero and 1000. Do you have any experience with such climbing? $\endgroup$ – Ric Jul 24 '15 at 13:38
  • $\begingroup$ I would not "fit a lm" . Why don't you post your data (excel format) and I will try ant take a look at it. $\endgroup$ – IrishStat Jul 24 '15 at 14:00
  • $\begingroup$ I have attached plots of a subperiod of the data and a histogram. Moreover I have attached a link to a sample of data that looks like my data set. I would appreciate any comments on how one could model such data. Thanks! $\endgroup$ – Ric Jul 27 '15 at 11:21
  • $\begingroup$ You posted 3 days . Please post 700 days in a linked excel format (csv file) ; 730 rows (day) ; 24 columns (hour) $\endgroup$ – IrishStat Jul 27 '15 at 11:52
  • 1
    $\begingroup$ @Richard Since 24 individual models are developed , each one using GroupT as a predictor , the measure of accuracy would be by hour . I will expand mu answer to give you the results for hour 10. $\endgroup$ – IrishStat Jul 29 '15 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.