I fit a statsmodels.tsa.statespace.sarimax.SARIMAX model (statsmodels==0.8.0) but I'm getting unexpected forecasting behavior, in which the forecast has a negative slope (see last plot at the bottom).

Below are my endogenous and exogenous data, which have hourly sampling frequency. The endogenous variable appears to have 24 hour season.

enter image description here enter image description here

Below is a time series diagnostic plot of the endogenous data. In the top figure, the red line is the rolling mean and the purple line is the rolling std:

enter image description here

After applying a season difference and first difference (e.g. using pandas, endog.diff(24).diff().dropna() I get a diagnostic plot like:

enter image description here

Which lead me to believe SARIMAX(0,1,0)(1,1,1,24) might be appropriate. This is the code I used to instantiate and fit the model:

sarimax = SARIMAX(endog=endog_tr, exog=exog_tr,
                  order=(0,1,0), seasonal_order=(1,1,1,24),
res = sarimax.fit()

Here is the result summary:

                                 Statespace Model Results                                 
Dep. Variable:                              endog   No. Observations:                 6547
Model:             SARIMAX(0, 1, 0)x(1, 1, 1, 24)   Log Likelihood                8437.861
Date:                            Fri, 30 Mar 2018   AIC                         -16867.721
Time:                                    23:53:46   BIC                         -16840.574
Sample:                                01-01-2017   HQIC                        -16858.335
                                     - 09-30-2017                                         
Covariance Type:                              opg                                         
                 coef    std err          z      P>|z|      [0.025      0.975]
exog          -0.0013      0.001     -1.499      0.134      -0.003       0.000
ar.S.L24       0.2415      0.009     26.204      0.000       0.223       0.260
ma.S.L24      -0.9139      0.005   -189.131      0.000      -0.923      -0.904
sigma2         0.0044   4.27e-05    102.700      0.000       0.004       0.004
Ljung-Box (Q):                      290.33   Jarque-Bera (JB):              5668.80
Prob(Q):                              0.00   Prob(JB):                         0.00
Heteroskedasticity (H):               1.51   Skew:                            -0.11
Prob(H) (two-sided):                  0.00   Kurtosis:                         7.56

[1] Covariance matrix calculated using the outer product of gradients (complex-step).

After fitting the SARIMAX model, I did another diagnostic plot on the residuals:

enter image description here

I believe the data looks mostly stationary. There appears to be some seasonal correlations still, but I'm not sure how to get rid of that.

I did some in-sample plotting using res.predict(). The predictions appear to match the endogenous variable (labeled "target") quite well:

enter image description here

Now here's where things go wrong. I want to forecast several days out, but the forecast has an odd downward slope. Here's how I produce the forecast:

preds = res.forecast(exog_test.size, exog=exog_test.values.reshape((-1, 1)))

and here's the resulting plot, along with the ground truth test data:

enter image description here

Does anyone know why this is happening? I'd appreciate any help.


I've added a notebook reproducing my work as well as some sample data:

Edit 2:

I've added more notebooks:

  • $\begingroup$ Did you check that exog_test.values has the correct numbers? The forecast trend is too regular to be driven by fluctuating exog. I don't see any other reason why this should happen. Can you provide a full working example to investigate? (A link to a notebook would be the most convenient.) $\endgroup$
    – Josef
    Apr 1, 2018 at 16:55
  • $\begingroup$ Hi @user333700, thank you for commenting and I appreciate your input. The exog_test values appear to be correct from what I can tell. Good idea, I will produce a notebook as well as the dataset in a flat file. It may take me a day or so due to other obligations. $\endgroup$
    – trianta2
    Apr 1, 2018 at 21:32
  • $\begingroup$ @user333700 I've added a notebook and some sample data $\endgroup$
    – trianta2
    Apr 2, 2018 at 21:41
  • $\begingroup$ I;m having the exact same problem, the predict looks a lot like the target and then the forecast is a straight line :( I just can't understand the difference between predict and forecast, documentation is just awful $\endgroup$
    – Yuca
    Aug 16, 2018 at 23:17

2 Answers 2


You are specifying an I(2) process, so you're specifying that the change in the time series is itself integrated. The forecast for the change of the series is then like a random walk (i.e. it won't die out). This estimate of this change (and so the forecast going forward) is encapsulated by the last estimated state (i.e. when the model is cast in state space form).

Because the change of the series is fixed (either positive or negative depending on the last estimated states), the forecast will trend up or down, regardless of the AR and MA coefficients.

Since the model is seasonal, there will also be a seasonal pattern, but the same general explanation applies to a non-seasonal ARIMA model with d=2.

It seems like you don't need seasonal differencing here - have you considered SARIMAX(0,1,0)(1,0,1,24)?

  • $\begingroup$ Hi @cfulton, I tried SARIMAX(0,1,0)(1,0,1,24) in the "First difference test" notebook I added (see cell In [6]), but the problem remains. The first difference ACF and PACF plots in In [4] made me think SARIMAX(1,1,0)(1,0,0,24) might be appropriate, but the prediction plot in Out[8] shows that the forecast quickly attenuates. $\endgroup$
    – trianta2
    Apr 5, 2018 at 21:44
  • $\begingroup$ Actually, sorry, I think eliminating the non-seasonal differencing is better. I'm getting pretty reasonable results with SARIMAX(1, 0, 0)(1, 1, 1, 24) $\endgroup$
    – cfulton
    Apr 7, 2018 at 2:09
  • $\begingroup$ Indeed, SARIMAX(1, 0, 0)(1, 1, 1, 24) appears to work nicely and has removed the downward slope. Thanks! $\endgroup$
    – trianta2
    Apr 10, 2018 at 15:55

Kudos on your presentation . The problem you are running into maybe due to the model that you are specifying . Hourly data is often quite dependent on daily activity which in turn can often be described (instantiate in your words) by deterministic effects. We have found that a two level approach can be useful Hidden markov model to detect Stock outs in Hourly sales Time series data where 24 hourly models are developed using as a predictor the daily totals.

It appears that your non-significant predictor may be driving the downward forecast but I would have to delve much further into your data to substantiate that reflection . I have also seen the inclusion of a non-significant trend constant can lead to your circumstance.

  • $\begingroup$ Hi @IrishStat, thank you for your answer. I'm less experienced with HMMs. To clarify, are you suggesting that I could use a HMM to predict latent state, and use that state as predictor for the hourly model? If so, thinking along those same lines, if days are correlated, do you think summary statistics of the previous day might help? Regarding your comment on non-significant predictor, I've tried removing the exog variable and the issue persists. I also believe my existing model was specified to not include a trend constant. $\endgroup$
    – trianta2
    Apr 3, 2018 at 14:56
  • $\begingroup$ If summary statistics i.e. the previous day's total is important it will be incorporated as part of the prediction model for the daily totals and subsequently possibly impact the expected value for each hour. See stats.stackexchange.com/questions/162966/… for another discussion $\endgroup$
    – IrishStat
    Apr 3, 2018 at 15:19
  • $\begingroup$ Hi @IrishStat, I wanted to quickly evaluate your proposal of using a daily predictor, so I used the daily means from the dataset directly as a predictor in the hourly model. However, I'm not sure if this is a kosher approach to ceiling analysis in this scenario. The results are in the "Daily predictor test" notebook I added, and unfortunately the daily predictor did not help. $\endgroup$
    – trianta2
    Apr 5, 2018 at 21:51
  • $\begingroup$ I am a little confused and monologues are not really helpful. Did you build 24 individual models ? Perhaps a skype session will clear up things $\endgroup$
    – IrishStat
    Apr 5, 2018 at 23:33
  • $\begingroup$ IrishStat, thank you for your generous offer of a skype session to help debug my issue, however cfulton's suggestion appears to have fixed my problem. I will continue to explore your two-level approach to improve my forecasts. $\endgroup$
    – trianta2
    Apr 10, 2018 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.