2
$\begingroup$

I'm working with demand forecasting and we are using Statsmodels SARIMAX on a regular basis. One thing we have noted is that the model doesn't seem to listen much at all to the EXOG argument to which we pass a seasonal decompose.

I therefore made a test with deterministic synthetic data, where I pass the target variable and a small degree of random noise as exog, and make percent offsets per month every year to see if the model listens to this "seasonality" passed as exog.

My conclusion so far is that when having extremely low noise ("uncertainty" in exog), the model listens, but when adding a reasonable level of noise/uncertainty, the model only very carefully nudges the forecast in the right direction.

Am I misunderstanding how to use Exog, or how the optimization works under the hood? Do you have any ideas on how to get it working better? (Our current approach has been to blend it with a long term model that listens better to the seasonality and weight the blending with days ahead in the forecast.)

Python code to reproduce

import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import numpy as np
plt.rcParams.update({'figure.figsize':(20,4), 'figure.dpi':120})


def generate_data( 
    weekday_offset=False,
    month_offset=False,
    freq='d',
    start='2017-12-01 00:00',
    stop='2021-01-07 23:00',
    start_value=100,
    label='target'
):

    start = pd.to_datetime(start)
    stop = pd.to_datetime(stop)

    # Create placeholder df
    df = pd.DataFrame(index=[start, stop])
    df = df.asfreq(freq)

    df[label] = start_value

    # Offset day of week
    if weekday_offset:
        weekday_offset_dict = {
            0: 1.2,
            1: 1.1,
            2: 1.05,
            3: 1,
            4: 1,
            5: 0.90,
            6: 0.90
        }
        for weekday in weekday_offset_dict.keys():
            df.loc[df.index.weekday == weekday, label] = (
                    df.loc[df.index.weekday == weekday, label] * weekday_offset_dict[weekday])

    # Offset by month
    if month_offset:
        month_offset_dict = {
            0: 1.15,
            1: 1.2,
            2: 1.15,
            3: 1.1,
            4: 1.05,
            5: 1,
            6: 0.95,
            7: 0.8,
            8: 0.85,
            9: 0.9,
            10: 1,
            11: 1.05,
            12: 1.1
        }
        for month in month_offset_dict.keys():
            df.loc[df.index.month == month, label] = (
                    df.loc[df.index.month == month, label] * month_offset_dict[month])


    return df


# Generate synthetic data
df = generate_data(weekday_offset='True', month_offset=True, freq='d')[['target']]

# Add some noise to target column, this col with noise will be used as exog
df['noise'] = np.random.normal(
    loc=0, 
    scale=2, # <-- TRY NOISE LEVELS HERE
    size=len(df)
)  
df['target_noise'] = df['target'] + df['noise']
df = df.drop(columns=['noise'])

# Plot noise comparison
df[['target', 'target_noise']].plot(title='Noise to actual comparison')

# Simple train test split
split = pd.to_datetime('2020-06-15')
train = df.loc[df.index<split]
test = df.loc[df.index>=split]

# Define and fit model including exog
params = {'order': (1, 0, 1), 'seasonal_order': (0, 1, 1, 7)}
m = sm.tsa.statespace.SARIMAX(train[['target']], exog=train[['target_noise']], **params)
model = m.fit(disp=0)

# Define and fit model excluding exog
m2 = sm.tsa.statespace.SARIMAX(train[['target']], **params) 
model_no_exog = m2.fit(disp=0)

# Define timeframe for forecast
start = split
days_ahead = 100
end = start + pd.Timedelta(days=days_ahead)

# Slice exog for forecast
exog = test.loc[start:end][['target_noise']]

# Make forecasts with and without exog
pred_exog = pd.DataFrame(columns=['exog'], data = model.predict(start, end, dynamic=True, exog=exog))
pred_no_exog = pd.DataFrame(columns=['no_exog'], data = model_no_exog.predict(start, end, dynamic=True))

# Compare them in plot
compare = pd.concat([pred_exog, pred_no_exog, test], axis=1, join='inner')
compare['exog'] = compare['exog'] + 1 # Visibility dummy for plot
compare[['exog', 'no_exog', 'target']].plot(title='Check if listens to exog')
plt.ylim([60, 125])
plt.show()

enter image description here

$\endgroup$

2 Answers 2

0
$\begingroup$

The basic issue here here is that your left-hand-side variable, df['target'] is peculiar in a number of ways, and in particular it has no noise. So if you look at the 7-period difference, you get:

left hand side

With the exception of a relatively few periods, this is very easy to predict (it's equal to zero).

If I add in the the 7-period difference of df['target_noise'], it looks like:

left and right hand sides

The orange line here is not helpful at predicting most of the periods (which are zeros), although it is somewhat helpful at predicting the periods with jumps.

When you include 'seasonal_order': (0, 1, 1, 7), you are essentially allowing the model to work with the 7-period difference. As a result, it simply is not using df['target_noise'] because it doesn't need to.

If you drop the seasonal order, you'll see that the model uses df['target_noise'] to make its predictions.

$\endgroup$
3
  • $\begingroup$ Thanks for taking the time Chad, that clarifies a lot! : - ) I hadn't realised that the differencing is applied to exog as well. $\endgroup$ Commented Apr 20, 2021 at 7:51
  • $\begingroup$ I can also note that after recalculating our seasonal decompose of the seasonality in the real data, making it multiplicative in absolute numbers and using a trend that we extrapolate into the future as well, then Exog starts making more sense and the model listens a lot to it. So seems the question is solved! :-) $\endgroup$ Commented Apr 27, 2021 at 12:41
  • $\begingroup$ @ml_enthusiast The differencing is applied to the exogenous regressors as well because statsmodels' "SARIMAX" is not a SARIMAX model at all, it's a "regression with SARIMA errors". In an actual SARIMAX model, no such transformation would be applied to the exogenous regressors. $\endgroup$
    – Chris Haug
    Commented Nov 20, 2021 at 17:10
0
$\begingroup$

You could find useful the following link "Comparing trends and exogenous variables in SARIMAX, ARIMA and AutoReg"

The article addresses details on the statsmodels implementation of SARIMAX and ARIMA models, specifically when using exog variables, there are code snippets and interesting sections as how "Reconstructing residuals, fitted values and forecasts in SARIMAX and ARIMA"

https://www.statsmodels.org/dev/examples/notebooks/generated/statespace_sarimax_faq.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.