# How to adjust hyper-parameter values of SARIMAX as we move month on month

I am trying to build a SARIMAX forecasting model to forecast availability of technicians across all 50 US states over 12 weeks horizon. I have a seasonal data hence going in with SARIMAX.

Sample data looks like:

            capacity

2024-01-01  83
2024-02-01  86
2024-03-01  86
2024-04-01  85
2024-05-01  78


To understand and get the ideal hyper-parameters for the model, I ran the SARIMAX over a parameter space to get the best combination of hyper-parameters using Python's itertool module's product method for a given state. I used the last 12 weeks data as test set to establish the RMSE and thus get the best hyper-parameters.

The issue I am facing is that when I take the best parameter combinations for Jan to Mar and use the same set to forecast from Dec to Feb or Nov to Jan, the RMSE drops drastically.

This is expected because the seasonal parameters would differ accordingly. i.e. if seasonality occurs 12 months from Jan, going back 1 month to Dec, the same seasonality would be seen 11 months ago.

Because for forecasting we wouldn't be having a test data to measure our RMSE, we need to rely on the best hyper-parameter set from above run.

I am new to SARIMAX forecasting. Could someone please let me know how to adjust the parameter values from month on month considering I am using monthly data.

• By hyperparameters, do you mean lag orders and orders of integration (p, d, q, P, D, Q)? Commented May 7 at 20:03
• @RichardHardy, yes, I am referring to the (p, d, q, P, D, Q) Commented May 8 at 2:44

Suppose you want to forecast for some time point $$t'$$. You could use the data that is immediately prior to $$t'$$ as a train-validation set for tuning hyperparameters. The best-performing hyperparameters on the train-validation set are used for making a forecast at $$t'$$.

The example code below forecasts for each timepoint as follows:

• Randomly sample a set of hyperparameters
• Split the data that comes before $$t'$$ into n_splits timeseries folds
• For each split, train a model with the hyperparameters on the training set, and record the validation score
• Average the score over all splits
• After trialling n_iter sets of hyperparameters as above, choose the hyperparameters that scored best, and use them to forecast for $$t'$$.
• Move onto the next timepoint $$t'+1$$

With this algorithm, you'll find that the optimal hyperparameters may differ from one timepoint to the next, because each timepoint has its own hyperparameter selection (using the data immediately prior to it).

You could swap out the random sampling with a grid search, but the former is more efficient for large search spaces.

## Reproducible example

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

#
# Create a dataset for testing (5 years, monthly)
# Sinusoidal with a yearly period, plus a linear trend and noise
#
np.random.seed(0)
n_samples = 12 * 5
dates = pd.date_range('2019-01-01', periods=n_samples, freq='MS')
t = np.linspace(0, 1, n_samples)
data = (np.sin(t * 2 * np.pi * 3) * 4 + 80) * (t/5 + 1) + np.random.randn(n_samples) * 0.5
data = data.astype(int)

df = pd.DataFrame(data, index=dates, columns=['capacity'])

#View the data
ax = df.plot(
use_index=True, figsize=(9, 2), legend=False,  marker='.',
markeredgecolor='white', markeredgewidth=.6, ylabel='capacity'
)

import statsmodels.tsa.api as tsa
from pandas.tseries.offsets import MonthBegin
from sklearn.model_selection import ParameterGrid, ParameterSampler

#Set some constants
min_train_size = 12 * 2 #use 12*2 months of data before making first forecast
n_splits = 3

#Use the specified information to calculate when the first forecast should be
origin = df.index[0]
first_forecast_start = origin + MonthBegin(min_train_size + n_splits)

param_grid = {
'p': [0, 1, 2],
'd': [0, 1, 2],
'q': [0, 1, 2],

'P': [0, 1, 2],
'D': [0, 1],
'Q': [0, 1, 2],
's': [12]
}

n_iter = 20
param_sampler = ParameterSampler(param_grid, n_iter=1)

forecasts_list = []
for trnval_end in df[first_forecast_start - MonthBegin():].index:

params_list = []
params_scores = []

while len(params_list) < n_iter:
params = next(iter(param_sampler))
p, d, q, P, D, Q, s = [params[k] for k in param_grid]
order = (p, d, q)
seasonal_order = (P, D, Q, s)

#Each param set is evaluated 3 times
error_persplit = []
for trn_end in df[trnval_end - MonthBegin(n_splits):trnval_end - MonthBegin()].index:
results = tsa.ARIMA(
df[origin:trn_end],
order=order,
# seasonal_order=seasonal_order,
enforce_invertibility=False,
enforce_stationarity=False
).fit()

if not results.mle_retvals['converged']:
continue

forecast = results.forecast().item()
y = df.loc[trn_end + MonthBegin()].item()
error_persplit.append(forecast - y)

#Average the CV splits and record
rmse = (np.array(error_persplit)**2).mean()**0.5

params_list.append([order, seasonal_order])
params_scores.append(rmse)

#Choose params which lowest CV rmse
order, seasonal_order = params_list[np.argmin(params_scores)]

forecasts = tsa.ARIMA(
df[origin:trnval_end],
order=order,
# seasonal_order=seasonal_order,
enforce_stationarity=False,
enforce_invertibility=False
).fit().forecast(3) #forecast 3 steps
forecasts_list.append(forecasts)

forecasts_flat = [fcs.iloc[0] for fcs in forecasts_list] + forecasts_list[-1].values[1:].tolist()
forecasts_df = pd.DataFrame(
forecasts_flat,
index=pd.date_range(forecasts_list[0].index[0], forecasts_list[-1].index[-1], freq='MS')
).set_axis(['forecast'], axis=1)

ax = df.plot(
use_index=True, figsize=(9, 2), legend=False,  marker='.',
markeredgecolor='white', markeredgewidth=.6, ylabel='capacity',
label='data'
)
forecasts_df.plot(ax=ax, color='tab:red', linewidth=1, label='forecast')
ax.legend()