So far I have using this process:

1)split data into training and test

2)do model selection(p,d,q, P,D,Q,etc) using training data(in this case, I used autoarima)

from pmdarima import auto_arima
arima_model=auto_arima(x_train,start_p=1, start_q=1,
                        d=1, test='adf',
                         max_p=10, max_q=10,trace=True,seasonal=True)
  1. Doing the CV : I am using the whole data set, with hyperparameters found in step 2. the starting window is the training set from step 1: I train the model, forecast h steps, calculate metrics, then slide/expand the window(depending on rolling vs sliding), forecast/evaluate. repeat until the end.
from pmdarima.model_selection import RollingForecastCV
from statsmodels.tsa.arima.model  import ARIMA
cv2 = RollingForecastCV(step=1, h=5,initial =window_size)
cv_generator2 = cv2.split(d1)
for i in range(0,iterations):


    model = ARIMA(d1.iloc[a[0]], order=(3,1,3))
    model_fit = model.fit()

    yt_forecasted = model_fit.forecast(steps=5)

    rmse2.append(np.sqrt(mean_squared_error(d1.iloc[a[1]].to_numpy().flatten(), yt_forecasted)))
    mae2.append(mean_absolute_error(d1.iloc[a[1]].to_numpy().flatten(), yt_forecasted))

  1. average the metric somehow( in this case I'm just taking a simple average).

However I have been discussing with Bing Chat and they claim that the model selection with autoarima should be done in each iteration of the loop:"

The auto_arima function is used within the for loop in the example code to select the best ARIMA model for each training set. This allows the model to adapt to changes in the data over time. If the auto_arima function was used before the for loop, it would only select a single ARIMA model based on the first training set. This model may not be the best fit for subsequent training sets.

Using the SARIMAX function within the for loop allows us to refit the selected ARIMA model to each training set. This ensures that the model is updated with the most recent data and can make accurate predictions for each test set."

Is this a valid method, and if so, is it better than the steps I've been using? In fact, are the steps I'm using valid?


2 Answers 2


Is this a valid method, and if so, is it better than the steps I've been using?

Yes the suggestion by Bing Chat to refit the model each time the window is moved along the time series is correct. As new data is observed and added to the complete dataset the SARIMAX model identified in the first window may no longer be the best fitting. So we refit the model (run auto_arima in your case) before forecasting the next h steps.

In fact, are the steps I'm using valid?

Broadly speaking, yes the general idea is correct with the correction you mentioned from Bing Chat.

Sketch of sliding window cross-validation for time series data.

  • Take first n data points as your training dataset. Fit your model and forecast the next h steps. Calculate the error metric (rmse, mae, mape, wape etc.) and record it.
  • Move the window forward one step; adding the next data point to the training dataset and removing the first one. Refit the model and forecast next h steps. Calculate error metric and record it.
  • Repeat until there is no longer h time-points remaining after the end of the training dataset.
  • Calculate the aggregate error metric (mean, median, percentile etc.)
  • $\begingroup$ let me clarify to make sure I wasn't confusing: there's the selection of p,d,q in ARIMA(p,d,q) ; then there's the training of the model ,which I think bing chat is calling refitting the model. So training the modle would be like if we have ARIMA(1,0,1), $x_t$= $\phi_1$ $x_{t-1}$-$\theta_1$$w_{t-1}$+$w_t$, we would be getting the coefficients here in that step. I was only selecting p,d,q once, and getting new coefficients for every window. However, bing chat recommends selecting p,d,q for every window too. $\endgroup$
    – a12345
    Jun 7, 2023 at 22:45
  • $\begingroup$ Yes I agree that you should refit the model, finding the best values for p, d, q, etc. each time. $\endgroup$
    – pmxpp88
    Jun 8, 2023 at 5:51

In Rob Hyndman's blog post, he shows both methods as valid.

Multi-step forecasts with re-estimation An alternative approach is to extend the training data and re-estimate the model at each iteration, before each forecast is computed. This is what I call “time series cross-validation” because it is analogous to leave-one-out cross-validation for cross-sectional data. This time, I will store the forecasts from 1- to 5-steps ahead at each iteration.

h <- 5
train <- window(hsales,end=1989.99)
test <- window(hsales,start=1990)
n <- length(test) - h + 1
fit <- auto.arima(train)
order <- arimaorder(fit)
fcmat <- matrix(0, nrow=n, ncol=h)
for(i in 1:n)
  x <- window(hsales, end=1989.99 + (i-1)/12)
  refit <- Arima(x, order=order[1:3], seasonal=order[4:6])
  fcmat[i,] <- forecast(refit, h=h)$mean

A variation on this also re-selects the model at each iteration. Then the second >line in the loop is replaced with

 refit <- auto.arima(x)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.