I ran a manual gridsearch of SARIMA across several parameters and now I have 7875 rows of scores (RMSE, MAE, MAPE each) from it. These were the parameters (30k+ permutations) I ran a grid search over-

p = [0 to 10]
d = [0,1,2]
q = [0 to 12]
P = [0 to 5]
D = [0,1]
Q = [0,1,2]
S = [0,7]

These are the top 20 rows of the results sorted by RMSE in ascending. Parameters are in the order ((p,d,q),(P,D,Q,S)). How should I go about selecting the best model from this? Should I do it based on the lowest RMSE or should I select a higher one in which the model parameters are not as high?

enter image description here

import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/vyaduvanshi/helper-files/master/df_timeseries.csv')
train_df, test_df, val_df = df[:-60], df[-30:], df[-60:-30]
result_df = pd.read_csv('https://raw.githubusercontent.com/vyaduvanshi/helper-files/master/metrics_timeseries.csv')
result_df = result_df.rename(columns={'Unnamed: 0':'parameters'})

The model is fitted on train_df and predictions made on val_df


enter image description here

  • $\begingroup$ Are your error measures in-sample or out-of-sample? $\endgroup$ Jul 22 at 6:59
  • $\begingroup$ It's out of sample, on a validation set. $\endgroup$ Jul 22 at 7:00

It's good that you are looking at out-of-sample metrics.

That said, as you see, the errors among your top 20 models do not vary a lot. I would assume the differences are not statistically significant. Thus, whether model A that performs better than model B in this holdout exercise will also perform better on new data is a toss-up. In such a situation, I would definitely use a simpler model rather than a more complex one.

Note that the gold standard in automatic ARIMA fitting only uses orders up to (5,2,5), and with good reason. (Any particular reason why you want to roll your own model selection, rather than go with an established tool?)

You could compare models with the same differencing using information criteria, like AIC, AICc or BIC, choose the optimal model in each case and then use your errors to decide which differencing order is the one you want to go with. (Note that you can't compare information criteria between models with different orders of differencing.)

  • $\begingroup$ I wasn't able to find an equivalent of auto.arima in python so I implemented my own gridsearch. As for taking the range as far as 12, it's because in a single order difference, the ACF and PACF plots took that many instances to get within the confidence interval range, thus deciding the AR and MA (I am new to time series so I could be wrong). Could you tell me what a similar gold standard would be for P,D,Q i..e seasonal ARIMA. $\endgroup$ Jul 24 at 15:18
  • 1
    $\begingroup$ Don't look to ACF/PACF plots to select models. They can only help you with ARIMA(p,d,0) and ARIMA(0,d,q) anyway. Especially if you have a long series, they can be outside the confidence bands, but a simpler model is still better. Much better to use information criteria, like AIC, AICc or BIC. For P and Q, I would also go no further than 5, and D no higher than 2. $\endgroup$ Jul 24 at 16:43
  • 1
    $\begingroup$ Take a look at Forecasting: Principles and Practice (3rd ed.) by Athanasopoulos & Hyndman, and the forecast and fable packages for R. This particular wheel has been invented and extremely well understood. The authors of this book and the packages are gurus in the field. Chances are small you can improve systematically on their decades of experience. $\endgroup$ Jul 24 at 16:44
  • $\begingroup$ Thanks for the excellent resource, going through it is going to take me some time. I added AIC and BIC scores for 2 range of parameters (order & seasonal order). And the result seems .. rather odd to me as the score jumps from tens to directly ten thousands. I've never had to use IC before so I can't say for sure, but it seems odd. $\endgroup$ Jul 27 at 9:27
  • 1
    $\begingroup$ Note that you cannot use ICs to compare models with different orders of (seasonal or nonseasonal) differencing. auto.arima() uses unit root tests and seasonality tests to determine the orders of differencing, then, for fixed $d$ and $D$, determines the other orders based on ICs. $\endgroup$ Jul 27 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.