0
$\begingroup$

I use SARIMAX from statsmodels.tsa.statespace.sarimax in Python.
I have a simple one column energy consumption dataset with 27679 rows. The frequency is Hour.

I do hyperparameters optimization thanks to Gridsearch and Hyperopt.
To do so, I split NOT RANDOMLY (I used shuffle=False) the dataset into X_train and X_test datasets, and then I split X_train a second time into X_train_subset and X_valid set so I can do the optimization over the X_valid set, and I never use the X_test set except at the end.

from sklearn.model_selection import train_test_split

# We want to forecast on 72 hours, 3 days
test_size = 72/len(result_data)

X_train, X_test = train_test_split(result_data, test_size=test_size, shuffle=False)
X_train_subset, X_valid = train_test_split(X_train, test_size=0.3, shuffle=False)

The problem is that, sometimes, I got the best hyperparameters which results from the optimization (fitting on X_train_subset & calculating metrics over X_valid) and then, when I fit the model with these hyperparams on X_train, it won't fit, I got :

ValueError: Non-stationary starting autoregressive parameters found with `enforce_stationarity` set to True.

To resume, for one Sarima model with one combination of hyperparams:

  • It fits on X_train_subset.
  • It does not fit on X_train.
  • X_train_subset is 70% of X_train.
  • Both X_train and X_train_subset are stationnary (Dickey Fuller test < 10e-11)

Does someone knows why I got this behaviour and what it means about my model ?

$\endgroup$

closed as off-topic by Michael Chernick, mkt - Reinstate Monica, kjetil b halvorsen, mdewey, Siong Thye Goh Aug 13 at 8:28

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, mkt - Reinstate Monica, kjetil b halvorsen, mdewey, Siong Thye Goh
If this question can be reworded to fit the rules in the help center, please edit the question.

1
$\begingroup$

From your code, it looks like you may be using random training/validation splits to fit models. This is OK for data sets where each observation is statistically independent, but it's not appropriate for time series analysis.

Because there's autocorrelation in time series data, there's a lot of similarity between adjacent time points. If you split the data randomly into training/validation sets, there will be points that contain very similar information in both the training and validation sets. For example, y100 is probably very similar to y101, so when y100 is in your training set and y101 in the validation set, this point is no longer acting as an independent validation of your model.

I recommend you use a rolling time window cross-validation instead to tune your model. There's some more description of this here: https://otexts.com/fpp2/accuracy.html

$\endgroup$
  • $\begingroup$ thank you for your answer, but it is not random splits, I used shuffle=False $\endgroup$ – Laure Decaudin Aug 7 at 8:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.