How to handle hyperparameter tuning for LSTM with early stopping?

Question

I am looking for advice on the best practice to determine hyperparameters for my LSTM model. I have time series data that I have divided into train and test sets. I was planning to use an expanding walk forward cross validation scheme on my train set to determine the hyperparameters and then use the test to obtain a final evaluation score. However, I am confused as to how to use the early stopping with this setup. Should I:

1. At each CV iteration use the test fold as a validation set to determine how many epochs to train for with early stopping? This potentially will skew the CV score based on how much the model overfits the test fold at each CV iteration.
2. Just preset the number of epochs for each fold and not use early stopping? - due to the different number of data points in each fold, this may not make sense.

Also if I do use CV, how do I determine my early stopping rounds for the final model (without using the test set)?

Would I be better off just dividing my data into a train/val/test split and tuning both hyperparameters and early stopping on the validation set?

Thank you!

Answer 1

I think it's better if you do train/val/test for each cross-validation fold. The validation set is used only for early stopping and the test set is used for just performance evaluation for the chosen hyper parameter configuration.

So, you do your usual time series cross-validation, and while doing so, separate the train set into two i.e. (train/val again, e.g. 80/20) and use the val only for early-stopping.