I am looking for advice on the best practice to determine hyperparameters for my LSTM model. I have time series data that I have divided into train and test sets. I was planning to use an expanding walk forward cross validation scheme on my train set to determine the hyperparameters and then use the test to obtain a final evaluation score. However, I am confused as to how to use the early stopping with this setup. Should I:
- At each CV iteration use the test fold as a validation set to determine how many epochs to train for with early stopping? This potentially will skew the CV score based on how much the model overfits the test fold at each CV iteration.
- Just preset the number of epochs for each fold and not use early stopping? - due to the different number of data points in each fold, this may not make sense.
Also if I do use CV, how do I determine my early stopping rounds for the final model (without using the test set)?
Would I be better off just dividing my data into a train/val/test split and tuning both hyperparameters and early stopping on the validation set?