1
$\begingroup$

I am looking for advice on the best practice to determine hyperparameters for my LSTM model. I have time series data that I have divided into train and test sets. I was planning to use an expanding walk forward cross validation scheme on my train set to determine the hyperparameters and then use the test to obtain a final evaluation score. However, I am confused as to how to use the early stopping with this setup. Should I:

  1. At each CV iteration use the test fold as a validation set to determine how many epochs to train for with early stopping? This potentially will skew the CV score based on how much the model overfits the test fold at each CV iteration.
  2. Just preset the number of epochs for each fold and not use early stopping? - due to the different number of data points in each fold, this may not make sense.

Also if I do use CV, how do I determine my early stopping rounds for the final model (without using the test set)?

Would I be better off just dividing my data into a train/val/test split and tuning both hyperparameters and early stopping on the validation set?

Thank you!

$\endgroup$

1 Answer 1

1
$\begingroup$

I think it's better if you do train/val/test for each cross-validation fold. The validation set is used only for early stopping and the test set is used for just performance evaluation for the chosen hyper parameter configuration.

So, you do your usual time series cross-validation, and while doing so, separate the train set into two i.e. (train/val again, e.g. 80/20) and use the val only for early-stopping.

$\endgroup$
2
  • $\begingroup$ That's a great idea - so for the final model then, I again have to have a train/val/test split to determine the number of rounds? $\endgroup$
    – Merry
    Jun 1 at 21:36
  • $\begingroup$ For final evaluation, also yes, because early stopping should be decided based on the validation. $\endgroup$
    – gunes
    Jun 2 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.