3
$\begingroup$

I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and came across: QA1 and QA2

Given I should implement walk-forward splits my depiction of it is: enter image description here

Where each line is a Run followed by obtaining the best model.

  1. How should the best model be decided. More specifically, how is the test data used? do we evaluate at each "Run" with test data? i.e I use checkpoints to save the best epoch based on validation data.

    • Do I use the best one?
    • Do I need to evaluate all the models/epochs against test data and take the best of these?
    • or?

  2. On the final "Run", what is the construct? Do we have no validation or no test data? Train&Val

    • If we don't have validation data, is it possible to run model.fit without validation data?

    Train&Test

    • If we don't have test data, do we not have a final evaluation?
Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
5
$\begingroup$

This is nested cross validation (CV). The test data is used to estimate the error of that run. Then, you average the errors obtained over each run's test data. This completes the outer part of CV. Its purpose is to estimate the real world performance of your procedure.

The validaton data is for tuning the hyper-parameters. In your case, this can be number of epochs, number of hidden dimensions etc. The best performing model on the validation fold is selected to estimate the performance on the test set.

In the final run, where you're going to deploy the model, it's still preferable to have train+validation+test split, so that you can still optimize the hyper-parameters and do a final check on the test data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! Does this mean that this method does not actually improve my model, unless I go in and make changes once I received the overall error? i.e this is just an indicator of performance? Also how am I receiving the error on each run (is it just my own evaluation of my metrics?) $\endgroup$
    – Panda
    Feb 14 at 13:46
  • 1
    $\begingroup$ The model is improved in validation stages (inner loops). But, that's right, Run N does not benefit from the improvements in Run (N-1). The outer loop is an indicator of performance. You're receiving the error at each Run, because each Run is evaluated over a particular test set, marked by red in your graph. $\endgroup$
    – gunes
    Feb 14 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.