2
$\begingroup$

I know that a train-validation-test splits the data into:

  • a training dataset - obviously my "in-sample" data
  • a validation dataset
  • a test data set - obviously my "out-of-sample" data

My question is: Should I refer to the validation dataset as in-sample or out-of-sample data?

If we're using the validation dataset to fine-tune the parameter values, then the model has seen this data before. So I'm thinking it is "in-sample" data. Am I right?

Thanks for your help!

Kitty Kenty.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Generally splits are done like this:

a) Train

b) Test

Generally, the train data is then split in $n$ parts. $n-1$ of them are used for training and remaining $1$ is used for validation. And, this process is repeated until all the $n$ parts become validation sets once.

So, yes, validation data is your in-sample data

Improve this answer
$\endgroup$
1
  • $\begingroup$ What you are describing is k-fold cross-validation, and in that context as well as in regular one-time validation the validation set is out-of-sample data because for each fold, the model wasn't trained on it, only tested. A good in-depth description can be found here: machinelearningmastery.com/k-fold-cross-validation. In the context of hyperparameter tuning, however, you can argue that it is in-sample data because you have seen it and possibly tuned the model to overfit it. This is why we need a third set for testing the final model. $\endgroup$
    – runcoderun
    Commented May 22, 2019 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.