KFold Cross Validation with Train/Test/Validation-Set I want to do KFold Cross Validation on a specific model and I am wondering what data to use.
In my project I have got a Train, Test and Validation set (this was already provided).
Now I want to to Cross Validation.
My approach would be to concat the Train and Validation and split it into K Folds. Is this the right approach or do I also have to add the Test data before splitting?
Thanks in advance!
 A: K-fold cross validation is an alternative to a fixed validation set. It does not affect the need for a separate held-out test set (as in, you will still need the test set if you needed it before). So indeed, the data would be split into training and test set, and cross-validation is performed on folds of the training set. If you already have a validation set, you can add it to the training set. For more information, have a look at this related question.
A: Whether or not you keep the test set separate depends on what you want to do with the final model you build after running the K-fold cross validation.
If you use all the training/validation/test data in this process, you have no independent data to use for any further testing/evaluation. If you are happy with that, then use all the training/validation/test data in the cross-validation process; and use all the data to build your final model. In many situations this is fine, as you have the results of the cross-validation to assess model performance.
In some situations this may not be appropriate. For example, you may need to show that your final model performs as well as the cross validation results indicate it should. If that's the case, then if you don't use your test set for cross validation, you can use that to test your final model. The trade-off though, is this method may result in slightly lower-performing model, as you have used less data to train it.
