0
$\begingroup$

I want to do KFold Cross Validation on a specific model and I am wondering what data to use. In my project I have got a Train, Test and Validation set (this was already provided). Now I want to to Cross Validation. My approach would be to concat the Train and Validation and split it into K Folds. Is this the right approach or do I also have to add the Test data before splitting?

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ What is your reason for running cross-validation? To tune hyper-parameters? for model evaluation? or something else? $\endgroup$
    – Lynn
    Jun 1 at 8:06
  • $\begingroup$ I would use it for model evaluation $\endgroup$ Jun 1 at 8:47

2 Answers 2

0
$\begingroup$

K-fold cross validation is an alternative to a fixed validation set. It does not affect the need for a separate held-out test set (as in, you will still need the test set if you needed it before). So indeed, the data would be split into training and test set, and cross-validation is performed on folds of the training set. If you already have a validation set, you can add it to the training set. For more information, have a look at this related question.

$\endgroup$
2
  • $\begingroup$ Thanks! So you think its fine to concat train,test and validation set before doing the K-fold cross validation? $\endgroup$ Jun 1 at 9:41
  • $\begingroup$ You should still keep a separate test set. Using k-fold cross-validation, you will no longer need a separate validation set, but that does not mean you can do without the test set. I do not know your specific case, but having a separate test set is almost always a good idea, irrelevant of your cross-validation procedure. $\endgroup$
    – Scriddie
    Jun 1 at 11:16
0
$\begingroup$

Whether or not you keep the test set separate depends on what you want to do with the final model you build after running the K-fold cross validation.

If you use all the training/validation/test data in this process, you have no independent data to use for any further testing/evaluation. If you are happy with that, then use all the training/validation/test data in the cross-validation process; and use all the data to build your final model. In many situations this is fine, as you have the results of the cross-validation to assess model performance.

In some situations this may not be appropriate. For example, you may need to show that your final model performs as well as the cross validation results indicate it should. If that's the case, then if you don't use your test set for cross validation, you can use that to test your final model. The trade-off though, is this method may result in slightly lower-performing model, as you have used less data to train it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.