When using K-Fold cross validation is it a good or bad idea to split the dataset into two, With 70% (for example) being used for K fold CV and 30% used solely for testing in order to check for over fitting? Or is using a mixture not necessary?
IMHO, mixing k-fold cross validation and hold out rarely makes sense.
- Hold out splitting is subject to the same mistakes that lead to optimistic bias in cross validation, and if that happens it is subject to that same optimistic bias as well.
- The reason to use cross validation instead of hold-out is typically that due to small sample size, hold out estimates of performance are subject to higher variance (random uncertainty).
If you are talking about a situation with data-driven model optimization that uses the cross validation for auto-tuning of hyperparameters, you do need an independent evaluation (verification, validation) of the tuned model.
- This can be done by hold-out: the question then is whether it is acceptable or desirable to have higher random uncertainty on that performance estimate compared to the performance estimates inside the auto-tuning. If the random uncertainty for an estimate based on 30 % of your data is acceptable, go ahead.
- If you need lower random uncertainty for the verification as well, you can use cross valiation also here: this is called nested cross validation.
If your training does not use the cross validation for auto-tuning/data-driven optimization but rather for verification/validation purposes, hold out is not necessary.