Dataset partitioning for k-fold cv

Question

In general, is the size of the validation set for k-fold CV given by n/k and that of the training set given by n(k-1)/k and if so, why? (this is based on the ISLR book)

Alternatively, what are some cases (if any) where we'd also want to split the data into partitions for training, cross-validation and testing?

Answer 1

1. Let's use some numbers to clarify your question. Let's say you have 10,000 data points where you are going to do CV. You want to split data into 5 different folds. Then at it iteration you will have (n/k) = (10,000/5) = 2,000 points as validation set and [n(k-1)]/k = [10,000*(4)]/5 = 8,000 points. Therefore, you will have your data split into five different sets and will use one part as validation and the rest of it as training.

2. You ALWAYS want to split your data between train/test sets to assess performance. Let's says you have a data set again, 50,000 points. You split you data between train and test, 40,000 for training and 10,000 for testing. Now, with your 40,000 training points you would do Cross-Validation, to assess a few different things such as:

   • Which hyper parameters performance best on my data set? That is, I am going to tune my model.
   • Your model has high variance on its metrics? Let's investigate that. But, it can be the case that you have a lot of data, and you dont want to waste time, because training a new model at each iteration of CV is burdensome, then you can just instead of doing k-fold, you use holdout. In holdout, you split your training data set as before, and but never using your original test set.

To conclude, you always split your data into train/test. For CV, it will depend if you want to tune your model, and then you can use also holdout, or Leave One Out.