Data splitting and cross validation

Question

my question is about splitting data!

I used to split data into training and testing set using caret library in R

library(caret)
data(iris)
inTrain <- createDataPartition(y = iris$Species, p = 0.7, list = F)
training <- iris[inTrain, ]
testing <- iris[-inTrain, ]

but recently I encounter repeated cross validation in caret

library(caret)
control <- trainControl(method="repeatedcv", number = 10, repeats= 3)

so my question is... if I use the repeated cross validation, don't I need to split data into training and testing sets?

or

I split data into training and testing sets first and then use the repeated cross validation on training set?

Answer 1

You want to still split your data in to the respective training and testing data sets.

This is very important as regardless of the cross-validation method you use (k-fold, repeated k-fold, etc.) you ideally want to have an independent data set from the training process. This way you are able to test your final model on data it has never seen before (even though it isn't, strictly speaking, completely independent). This is important to make sure your model is not overfitting the data in training.

Answer 2

In addition to cdeterman's answer, it is not about certain method of validation (whatever cross-validation/bootstrap/train-test), but about how it is used. Basically, when you use a result of some validation procedure to make some decision about the model (setting some hyperparameter, using certain method, performing certain pre-/postprocessing, etc.) it becomes a part of the training and naturally cannot be used to assess it. Consequently, everything must be wrapped into additional validation to get a reliable information about its performance.

What validation is used where is a secondary question, and the answer depends on the size of the set and available computational resources; one can equally well use train/test for optimisation and CV for assessment or a so-called nested CV on both levels.