I have seen examples whereby the repeated k fold cross-validation is applied to the whole data set and then in others only to a subset of the data ( training).
In my analysis, I have applied this to the training subset of my data as follows;
data_spliter <- initial_split(withresiduals.new, prop = .7, strata=Disease) mod_training <- training(data_spliter) mod_testing <- testing(data_spliter) rf.fit <- train(Disease ~ ., data = mod_training, method = "rf", importance = TRUE, trControl = trainControl(method = "repeatedcv", number = 10, repeats = 5)) ##testing rf.predict<-predict(rf.fit , mod_testing) confusionMatrix(rf.predict, mod_testing$Disease, positive="UC")
I am uncertain of two things;
In this web-page they run the repeated k fold cross-validation on the whole data set and get the optimum mtry as 2. When splitting this data into a training set and running the repeated k fold cross validation with this the optimum mtry is 3.
I understand that with the 10 fold cross-validation the training data set is being split into 10 parts (10 folds) and each time the algorithm is run, it will be trained on 90% of the data and tested on 10%.
I presumed that it was correct to apply the cross-validation to the training subset of my data prior to predictions and not the whole data set ( if 10 fold cv, it will be trained on 90 % of my training data and then tested on 10% of my training data)?
When a researcher asked a similar question on research gate whether this should be applied to the training data set, the reply from one academic a few years ago was "Cross-validation performs training, validation, and testing. So just adopt its result". Please can anybody elaborate on this?