Am I implementing repeated k fold cross validation correctly?

Question

I have seen examples whereby the repeated k fold cross-validation is applied to the whole data set and then in others only to a subset of the data ( training).

In my analysis, I have applied this to the training subset of my data as follows;

 data_spliter <- initial_split(withresiduals.new, prop = .7, strata=Disease)
 mod_training <- training(data_spliter)
 mod_testing  <- testing(data_spliter)

     rf.fit <- train(Disease ~ ., 
                 data = mod_training, 
                 method = "rf",     
                 importance = TRUE, 
                 trControl = trainControl(method = "repeatedcv", 
                                     number = 10, 
                                     repeats = 5))

 ##testing 

  rf.predict<-predict(rf.fit , mod_testing)
  confusionMatrix(rf.predict, mod_testing$Disease, positive="UC")

I am uncertain of two things;

In this web-page they run the repeated k fold cross-validation on the whole data set and get the optimum mtry as 2. When splitting this data into a training set and running the repeated k fold cross validation with this the optimum mtry is 3.

https://machinelearningmastery.com/how-to-estimate-model-accuracy-in-r-using-the-caret-package/

I understand that with the 10 fold cross-validation the training data set is being split into 10 parts (10 folds) and each time the algorithm is run, it will be trained on 90% of the data and tested on 10%.

I presumed that it was correct to apply the cross-validation to the training subset of my data prior to predictions and not the whole data set ( if 10 fold cv, it will be trained on 90 % of my training data and then tested on 10% of my training data)?

When a researcher asked a similar question on research gate whether this should be applied to the training data set, the reply from one academic a few years ago was "Cross-validation performs training, validation, and testing. So just adopt its result". Please can anybody elaborate on this?

Answer 1

Your implementation is ok. I cannot see on the webpage you referred to, where the difference in mtry comes from. I will not be surprised if you end up with slightly different mtry if you train on full data or just on a training part of the data like you did. Most likely if you check the difference in accuracy / ROC or whichever metric used, the difference is small.

I hope it helps to think like this. Normally, after obtaining the optimal hyperparameter, you would fit the model on the whole training set, and check its performance on the test. The test dataset is not used at all in the training, so it's good to see whether your model overfits or underfits, or has predictive power.

This becomes useful when you want to compare between different models, for example if you want to see whether svm performs better than random forest for your data above.