I need to see whether different k-fold cross-validations with repeats will improve my classification accuracy.
For that, I have tested with repeatedCV (
k-fold=50, repeat=10, k-fold=100, repeat=10, and k-fold=1000, repeat=10)
But, accuracy values in the confusion matrix are same for the above different repeated cross-validations.
Please see my code below. I am using caret package for training the data.
I am thinking whether I am getting the same accuracy results because my my model is stable/robust or due to some other reason.
Any help is appreciated.
Thanks in advance.
standardized.X <- LossT[,-c(1:10)] set.seed(50) ind <- createDataPartition(LossT$LRPPcat, p=0.85, list = FALSE) train.X <- standardized.X[ind,] test.X <- standardized.X[-ind,] train.Y=LossT$LRPPcat[ind] test.Y=LossT$LRPPcat[-ind] set.seed(123) seeds <- vector(mode = "list", length = 501) for(i in 1:501) seeds[[i]] <- sample.int(1000, 20) ## For the last model: seeds[] <- sample.int(1000, 1) # Train the model ctrl <- trainControl(method = "repeatedcv",number = 50, repeats = 10, preProcOptions = list(thresh = 0.85, k = 5),seeds = seeds, selectionFunction = "best", savePredictions = TRUE, allowParallel = TRUE) modelknn <- train(train.X,train.Y, method = "knn",preProcess = c("pca"), trControl = ctrl,metric = "Accuracy",tuneLength = 20) fitted.results <- predict(modelknn,test.X) cm <- confusionMatrix(data=fitted.results, reference=test.Y)