0
$\begingroup$

I need to see whether different k-fold cross-validations with repeats will improve my classification accuracy.

For that, I have tested with repeatedCV (k-fold=50, repeat=10, k-fold=100, repeat=10, and k-fold=1000, repeat=10)

But, accuracy values in the confusion matrix are same for the above different repeated cross-validations.

Please see my code below. I am using caret package for training the data.

I am thinking whether I am getting the same accuracy results because my my model is stable/robust or due to some other reason.

Any help is appreciated.

Thanks in advance.

standardized.X <- LossT[,-c(1:10)]
set.seed(50)
ind <- createDataPartition(LossT$LRPPcat, p=0.85, list = FALSE)
train.X <- standardized.X[ind,]
test.X <- standardized.X[-ind,]
train.Y=LossT$LRPPcat[ind]
test.Y=LossT$LRPPcat[-ind]

set.seed(123)
seeds <- vector(mode = "list", length = 501)
for(i in 1:501) seeds[[i]] <- sample.int(1000, 20)
## For the last model:
seeds[[501]] <- sample.int(1000, 1)

# Train the model
ctrl <- trainControl(method = "repeatedcv",number = 50, repeats = 10, preProcOptions = list(thresh = 0.85, k = 5),seeds = seeds, selectionFunction = "best", savePredictions = TRUE, allowParallel = TRUE) 
modelknn <- train(train.X,train.Y, method = "knn",preProcess = c("pca"), trControl = ctrl,metric = "Accuracy",tuneLength = 20)

fitted.results <- predict(modelknn,test.X) 
cm <- confusionMatrix(data=fitted.results, reference=test.Y) 
$\endgroup$
4
  • 1
    $\begingroup$ I think you have a misunderstanding about what cross validation is for. K is not a tuning parameter in a model, it does not change the complexity of the model being fit to the data. Different k's give you different estimates of the same quantity, for the same model, for the same population. $\endgroup$ Aug 8, 2017 at 14:28
  • $\begingroup$ Thanks for your reply and appreciate it. But, when used "knn" the numbers of nearest neighbors (k) is the tuning parameter. That's why I have put the tuning length there. But, I cannot understand the reason why for different repeatedCV gives me the same classification output results. Any idea? Thanks again. $\endgroup$ Aug 8, 2017 at 17:12
  • $\begingroup$ As Matthew Drury pointed out, the $k$ in CV is not a tuning parameter. Conversely, the $k$ in KNN is. There is no relationship between the two. Perhaps you might want to review a textbook on these matters. $\endgroup$
    – Ami Tavory
    Aug 8, 2017 at 21:18
  • $\begingroup$ @Ami, Thanks for your comment. I totally understand it. Do you think my code needs to be revised? If yes, where should I change? Please let me know. Thanks a lot again. $\endgroup$ Aug 9, 2017 at 14:25

1 Answer 1

1
$\begingroup$

But, accuracy values in the confusion matrix are same for the above different repeated cross-validations.

This is how it should ideally be. The actual choice of the number of folds does not influence the results much, see also Choice of K in K-fold cross-validation

I am thinking whether I am getting the same accuracy results because my my model is stable/robust or due to some other reason.

The aggregated confusion matrix (summing up the results of all surrogate models) is not the best place to characterize stability [of the predictions wrt. small changes in the training set]. You can get that more directly by comparing predictions across the repetitions.

The "other reason" may be that you have many cases (as you do k = 1000), so the random (relative) differences you observe in the final result are typically smaller than those observed with small data sets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.