Hold-Out VS Cross-Validation - R caret

Question

I have a question regarding hold-out vs. cross-validation. I have a dataset with ~650 cases which I am analyzing in R using the caret package. There I have a regression problem and a classification problem.

First I created a training and test dataset with an 80:20 split and then I used cross-validation (10-fold, 3 repeats) on the training dataset to fit the final model. Then I tested the model on the test data set.

The question now is,

• what do I specify as RMSE and AUC ROC?
• The results of the fitted model on the training data set or on the test data set?

I have the feeling that the results in the test data set are very strongly dependent on the coincidence of the split in the training and test data set.

• Also, would it be okay to have only one training dataset?
• If so, how would one plot an AUC ROC and a bland-altman?

I'm a little confused that even though you have cross-validation, you should still have an additional independent test dataset. In principle one has then so to speak a training, validation and test dataset.

• Wouldn't a pure training and validation dataset be possible?

Answer 1

When the test performance relies too much on the random split, it's good practice to do nested cross-validation for test set performance. But, with this method, you won't end up with a champion model but an estimate of real data performance when you apply your training strategy.

The overall performance, e.g. RMSE or AUC, is always calculated on the test set for the final evaluation, and that's what you'll try to stabilize by nested CV.

Answer 2

Thanks for the responses so far! That has helped me a lot. If I take a nested CV for the training and then evaluate it on a hold-out dataset, the AUC ROC still fluctuates massively!

I have shown this once here:

           [,1]
 [1,] 0.8489011
 [2,] 0.8401598
 [3,] 0.7405095
 [4,] 0.8031968
 [5,] 0.7604895
 [6,] 0.8653846
 [7,] 0.8231768
 [8,] 0.8551449
 [9,] 0.8146853
[10,] 0.8381618

Thanks for your further help!

Code:

ctrl <- trainControl(method="cv", number=10, repeats=5, 
                                 classProbs = TRUE, summaryFunction = prSummary, search="random")
v <- c()
for (i in seq(1,10)) {
cfPartition <- createDataPartition(
  y = dsCf$VALUE,
  p = .80,
  list = FALSE
)
dsTrainCf <- dsCf[ cfPartition,]
dsTestCf  <- dsCf[-cfPartition,]
mGLMCf <- train($VALUE,~., data=dsTrainCf, method="glm", family = "binomial", 
                trControl=ctrl,  
                metric="AUC", preProcess = c("center","scale"))
rs <- data.frame(obs = dsTestCf$$VALUE,
                 pred = predict(mGLMCf,newdata=dsTestCf,type="raw"),
                 prob = predict(mGLMCf,newdata=dsTestCf,type="prob"))
roc.GLMCf <- roc(rs$obs,rs$prob.prcbYES)
v  <- rbind(v,as.numeric(roc.GLMCf$auc))
}
print(v)
mean(v)
sd(v)