I have a question regarding hold-out vs. cross-validation. I have a dataset with ~650 cases which I am analyzing in R using the caret package. There I have a regression problem and a classification problem.

First I created a training and test dataset with an 80:20 split and then I used cross-validation (10-fold, 3 repeats) on the training dataset to fit the final model. Then I tested the model on the test data set.

The question now is,

  • what do I specify as RMSE and AUC ROC?
  • The results of the fitted model on the training data set or on the test data set?

I have the feeling that the results in the test data set are very strongly dependent on the coincidence of the split in the training and test data set.

  • Also, would it be okay to have only one training dataset?
  • If so, how would one plot an AUC ROC and a bland-altman?

I'm a little confused that even though you have cross-validation, you should still have an additional independent test dataset. In principle one has then so to speak a training, validation and test dataset.

  • Wouldn't a pure training and validation dataset be possible?
  • $\begingroup$ I couldn't understand the question you ask in your third paragraph. If you can clarify, I'd be happy to amend my answer. But, the validation set is usually used for HP tuning, and test is for final evaluation. $\endgroup$
    – gunes
    Commented Nov 21, 2021 at 13:45
  • $\begingroup$ With so few cases, a train:test split is not a good idea. Working with the whole data set, then evaluating modeling performance with bootstrapping or repeated cross-validation, provides better precision in modeling and better power for testing. See Frank Harrell's blog post for extended argument on this. $\endgroup$
    – EdM
    Commented Nov 21, 2021 at 19:51
  • $\begingroup$ @EdM Harrell makes a quick but important comment in there that I think will cause a lot of disagreement between statisticians and computer scientists: if the signal-to-noise ratio is high, then data splitting does not require such a gigantic sample size. Harrrell works in low signal-to-noise ratio settings, while a lot of computer scientists work in high signal-to-noise ratio settings like speech recognition, where we have known for thousands of years that human minds have, more or less, solved speech recognition. $\endgroup$
    – Dave
    Commented Nov 22, 2021 at 15:50
  • $\begingroup$ @EdM Thank you very much! I decided to use all the data for a strong internal validation using bootstraping with 500 repeats. That also worked out well. However, how do I show a ROC curve using caret? Unfortunately I cannot use the package rms for my study. Here is the same question, but without reference to the caret package. stats.stackexchange.com/questions/103411/… Thanks a lot for your help! $\endgroup$
    – bckpex
    Commented Nov 30, 2021 at 20:47

2 Answers 2


When the test performance relies too much on the random split, it's good practice to do nested cross-validation for test set performance. But, with this method, you won't end up with a champion model but an estimate of real data performance when you apply your training strategy.

The overall performance, e.g. RMSE or AUC, is always calculated on the test set for the final evaluation, and that's what you'll try to stabilize by nested CV.


Thanks for the responses so far! That has helped me a lot. If I take a nested CV for the training and then evaluate it on a hold-out dataset, the AUC ROC still fluctuates massively!

I have shown this once here:

 [1,] 0.8489011
 [2,] 0.8401598
 [3,] 0.7405095
 [4,] 0.8031968
 [5,] 0.7604895
 [6,] 0.8653846
 [7,] 0.8231768
 [8,] 0.8551449
 [9,] 0.8146853
[10,] 0.8381618

Thanks for your further help!


ctrl <- trainControl(method="cv", number=10, repeats=5, 
                                 classProbs = TRUE, summaryFunction = prSummary, search="random")
v <- c()
for (i in seq(1,10)) {
cfPartition <- createDataPartition(
  y = dsCf$VALUE,
  p = .80,
  list = FALSE
dsTrainCf <- dsCf[ cfPartition,]
dsTestCf  <- dsCf[-cfPartition,]
mGLMCf <- train($VALUE,~., data=dsTrainCf, method="glm", family = "binomial", 
                metric="AUC", preProcess = c("center","scale"))
rs <- data.frame(obs = dsTestCf$$VALUE,
                 pred = predict(mGLMCf,newdata=dsTestCf,type="raw"),
                 prob = predict(mGLMCf,newdata=dsTestCf,type="prob"))
roc.GLMCf <- roc(rs$obs,rs$prob.prcbYES)
v  <- rbind(v,as.numeric(roc.GLMCf$auc))
  • $\begingroup$ I don't see those 10 AUC values as "fluctuating massively." They average to 0.82, with a standard deviation of only 0.04. I think you would be better off using all the data to build the model and evaluating the modeling performance by bootstrapping or repeated cross-validation, but you might in any event be looking for more precise estimates than your limited data set can support. $\endgroup$
    – EdM
    Commented Nov 22, 2021 at 15:57
  • $\begingroup$ Thank you for the answer! I have done that now. However, I would like to do two things: 1.) Create a ROC Curve using my final model. 2.) create a Bland-Altman plot also using my final model. But which data do I take now? All data have been used for the training? Thanks a lot for your help! $\endgroup$
    – bckpex
    Commented Nov 22, 2021 at 16:19
  • $\begingroup$ You show the ROC curve for the full data set. It's not clear what a Bland-Altman plot would show, which is typically used to evaluate two different ways of measuring the same thing. If you developed the model with the full data set and used bootstrapping or cross-validation to evaluate the model-building process, you should also report the results of those evaluations. That's implemented in the validate() function of the R rms package, in which the entire data set is used as a test set for multiple models from bootstrapped samples or CVs. $\endgroup$
    – EdM
    Commented Nov 22, 2021 at 17:03
  • $\begingroup$ Thank you for your answer. However, validate() from the rms package doesn't seem to work with a caret model. Do you have a code example? Thank you very much! $\endgroup$
    – bckpex
    Commented Nov 22, 2021 at 18:34
  • $\begingroup$ You could move the analysis to the rms package, if you're now modeling with the whole data set, and move away from caret. With rms you can do binomial regression with the lrm() function and ordinary least squares with the ols() function. There's a bit of a learning curve (e.g., needing to use datadist() on your data frame, then setting the name of the datadist() object as an option), but model validation and calibration, starting from a model on the full data set, then is almost automatic. $\endgroup$
    – EdM
    Commented Nov 22, 2021 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.