# How do I report results of an internal validation in Caret?

I have the following question. In a machine learning project I have to solve a regression and a classification task. See also: Hold-Out VS Cross-Validation - R caret

For this I have about ~650 cases available.

Since a split in training and test dataset is out of question, because the perfomance of the model would be decided by the randomness of the split, I used all data to train and did a bootstraping with 500 repetitions. This is recommended here: https://hbiostat.org/bbr/md/reg.html#internal-vs--external-model-validation

So far so good. I got RMSE, ROC AUC, sensitivity, specificity and other important parameters for my models.

But how do I report them? And is it correct to simply predicate the model against the training data at the end?

fitCtrlBootRandCf <- trainControl(method="boot",number = 500,
classProbs = TRUE, summaryFunction = twoClassSummary , search="random",
savePred=T)

set.seed(1)
mGLMCf <- train(Y_Class~., data=dsCf, method="glm", family = "binomial",
trControl=fitCtrlCVRandCf,
metric="ROC", preProcess = c("center","scale"))
mGLMCf

rs <- data.frame(obs = dsCf$$Y_Class, pred = predict.train(mGLMCf, type="raw"), prob = predict.train(mGLMCf, type="prob")) roc.GLMCf <- roc(rs$$obs,rs$prob.classYES) roc.GLMCf plot(roc.GLMCf) cm <- confusionMatrix(rs$$obs,rs$$pred) round(cm$$table/sum(cm$$table)*100,1) confusionMatrix(mGLMCf) fitCtrlBootRandRg <- trainControl(method="boot", number=500, search="random", savePredictions = T) mGLMRg <- train(Y_Value~., data=dsRg, method="glm", trControl=fitCtrlBootRandRg, metric="RMSE", preProcess = c("center","scale")) mGLMRg RMSE(dsRg$$Y_Value,predict(mGLMRg,dsRg)) plot(dsRg$$Y_Value,predict(mGLMRg,dsRg))  Output: > fitCtrlBootRandCf <- trainControl(method="boot",number = 500, + classProbs = TRUE, summaryFunction = twoClassSummary , search="random", + savePred=T) > > set.seed(1) > mGLMCf <- train(Y_Class~., data=dsCf, method="glm", family = "binomial", + trControl=fitCtrlCVRandCf, + metric="ROC", preProcess = c("center","scale")) > mGLMCf Generalized Linear Model 657 samples 11 predictor 2 classes: 'classYES', 'classNO' Pre-processing: centered (11), scaled (11) Resampling: Cross-Validated (10 fold, repeated 50 times) Summary of sample sizes: 592, 592, 591, 591, 591, 591, ... Resampling results: ROC Sens Spec 0.8291361 0.6606581 0.82225 > > rs <- data.frame(obs = dsCf$$Y_Class, + pred = predict.train(mGLMCf, type="raw"), + prob = predict.train(mGLMCf, type="prob")) > roc.GLMCf <- roc(rs$$obs,rs$prob.classYES)
Setting levels: control = classYES, case = classNO
Setting direction: controls > cases
> roc.GLMCf

Call:
roc.default(response = rs$$obs, predictor = rs$$prob.classYES)

Data: rs$$prob.classYES in 266 controls (rs$$obs classYES) > 391 cases (rs$$obs classNO). Area under the curve: 0.8419 > plot(roc.GLMCf) > > cm <- confusionMatrix(rs$$obs,rs$$pred) > round(cm$$table/sum(cm$table)*100,1) Reference Prediction classYES classNO classYES 27.4 13.1 classNO 10.4 49.2 > confusionMatrix(mGLMCf) Cross-Validated (10 fold, repeated 50 times) Confusion Matrix (entries are percentual average cell counts across resamples) Reference Prediction classYES classNO classYES 26.7 10.6 classNO 13.7 48.9 Accuracy (average) : 0.7568 > > fitCtrlBootRandRg <- trainControl(method="boot", number=500, + search="random", savePredictions = T) > > mGLMRg <- train(Y_Value~., data=dsRg, method="glm", + trControl=fitCtrlBootRandRg, + metric="RMSE", preProcess = c("center","scale")) > mGLMRg Generalized Linear Model 657 samples 11 predictor Pre-processing: centered (11), scaled (11) Resampling: Bootstrapped (500 reps) Summary of sample sizes: 657, 657, 657, 657, 657, 657, ... Resampling results: RMSE Rsquared MAE 51.5589 0.6570902 39.82216 > RMSE(dsRg$$Y_Value,predict(mGLMRg,dsRg)) [1] 50.10974 > plot(dsRg$$Y_Value,predict(mGLMRg,dsRg))  ## 1 Answer I have now found my own answer. One can use savePrediction=T to query the predictions. Then by means of one these grouped per observation case and can plot this. If a tuning grid was used, of course only the parameters of the model$bestTune should be used.

The resulting curve corresponds in the ROC and also in the RMSE to the mean of the resamples. This is valuable because it allows one to estimate how approximately the model performs on unseen data. See also the article by Frank Hassel: https://hbiostat.org/bbr/md/reg.html#internal-vs--external-model-validation