Repeated CrossValidation, finalModel and ROC curves

Question

I got a problem understanding the meaning of the finalModel when using a repeated CV.

ctrl = trainControl(method="repeatedcv", number=10, repeats = 300, savePredictions = TRUE, classProbs = TRUE)
mdl = train("Label~.", data=Data, method = "glm", trControl = ctrl)
pred = predict(mdl, newdata = Data, type="prob")
roc.1 = roc(Data$Label, pred$control)
roc.2 = roc(mdl$pred$obs,mdl$pred$control)

from what I understand :

1. In roc.1 I tested the "finalModel" of the rCV training on all my database then build the ROC curve associated to this "finalModel".
2. In roc.2 I build the "average" ROC curve using all the rCV process results.

What I don't get is: what does the "finalModel" represent?

• Is it a model averaging all the trained models coefficients?
• Is it correct to use it a predictive model (on both the training dataset and a different set)?

Answer 1

For all caret models, the final model is trained on the full dataset. caret::train uses the cross-validation scheme you chose to select model parameters (e.g. mtry for a random forest) and estimate out-of-sample performance of the model. Once the cross-validation is done, caret retrains the model on the full dataset, using the parameters it selected during cross-validation.

So roc.1 is an in-sample roc curve.

The model does not average the trained model's coefficients. It re-fits the model on the full dataset.

It is NOT correct to use the final model on the training data, but it is correct to use on a different dataset.

Answer 2

So finally to summarize :

ctrl = trainControl(method="repeatedcv", number=10, repeats = 300, savePredictions = TRUE, classProbs = TRUE)
mdl = train("Label~.", data=Data, method = "glm", trControl = ctrl)
pred = predict(mdl, newdata = Data, type="prob")
roc.1 = roc(Data$Label, pred$control)
roc.2 = roc(mdl$pred$obs,mdl$pred$control)
roc.3 = roc(as.numeric(mdl$trainingData$.outcome=='case'),aggregate(case~rowIndex,mdl$pred,mean)[,'case'])

• roc.1 is irrelevant as it evaluates a model on the same data used to train it (the finalModel is just the fit on Data ignoring the CV argument, built to apply on a different dataset for future prediction)
• roc.2 is 'almost' accurate as it will consider each prediction independently (averaging the prediction, not the probabilities)
• roc.3 is the correct way to do it as it averages the prediction probabilities for each sample among the repeated CV (contrary to roc.2 where the prediction results are averaged)