Cross Validation Random Forest AUC very high

Question

first time poster so hopefully this makes sense and please let me know of any important information I may leave out.

I'm trying to identify my cross-validated AUCs for basic GLM, elastic net, and random forest, using the caret package. See below:

controlroc <- trainControl(method = "repeatedcv", 
                           number = 10,
                           repeats = 5,
                           savePredictions = "final",
                           classProbs = TRUE,
                           selectionFunction = "oneSE", 
                           summaryFunction = twoClassSummary, allowParallel = TRUE )

set.seed(1234)

model_listworks <- caretList(
  DV ~., 
  data = train,
  trControl=controlroc,
  metric="ROC",
  tuneList=list( 
    glmmodel = caretModelSpec(method="glm", family = "binomial"),
    enetmodel=caretModelSpec(method="glmnet", tuneGrid = expand.grid (alpha = c (0, .1, .2, .4, .6, .8, 1), lambda = seq (.01, .2, length = 40))),
    rfmodel = caretModelSpec(method="rf", tuneGrid = expand.grid (.mtry=c(1:10)), ntrees = 1000)
  ) 
)

As expected, random forest overfitted the training dataset. However, this continues despite CV and I believe CV isn't being accounted for somehow. I have calculated the AUC without CV, shown below with AUC of 0.999, and then swapped the same process with a model that should have CV implemented (i.e., model_listworks) and receive identical AUCs. When I just run the model_listworks model alone, however, I receive the expected CV AUC output of 0.713.

rffit = train(DV ~., data=train, method="rf", tuneGrid = expand.grid (.mtry=c(1:10)), ntrees = 1000, metric="ROC", trControl=controlroc)
print(rffit)

#Train
probsrf1 = predict(rffit, train, type = "prob")
train$probsrf1 = probsrf1[,"yesi"]
roc.rf1 = roc(response = train$DV, predictor = train$probsrf1)
#ROC: 0.9994

probsrf2 = predict(model_listworks$rfmodel, train, type = "prob")
train$probsrf2 = probsrf2[,"yesi"]
roc.rf2 = roc(response = train$DV, predictor = train$probsrf2)
#ROC: 0.9994

model_listworks$rfmodel
#ROC: 0.7128026

Is there something I'm missing here? I just need my training CV AUCs for these three models.

Answer 1

So what you have done is done cross-validation to find out the best hyper parameter for the data. What CV does is to predict the model on the fold not used in training, and this step is supposed to reduce overfitting. If you choose a hyper parameter that follows the training data too closely, it will perform badly on the test fold, making it less likely to be chosen.

If you look under model_listworks$rfmodel you will see what is the mtry chosen.

In this part of the code:

probsrf2 = predict(model_listworks$rfmodel, train, type = "prob")

You are essentially take the model that is trained on all the training data, and predicting it again on the same data, hence you get a very high AUC, because it's the same data used in the training.

The training CV AUCs will be under model_listworks, like you have done, model_listworks$rfmodel for random forest. These are basically the AUC calculated using the test fold.