first time poster so hopefully this makes sense and please let me know of any important information I may leave out.
I'm trying to identify my cross-validated AUCs for basic GLM, elastic net, and random forest, using the caret package. See below:
controlroc <- trainControl(method = "repeatedcv", number = 10, repeats = 5, savePredictions = "final", classProbs = TRUE, selectionFunction = "oneSE", summaryFunction = twoClassSummary, allowParallel = TRUE ) set.seed(1234) model_listworks <- caretList( DV ~., data = train, trControl=controlroc, metric="ROC", tuneList=list( glmmodel = caretModelSpec(method="glm", family = "binomial"), enetmodel=caretModelSpec(method="glmnet", tuneGrid = expand.grid (alpha = c (0, .1, .2, .4, .6, .8, 1), lambda = seq (.01, .2, length = 40))), rfmodel = caretModelSpec(method="rf", tuneGrid = expand.grid (.mtry=c(1:10)), ntrees = 1000) ) )
As expected, random forest overfitted the training dataset. However, this continues despite CV and I believe CV isn't being accounted for somehow. I have calculated the AUC without CV, shown below with AUC of 0.999, and then swapped the same process with a model that should have CV implemented (i.e., model_listworks) and receive identical AUCs. When I just run the model_listworks model alone, however, I receive the expected CV AUC output of 0.713.
rffit = train(DV ~., data=train, method="rf", tuneGrid = expand.grid (.mtry=c(1:10)), ntrees = 1000, metric="ROC", trControl=controlroc) print(rffit) #Train probsrf1 = predict(rffit, train, type = "prob") train$probsrf1 = probsrf1[,"yesi"] roc.rf1 = roc(response = train$DV, predictor = train$probsrf1) #ROC: 0.9994 probsrf2 = predict(model_listworks$rfmodel, train, type = "prob") train$probsrf2 = probsrf2[,"yesi"] roc.rf2 = roc(response = train$DV, predictor = train$probsrf2) #ROC: 0.9994 model_listworks$rfmodel #ROC: 0.7128026
Is there something I'm missing here? I just need my training CV AUCs for these three models.