first time poster so hopefully this makes sense and please let me know of any important information I may leave out.
I'm trying to identify my cross-validated AUCs for basic GLM, elastic net, and random forest, using the caret package. See below:
controlroc <- trainControl(method = "repeatedcv",
number = 10,
repeats = 5,
savePredictions = "final",
classProbs = TRUE,
selectionFunction = "oneSE",
summaryFunction = twoClassSummary, allowParallel = TRUE )
set.seed(1234)
model_listworks <- caretList(
DV ~.,
data = train,
trControl=controlroc,
metric="ROC",
tuneList=list(
glmmodel = caretModelSpec(method="glm", family = "binomial"),
enetmodel=caretModelSpec(method="glmnet", tuneGrid = expand.grid (alpha = c (0, .1, .2, .4, .6, .8, 1), lambda = seq (.01, .2, length = 40))),
rfmodel = caretModelSpec(method="rf", tuneGrid = expand.grid (.mtry=c(1:10)), ntrees = 1000)
)
)
As expected, random forest overfitted the training dataset. However, this continues despite CV and I believe CV isn't being accounted for somehow. I have calculated the AUC without CV, shown below with AUC of 0.999, and then swapped the same process with a model that should have CV implemented (i.e., model_listworks) and receive identical AUCs. When I just run the model_listworks model alone, however, I receive the expected CV AUC output of 0.713.
rffit = train(DV ~., data=train, method="rf", tuneGrid = expand.grid (.mtry=c(1:10)), ntrees = 1000, metric="ROC", trControl=controlroc)
print(rffit)
#Train
probsrf1 = predict(rffit, train, type = "prob")
train$probsrf1 = probsrf1[,"yesi"]
roc.rf1 = roc(response = train$DV, predictor = train$probsrf1)
#ROC: 0.9994
probsrf2 = predict(model_listworks$rfmodel, train, type = "prob")
train$probsrf2 = probsrf2[,"yesi"]
roc.rf2 = roc(response = train$DV, predictor = train$probsrf2)
#ROC: 0.9994
model_listworks$rfmodel
#ROC: 0.7128026
Is there something I'm missing here? I just need my training CV AUCs for these three models.