How to use R package caret to build a model and get the internal validation result?

Question

I am a learner of R and machine learning. I don't really understand caret's train function. To make it simple, for example, I want to build a model and get the internal validation result.

At the beginning, I don't use package caret.

# 1 prepare data_table


library(tidyverse)

library(mlbench)

data("PimaIndiansDiabetes")

data_table <- PimaIndiansDiabetes



# 2 use glm to build a model

glm_model <- glm(diabetes ~ ., data = data_table, family = binomial)

glm_model$coefficients


# 3 get internal validation result and calclate AUC

predicted_probs <- map_dfc(1:3, function(num) {

    predicted_prob <- vector(mode = 'numeric', length = nrow(data_table))

    folds <- createFolds(data_table$diabetes, k = 10, list = TRUE, returnTrain = TRUE)

    for (i in 1:10) {

        training_index <- folds[[i]]
        test_index <- setdiff(1:nrow(data_table), training_index)

        training_table <- data_table[training_index, ]
        test_table     <- data_table[test_index, ]

        train_model <- glm(diabetes ~ ., data = training_table, family = 'binomial')

        predicted_prob[test_index] <- predict(train_model, newdata = test_table, type = 'response')
    }

    roc <- pROC::roc(data_table$diabetes ~ predicted_prob, quiet = TRUE)

    return(predicted_prob)
})

final_predicted_prob <- rowMeans(predicted_probs)

final_roc <- pROC::roc(data_table$diabetes ~ final_predicted_prob, quiet = TRUE)

# internal validataion result

print(final_roc$auc)

My first question is: is the above method correct to get the internal validation result?

Now I want to use package caret as following:

# 4 caret

library(caret)

train_control <- trainControl(method = "cv", number = 10, repeats = 3,
    savePredictions = TRUE, summaryFunction = twoClassSummary)

caret_model <- caret::train(diabetes ~ ., data = data_table,
    method = "glm", family = "binomial",
    metric = 'ROC', trControl = train_control)

print(caret_model$finalModel$coefficients)

caret_predicted_prob <- caret_model$pred %>% arrange(rowIndex) %>% pull(pos)

caret_roc <- pROC::roc(data_table$diabetes ~ caret_predicted_prob, quiet = TRUE)

print(caret_roc$auc)

This way to use caret is much simpler than my own method.

My second question is: is the caret method correct to get the final model and internal validation result?

By the way, since caret's train function already has the option of cross validation, why the following web page still split data into training and testing before calling train function:

https://topepo.github.io/caret/model-training-and-tuning.html#basic

library(caret)

inTraining <- createDataPartition(Sonar$Class, p = .75, list = FALSE)

training <- Sonar[ inTraining,]

testing  <- Sonar[-inTraining,]

fitControl <- trainControl(
    method = "repeatedcv",
    number = 10,
    repeats = 10)

gbmFit1 <- train(Class ~ ., data = training, 
    method = "gbm", 
    trControl = fitControl)

My third question is: do I need to split data_table into training & testing data when using caret to do cross validation?

Thank you very much!

Answer 1

1. It seems you're averaging the target values for each repeat to get a single one. It's not wrong, and I'm not sure what caret does if it does anything at all, but calculating the ROC values for each repeat and averaging the AUCs would make more sense. Especially, in multi-class scenarios.

2. You may need to select repeatedcv since you have repeats != 1. I'm not what the folllowing line does as my R knowledge is limited, but it seems you're again creating a single prediction out of repeated ones. I'm not sure if it's the average of repeats, but this aligns with your first implementation.

caret_predicted_prob <- caret_model$pred %>% arrange(rowIndex) %>% pull(pos)

3. The final test is used for final evaluation. A separate validation set, or cross-validation is used to tune the hyperparameters (HP). That's what the authors are doing in section 5.5.2. You don't search for best set of HPs above, so there is no need to separate into train and test. You're already estimating the generalization performance using repeated cross validation.