For reproducibility, I have the following code, using the TitanicSurvival data set (10 fold cross validation repeated 3 times) in R.
Also, I am new to the forums and quite new to R, so I apologize in advance if I've broken any forum guidelines. I'll correct any mistakes if noted (originally posted in stackoverflow).
library(caret) library(pROC) df <- TitanicSurvival df2 <- na.omit(df) #Multiple regression model, then calculate AUC/Sense/Specif titanic <- glm(survived~ sex + age, family=binomial, data = df2) roc(df2$survived, titanic$fitted.values) # 0.7735 pred <- ifelse(predict(titanic, df2, type="response")>0.5, 1, 0) actual <- titanic$y conf_mat <- table(pred, actual) sensitivity(conf_mat) # 0.8449111 specificity(conf_mat) # 0.6838407 # Now using 10-fold cross validation method: ctrl <- trainControl(method="repeatedcv", repeats = 3, number = 10, classProbs = TRUE, summaryFunction = twoClassSummary, savePredictions = T) model <- train(survived ~ sex + age, data = df2, trControl=ctrl, method="glm", preProc = c("center", "scale"), metric="ROC") # Model ROC(AUC): 0.7738766, Sens: 0.8449321, Spec: 0.6840347
Now, my questions are:
- How are the AUC/Sens/Spec values in the cross validation model calculated? I'm trying to understand this output considering that the 10-fold CV creates multiple training and testing sets.
- How do the AUC/Sens/Spec values in the cross validation model differ from that shown in the "titanic" model?
- In the medical literature, some studies perform multiple regression with k-fold cross validation, without an external validation set. In this particular study, when they write "model accurately distinguished presence of HCC with c-statistics of 0.84 (95%CI 0.81-0.86) and 0.83 (95%CI 0.80-0.85) in derivation and validation cohorts (Figure 1), respectively", what does this mean? Since k-fold CV creates multiple training sets and testing sets, how does one come up with these respective values?source