1
$\begingroup$

For reproducibility, I have the following code, using the TitanicSurvival data set (10 fold cross validation repeated 3 times) in R.

Also, I am new to the forums and quite new to R, so I apologize in advance if I've broken any forum guidelines. I'll correct any mistakes if noted (originally posted in stackoverflow).

library(caret)
library(pROC)
df <- TitanicSurvival
df2 <- na.omit(df)

#Multiple regression model, then calculate AUC/Sense/Specif
titanic <- glm(survived~ sex + age, family=binomial, data = df2)
roc(df2$survived, titanic$fitted.values) # 0.7735
pred <- ifelse(predict(titanic, df2, type="response")>0.5, 1, 0)
actual <- titanic$y
conf_mat <- table(pred, actual)
sensitivity(conf_mat) # 0.8449111
specificity(conf_mat) # 0.6838407

# Now using 10-fold cross validation method:
ctrl <- trainControl(method="repeatedcv", repeats = 3, number = 10, classProbs = TRUE, 
                          summaryFunction = twoClassSummary, savePredictions = T)

model <- train(survived ~ sex + age,
                  data = df2, trControl=ctrl, method="glm", preProc = c("center", "scale"),
                  metric="ROC")

# Model ROC(AUC): 0.7738766, Sens: 0.8449321, Spec: 0.6840347

Now, my questions are:

  1. How are the AUC/Sens/Spec values in the cross validation model calculated? I'm trying to understand this output considering that the 10-fold CV creates multiple training and testing sets.
  2. How do the AUC/Sens/Spec values in the cross validation model differ from that shown in the "titanic" model?
  3. In the medical literature, some studies perform multiple regression with k-fold cross validation, without an external validation set. In this particular study, when they write "model accurately distinguished presence of HCC with c-statistics of 0.84 (95%CI 0.81-0.86) and 0.83 (95%CI 0.80-0.85) in derivation and validation cohorts (Figure 1), respectively", what does this mean? Since k-fold CV creates multiple training sets and testing sets, how does one come up with these respective values?source
$\endgroup$
  • $\begingroup$ What do you mean by development set? $\endgroup$ – Windstorm1981 Nov 21 '19 at 3:07
1
$\begingroup$

The key assumption of cross validation here is that the $k$ surrogate models are assumed to be (approximately) equal, and also (approximately) equal to the model trained on the whole data set.

These assumptions allow us to pool the test results from all $k$ folds and use them as approximation for the generalization performance of the model trained on the whole data set: at that stage, you have one confusion matrix estimated by cross validation and proceed just as with a single test set (whether internal or external).

Derivation and validation cohorts are their terms for training and test sets (within cross validation). c-statistic is AFAIK another name for the area under the ROC (AUC).

$\endgroup$
  • $\begingroup$ Thanks for your answer! Now, then within a k-fold validation model (like one shown in my question), how does one extract the AUC/ROC for training vs testing set? $\endgroup$ – dkim11 Nov 18 '19 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.