1
$\begingroup$

I want to generate ROC curves using the training data and results from the rfe function in caret. I have managed to do this with the code below but there is some inconsistency between the ROC value that caret gives me and the one that I calculate with proc package. I have read in a different forum that this is somewhat expected but I thing that the difference in my dataset is bigger (caret ROC 92.2%, pROC 89.15%)(ROC curve from training data in caret)

So my question is, is it normal to observe this difference, or the way I am calculating the ROC is wrong? I am posting a reproducible example below, any help or explanation is very much appreciated!

data(PimaIndiansDiabetes)
require(caret)
require(pROC)

rfFuncs$summary <- twoClassSummary
ctrl <- rfeControl(functions = rfFuncs,
                   method = "repeatedcv",
                   number = 10,
                   repeats = 3,
                   verbose = TRUE,
                   saveDetails = TRUE,
                   returnResamp = "all")

trainctrl <- trainControl(classProbs= TRUE,
                          verboseIter = TRUE,
                          summaryFunction = twoClassSummary, 
                          method = "cv", 
                          number = 10 ,
                          returnResamp = "final", 
                          returnData = TRUE)
set.seed(12)
tunegrid <- expand.grid(.mtry=c(1:10))
rfe_rf <- rfe(PimaIndiansDiabetes[,1:8], PimaIndiansDiabetes[,9], sizes=c(1:8),
           method="rf",
           rfeControl = ctrl, 
           metric = "ROC", 
           trControl = trainctrl,
           tuneGrid = tunegrid,
           preProc = c("center", "scale"))

>rfe_rf

Recursive feature selection

Outer resampling method: Cross-Validated (10 fold, repeated 3 times) 

Resampling performance over subset size:

 Variables    ROC   Sens   Spec   ROCSD  SensSD  SpecSD Selected
         1 0.7256 0.8680 0.3914 0.05285 0.06025 0.11233         
         2 0.7695 0.8380 0.5523 0.05231 0.04649 0.09403         
         3 0.8009 0.8367 0.5759 0.04652 0.06036 0.07620         
         4 0.8111 0.8313 0.6096 0.04365 0.05698 0.07724         
         5 0.8196 0.8353 0.5958 0.04531 0.05551 0.08942         
         6 0.8258 0.8513 0.5997 0.04613 0.05399 0.09011         
         7 0.8274 0.8467 0.5897 0.04363 0.05973 0.09633         
         8 0.8307 0.8620 0.6132 0.04679 0.05235 0.10762        *

The top 5 variables (out of 8):
   glucose, mass, age, pregnant, insulin


selectedIndices <- rfe_rf$pred$Variables == rfe_rf$optsize
require(pROC)
ROC = plot.roc(rfe_rf$pred$obs[selectedIndices],
         rfe_rf$pred$neg[selectedIndices], legacy.axes = TRUE)

ROC

Call:
plot.roc.default(x = rfe_rf$pred$obs[selectedIndices], predictor = rfe_rf$pred$neg[selectedIndices],     legacy.axes = TRUE)

Data: rfe_rf$pred$neg[selectedIndices] in 1500 controls (rfe_rf$pred$obs[selectedIndices] neg) > 804 cases (rfe_rf$pred$obs[selectedIndices] pos).
Area under the curve: 0.8287

I can reproduce the ROC given by caret by averaging the ROC values from each resample for the optimal subset.

mean(rfe_rf$resample$ROC[which(rfe_rf$resample$Variables == 8)])
[1] 0.830746
$\endgroup$
  • 1
    $\begingroup$ I don't understand your question. You're mentioning you're getting different AUCs of 92.2%, and 89.15% but I don't see these anywhere further down where you have 82.9 and 83.1. Also I'm not sure what you mean by "the ROC value that caret gives me" as it doesn't give you a value, but you rather calculate a mean. Can you please clarify what you're asking exactly? $\endgroup$ – Calimo Sep 29 '18 at 7:52
  • $\begingroup$ Hi Calimo, thanks for your reply. Apologies I understand that what i wrote is not super clear. AUCs of 92.2%, and 89.15% are the ones that I get using my own dataset (not show in this example). I just provided a reproducible example showing the code I use so its easier to understand the problem. My main question is, how do I draw ROC curves using the training cv data from caret? Why do I get different results when i calculate ROC using the proc package? Should I calculate the ROC curves in a different way? $\endgroup$ – Mati Sep 29 '18 at 17:17
  • $\begingroup$ What makes you expect the ROC curve of the model to be the same as the mean of the ROC curves of the individual variables? Isn't the whole point of a model to improve the classification? It might help to link to that "different forum" you mention as it could be pretty context dependent... $\endgroup$ – Calimo Oct 1 '18 at 12:11
  • $\begingroup$ Hi Calimo, I don't expect the ROC curve of the model to be the same as the mean of the ROC curves of the individual variables. I just don't understand why when I use all the resampled results for variable size 8 to calculate the ROC curves I get different ROC than when I calculate the ROC for each resample separately and then average over them for variable size 8 (which in my understanding is what caret does). $\endgroup$ – Mati Oct 1 '18 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.