5
$\begingroup$

I'm evaluating a grid of tuning parameters using Caret with metric="ROC" for cross-validation. Is there any simple way to use as metric the area under the curve for an specified interval of the ROC curve?

My code is similar to this:

fitControl <- trainControl(method = "repeatedcv",
                       number = 10,
                       repeats = 10,
                       classProbs = TRUE,
                       ## Evaluate performance using 
                       ## the following function
                       summaryFunction = twoClassSummary)

gbmFit3 <- train(Class ~ ., data = training,
             method = "nnet",
             trControl = fitControl,
             verbose = FALSE,
             tuneGrid = myGrid,
             metric = "ROC")

And I would like, using caret, a metric as the partial area under the curve. The most simple way I think it's using cross-validation + pROC package without using caret, but I would like to know if there is a simple way to do it before I try my custom cv.

Anyone?

$\endgroup$
3
$\begingroup$

You can emulate what the package's twoClassSummary function does. See the help page for custom performance metrics.

Max

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.