1
$\begingroup$

When I am running cross validation on the training set for a binary classification problem, what metric should I use if I am only interested in obtaining the largest AUC (area under receiver operating characteristic Curve) value of the corresponding test set, AUC as the metric in the training set (available in caret package in r) or other cross-validation error metric such as Bernoulli deviance and 0-1 loss?

Ref for Bernoulli deviance: Logistic Regression: Bernoulli vs. Binomial Response Variables

Improve this question
$\endgroup$
2
  • $\begingroup$ Why wouldn't you cross-validate using the same metric you are optimizing for in the test set? $\endgroup$
    – frelk
    Jul 23, 2016 at 18:03
  • $\begingroup$ @frelk In r packages such as gbm, it is not an option to choose the cross-validation metric in the training set. What I can choose is the metric used in the test set. For example, I can choose confusion matrix as the metric in the test set in binary classification. Now with the package caret, it becomes an option to choose what metric I can use in the cross validation in the training set. So the question comes: is there any difference in using different metrics in training set and test set, which is almost equivalent to your comment. $\endgroup$
    – vtshen
    Jul 24, 2016 at 15:29

1 Answer 1

Reset to default
1
$\begingroup$

As @frelk mentioned: if your goal is to obtain a high AUC for unseen data, such as your test data, and you have the option to use AUC for both training and test (training: evaluation using cross validation to e.g. choose the best suited hyperparameters), then using AUC for both training and test is very likely the way to go.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.