When I am running cross validation on the training set for a binary classification problem, what metric should I use if I am only interested in obtaining the largest AUC (area under receiver operating characteristic Curve) value of the corresponding test set, AUC as the metric in the training set (available in caret package in r) or other cross-validation error metric such as Bernoulli deviance and 0-1 loss?

Ref for Bernoulli deviance: Logistic Regression: Bernoulli vs. Binomial Response Variables

  • $\begingroup$ Why wouldn't you cross-validate using the same metric you are optimizing for in the test set? $\endgroup$ – frelk Jul 23 '16 at 18:03
  • $\begingroup$ @frelk In r packages such as gbm, it is not an option to choose the cross-validation metric in the training set. What I can choose is the metric used in the test set. For example, I can choose confusion matrix as the metric in the test set in binary classification. Now with the package caret, it becomes an option to choose what metric I can use in the cross validation in the training set. So the question comes: is there any difference in using different metrics in training set and test set, which is almost equivalent to your comment. $\endgroup$ – vtshen Jul 24 '16 at 15:29

As @frelk mentioned: if your goal is to obtain a high AUC for unseen data, such as your test data, and you have the option to use AUC for both training and test (training: evaluation using cross validation to e.g. choose the best suited hyperparameters), then using AUC for both training and test is very likely the way to go.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.