Evaluating Classifiers k fold CV or ROC

Question

I've been doing a project to determine the 'best' classifier for classification on a dataset from UCI. I used 10 fold stratified cross validation to calculate the mean accuracy. However it was suggested that I use ROC AUC instead.

My questions are:

1)Which is better cross validation or ROC?

2) Do you perform ROC on the test set, the training set or the whole dataset?

3) If it is the training set or test set. Do I perform it using each fold from the cross validation or should I just split the data once?

Answer 1

AUROC is a measure, just like accuracy, both can be used in a CV setting or outside (validation set, test set).

1. Depends on the objective and your data. You said you used accuracy, could it be that your classes are unbalanced? ROC as a measure is usually used when you have imbalanced classes, this may be why they recommended you use it instead of accuracy
2. Since you mentioned CV (which is done on the training set) then you would perform the ROC on the training set also, that is using CV but estimating ROC instead of accuracy. If you had a test set you would have no need for CV, you would therefore use the test set to evaluate your model (accuracy, ROC, ...)
3. CV is a better option then using the same exact training set that you used to build your model, so you would still do CV with a ROC estimate.

Answer 2

1)Which is better cross validation or ROC?

ROC AUC is a metric, just as accuracy, F1-score etc., and shouldn't be confused with cross validation. You can still employ CV and use ROC AUC as your success metric for example.

2) Do you perform ROC on the test set, the training set or the whole dataset?

When selecting models you plot ROC curves, report ROC AUC etc. You can also use ROC curve (e.g. AUC) to report the final success of your models. But, test data shouldn't be in your model selection.

3) If it is the training set or test set. Do I perform it using each fold from the cross validation or should I just split the data once?

If your success metric is ROC AUC, then you calculate it for each fold and take the average. After model selection, you can finally plot ROC curve, calculate AUC, on the whole training (or test) data based on your choice of analysis.