2
$\begingroup$

In case of caret package test function metric option, one can use either accuracy or ROC as a metric that will be used to finalize values of tuning parameters. I felt that accuracy and ROC are the same

Accuracy = total correct predictions/total samples

ROC= looks at various probability cutoffs and gives probability cutoff at which accuracy will be the best.

Am I correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Generally speaking, Area Under the RoC (AUROC) statistic is used when you have imbalanced classes. For ex: 5% 1's and 95% 0's.

In practice, we are more interested in the AUROC to judge how well the model rank orders cases (i.e., rank from high probablity to low probablity of being a 1) where as Accuracy is... well you already know that.

In the context of model tuning, my advice would be to use AUC (especially if you have imabalanced classes) instead of Accuracy.

Improve this answer
$\endgroup$
3
  • $\begingroup$ When is it better to use accuracy instead of ROC? also when we use accuracy metric, does train function provide percentage cutoff which is giving the best accuracy or does the function only provides accuracy? isn't information provided by ROC always contains information given by accuracy and in addition to that quite a lot more information? for example train function will provide output that for a specific values of tuning parameters accuracy is the best. But in the same case ROC will provide what is accuracy over various cutoffs for the same value of tuning parameters $\endgroup$
    – user2543622
    Commented Aug 30, 2013 at 16:03
  • 1
    $\begingroup$ You would want to use accuracy as a measure if you have roughly equal proportion of 1s & 0s. Otherwise use AUC. Please read this paper to understand AUC/ROC better: Fan et al. (2006). Understanding receiver operating characteristic (ROC) curves. $\endgroup$
    – user12555
    Commented Aug 30, 2013 at 20:42
  • $\begingroup$ thanks, making some sense. When accuracy is the selection criteria to finalize tuning parameters, what is the probability cutoff to finalize the class (for caret -> train function) ? is it 0.5? lets say we built some model and we get output that an observation has 55% chance of being in class A. Would it be classsfied as A or B? Does caret package consider actual class distributions to finalize the class? i mean if class distribution is 70% (A) and 30% (B). Then if a probablity of an observation being in class A is 40%, would it be put into A bucket? $\endgroup$
    – user2543622
    Commented Aug 30, 2013 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.