I came a cross two ways to evaluate the performance of binary classifiers: accuracy and precision. When to choose each? And what are the advantages and disadvantages of each one?

up vote 2 down vote accepted

This depends entirely on the task at hand. Note that both accuracy and precision (and recall, sensitivity, specificity, ...) are measures for a specific operating setting of a classifier.

Usually one is interested in overall accuracy after choosing a specific operating setting. For highly unbalanced data sets with many more negatives than positives, precision becomes important because obtaining high accuracy is trivial for such data (always predicting negative would result in high accuracy). As an example: suppose you want to build a classifier for diagnosis of a disease with 1% prevalence. If the classifier predicts everyone to be healthy, it has 99% accuracy even though it is entirely uninformative. Precision would be useful in such a scenario.

Typical classification models, such as logistic regression, support vector machine or random forest, give more than just a binary label: they also give some measure of confidence in the prediction. Note that this confidence is not necessarily a probability (for example, SVM yields distance to the separating hyperplane).

A much better way to assess classifiers is by accounting for their performance over the full operating range by accounting for the confidence of predictions. This is done, for example, using the (area under) receiver operating characteristic (ROC) or precision-recall (PR) curves.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.