7
$\begingroup$

How to compare 2 classifiers using Confusion Matrix?

For example, if we have 2 confusion matrix(binary classification) obtained from different classifiers or using different features, how I can compare the performance of classifiers using Confusion Matrix?

confusion matrix
    60 34
    1  12

confusion matrix
    90 4
    8  5
Improve this question
$\endgroup$

3 Answers 3

Reset to default
4
$\begingroup$

As @Enrique mentioned, there are many statistics you can calculate from a confusion matrix. To determine which ones are appropriate depends on the specific characteristics of your problem, such as the relative costs associated with true positives and false positives.

Chapter 11 of Applied Predictive Modeling gives a very detailed overview of how to think about evaluating classification models.

An Introduction to Statistical Learning, which is freely available as a pdf, provides a less detailed overview in chapter 4.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What statistics do you recomend to classification problem with unbalanced data? $\endgroup$
    – mrgloom
    Commented Aug 17, 2015 at 17:06
  • 1
    $\begingroup$ If you have highly imbalanced data, a lot of the basic measures of performance won't be terribly useful (e.g., you can get a high accuracy by just predicting all negatives). AUC (area under the ROC curve) can be good because it can help you identify what cutoff to choose. That'll let you decide what ratio of true positives to false positives you're comfortable with. Positive Predictive Value is another option. $\endgroup$
    – Tchotchke
    Commented Aug 17, 2015 at 17:12
1
$\begingroup$

From the confusion matrices you can compute the sensitivity, specificity, accuracy, precision, among other performance metrics for each of the classifiers. Then you can evaluate them in terms of those metrics. Here you can find the definition of several metrics and how they can be computed.

Improve this answer
$\endgroup$
1
$\begingroup$

Confusion Matrix is a square matrix, which in the ideal case, its main diagonal must be valued and other sides must be none.

Confusion Matrix in binomial class:
------+-------
| TP  |  FP  |
--------------
| FN  |  TN  |
------+-------

Thus, there are two main methods to evaluate a predictor: Using accuracy or F1-score (in imbalanced cases should be used).

ACC = (TP + TN) / (P + N)

F1 = 2TP / (2TP + FP + FN)

So in your case, the second predictor is better with 89% accuracy than the first one with 67%.

Here is a Confusion Matrix Online Calculator.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.