6
$\begingroup$

Let's say I have 2 models:

1) High Matthew's correlation coefficient (MCC) score, low area under the curve (AUC)

2) Low MCC, high AUC

When I say high and low, I mean relatively to the other model. I'm not quite sure which model is "better" and how to interpret this difference between the 2 models. Also to clarify, these 2 models both return probability estimates. MCC is determined at a threshold of 0.5.

$\endgroup$
  • $\begingroup$ It is good to start off using the correct terminology. If you can compute an AUC, the method you are using is not a classifier but rather is a risk (or other) predictor. $\endgroup$ – Frank Harrell Aug 12 '14 at 12:31
  • $\begingroup$ I guess that is true, but to just clarify, in the end I threshold the score to obtain a class prediction. $\endgroup$ – user2422566 Aug 12 '14 at 16:34
  • $\begingroup$ No, you don't need classification unless you have a loss/utility function. Lookup lift curve for one method of using the results rationally - a cost/benefit analysis. Predicted risk also gives you the luxury of not classifying hard-to-classify observations. $\endgroup$ – Frank Harrell Aug 13 '14 at 14:17
7
$\begingroup$

A binary classifier might produce at prediction time either directly the class labels for each classified instance, or some probability values for each class. In the later case, for each instance it will produce in binary case a probability value $p$ for one class, and a probability value $q=1-p$ for the second class.

If the classifier produces probabilities, one have to use a threshold value in order to obtain classification labels. Usually, this threshold value is $0.5$, but this is often not the case, and sometimes it is not the best possible value.

Now, MCC is computed directly from classification labels, which means that it was used a single threshold value to transform probabilities into classification labels, no matter what the threshold value was. AUC, on the other hand is using the whole range of threshold values.

The idea is that these two values, AUC and MCC, measure different things. While MCC measures a kind of statistical accuracy (related with Chi squared test which gives some hints on the significance of the differences), the AUC is more related with the robustness of the classifier. AUC and ROC curves gives more hints on how well the classifier separates the binary classes, for all possible threshold values. Even for the degenerate case, where AUC is computed directly on labels (not advisable since it looses a lot of information), the purpose of AUC remains the same.

Model selection is a hard problem. My advice for you would be to try yourself to answer to the question: what means a better classifier? Find an answer which includes considerations like: cost matrix, robustness to unbalanced data, an optimistic or conservative classifier, etc. Anyway, provide enough details until you find one or few criteria which can be used to select a metric before trying to measure different things related with accuracy and ask later what to do with them.

[Later edit - related with usage of robust word]

I used the term "robust" because I could not find a proper single word for "how well a classifier separates the two classes". I know that the term "robust" has some special meaning in statistics.

Generally, an AUC close to $1.0$ separates well binary cases for many values of the threshold. In this sense, the AUC close to $1.0$ means that it is less sensitive to which threshold values is used, which means its robust to this choice. However, a value which is not close to $1.0$ does not mean the contrary, which does not necessarily mean that there is not enough range of good threshold values. In most cases, a graphical inspection of ROC curve is necessary. This is one of the main reasons why AUC is often considered misleading. AUC is a measure over all possible classifiers (one for each possible threshold value), and is not a measure of a specific classifier, but in practice one can't use more than one threshold value. While AUC can give hints about the well-separation (my use of the term "robustness"), it is not used alone as a single authoritative measure of accuracy.

$\endgroup$
  • $\begingroup$ Clarified the original question to indicate both models return probability estimates. $\endgroup$ – user2422566 Aug 11 '14 at 21:25
  • $\begingroup$ Could you elaborate what you mean by robustness of the classifier for AUC? $\endgroup$ – user2422566 Aug 11 '14 at 21:37
  • $\begingroup$ I will add some details to the answer. $\endgroup$ – rapaio Aug 12 '14 at 7:37
  • 1
    $\begingroup$ Thanks, that helped! So I guess a high AUC (which is over all thresholds) is not necessarily related to a high MCC (at a 0.5 threshold). And also vice versa. $\endgroup$ – user2422566 Aug 12 '14 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.