2
$\begingroup$

Recently, I built a classification model based on the imbalanced data set(positive sample is minority and negative sample is majority), and the model gave the following result for the test set:

True Positives = 0

True Negatives = 139

False Positives = 0

False Negatives = 10.

My question is: for the result, can Matthews correlation coefficient (MCC ) and F-measure be used for estimating the classifier?

Since the denominators for MCC and F-measure are zero, it seems meaningless. If so, MCC and F-measure is not always works for estimating the classifier, and sensitivity and specificity as well as g-mean should be better. Is that right?

Any help is appreciated.

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Is TP true positive, FP false positive, Tn true negative and FN false negative? If so please edit your question, abbreviations are typically not uniquely defined. $\endgroup$ – Momo Oct 16 '13 at 20:55
  • 1
    $\begingroup$ Any measure that did not give your classifier a 0 would be highly suspect. Your classifier is not doing anything. In a sense, no estimation of your classifier is needed other than the fact that it is not predicting any positives. $\endgroup$ – Peter Flom♦ Oct 16 '13 at 22:38
  • 1
    $\begingroup$ @PeterFlom, well said! I think the OP got hung up on precision/recall definition of F measure, which gives you an undefined (0/0) answer. $\endgroup$ – Matt Krause Oct 17 '13 at 0:06
3
$\begingroup$

This is only really a problem if you compute the precision and recall first, then plug them in.

One can also compute the $F_1$ score as $$F_1 = \frac{2 \cdot \textrm{True Positive}}{2 \cdot \textrm{True Positive} + \textrm{False Positive} + \textrm{False Negative}}$$

Plugging in your numbers, you'll arrive at an $F_1$score of zero, which seems appropriate since your classifier is just guessing the majority class.

There is an information-theoretic measure called proficiency that might be of interest if you are working on fairly unbalanced data sets. The idea is that you want it to remain sensitive to both classes as either the number of true positives or negatives approaches zero. It's essentially $$ \frac{I(\textrm{predicted labels}; \textrm{actual labels})}{H(\textrm{actual labels)}}$$

See pages 5--7 of White et al. (2004) for more details about its calculation and interpretation

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Glad I could help. Keep @PeterFlom's advice in mind though--the problem here isn't the evaluation method; it's the classifier. $\endgroup$ – Matt Krause Oct 17 '13 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.