Return to Answer

This answer is about classification comparison indices. See a counterpart Q/A about clustering comparison indices.

This answer is about classification comparison indices. See a counterpart Q/A about clustering agreement indices.

• Accuracy aka Rand aka Simple Matching coefficient, symmetric, ranges [0,1].

  $ACC=(a+d)/N$

  1-ACC is called Missclassification (or Error) Rate and is linearly equivalent to the squared euclidean distance.

• Recall aka Sensitivity aka True Positive Rate aka Hit Rate aka Positive Accuracy, asymmetric (counterpart value =PRE), ranges [0,1].

  $REC=a/(a+b)$

• Specificity aka True Negative Rate aka Negative Accuracy, asymmetric (counterpart value =NPV), ranges [0,1].

  $SPE=d/(d+c)$

• Youden's index, asymmetric, ranges [-1,1].

  $YOUD=REC+SPE-1$

• Precision aka Positive Predictive Value, asymmetric (counterpart value =REC), ranges [0,1].

  $PRE=a/(a+c)$

  1-PRE is called False Discovery Rate.

• Negative Predictive Value, asymmetric (counterpart value =SPE), ranges [0,1].

  $NPV=d/(d+b)$

• Markedness index, asymmetric, ranges [-1,1].

  $MARK=PRE+NPV-1$

• F1 aka F Measure aka Dice Matching coefficient, symmetric, ranges [0,1], it is the harmonic mean of REC and PRE.

  $F1=2a/(2a+b+c)=2PRE∙REC⁄(PRE+REC)$

• F-beta aka generalized or weighted F Measure, generally asymmetric, ranges [0,1], it is the weighted harmonic mean of REC and PRE.

  $FBETA=(1+ beta^2)PRE∙REC⁄(beta^2 PRE+REC)$

  where parameter beta (0,+∞): If beta<1, PRE receives greater weight than REC; if beta>1, REC receives greater weight than PRE. At beta=1 FBETA turns into F1.

• Kulczynski 2 coefficient, symmetric, ranges [0,1], it is the arithmetic mean of REC and PRE.

  $KULCZ2=(PRE+REC)/2$

• logarithm of Diagnostic Odds Ratio, symmetric, ranges [-∞,+∞].

  $LNDOR= \ln ⁡(ad⁄bc)$

• Discriminant Power, symmetric, ranges [-∞,+∞].

  $DP= \frac{\sqrt 3}{\pi} (\ln⁡ \frac{REC}{1-SPE} + \ln \frac{SPE}{1-REC})$

• Matthews correlation aka Phi correlation, symmetric, ranges [-1,1]. This is just the Pearson correlation in case of binary data.

  $CORR= \frac{ad-bc}{\sqrt{(a+b)(a+c)(b+d)(c+d)}} = \frac{a/N-SP}{\sqrt{SP(1-S)(1-P)}}$

  where $S=(a+b)/N$ and $P=(a+c)/N$.

• Balanced Classification Rate, asymmetric, ranges [0,1], it is the arithmetic mean of REC and SPE and it is the AUC (area under the curve) in the ROC space, for a single point there.

  $BCR=(REC+SPE)/2$

• GM Measure, asymmetric, ranges [0,1], it is the geometric mean of REC and SPE.

  $GM=\sqrt{REC∙SPE}$

• Adjusted GM Measure, asymmetric, ranges [0,1], is a modification of GM designed to better cope with unbalanced classes.

  $AGM= \frac{GM+SPE∙Q}{1+Q}$, and with REC=0, AGM=0

  where $Q=(d+c)/N$

  If the positive class is disproportionally small, this measure may be preferable to GM because it is more sensitive to changes in SPE than to changes in REC.

• Optimized Precision, asymmetric, ranges [-∞,1]. The measure is higher when a and d both are high.

  $OPRE=ACC - |REC-SPE|/(REC+SPE)$

• Jaccard aka Tanimoto Matching coefficient, symmetric, ranges [0,1].

  $JACCARD=a/(a+b+c)$

This answer is about classification comparison indices. See a counterpart Q/A about clustering comparison indices.

• Accuracy aka Rand aka Simple Matching coefficient, symmetric, ranges [0,1].

  $ACC=(a+d)/N$

  1-ACC is called Missclassification (or Error) Rate and is linearly equivalent to the squared euclidean distance.

• Recall aka Sensitivity aka True Positive Rate aka Hit Rate aka Positive Accuracy, asymmetric (counterpart value =PRE), ranges [0,1].

  $REC=a/(a+b)$

• Specificity aka True Negative Rate aka Negative Accuracy, asymmetric (counterpart value =NPV), ranges [0,1].

  $SPE=d/(d+c)$

• Youden's index, asymmetric, ranges [-1,1].

  $YOUD=REC+SPE-1$

• Precision aka Positive Predictive Value, asymmetric (counterpart value =REC), ranges [0,1].

  $PRE=a/(a+c)$

  1-PRE is called False Discovery Rate.

• Negative Predictive Value, asymmetric (counterpart value =SPE), ranges [0,1].

  $NPV=d/(d+b)$

• Markedness index, asymmetric, ranges [-1,1].

  $MARK=PRE+NPV-1$

• F1 aka F Measure aka Dice Matching coefficient, symmetric, ranges [0,1], it is the harmonic mean of REC and PRE.

  $F1=2a/(2a+b+c)=2PRE \cdot REC⁄(PRE+REC)$

• F-beta aka generalized or weighted F Measure, generally asymmetric, ranges [0,1], it is the weighted harmonic mean of REC and PRE.

  $FBETA=(1+ beta^2)PRE \cdot REC⁄(beta^2 PRE+REC)$

  where parameter beta (0,+∞): If beta<1, PRE receives greater weight than REC; if beta>1, REC receives greater weight than PRE. At beta=1 FBETA turns into F1.

• Kulczynski 2 coefficient, symmetric, ranges [0,1], it is the arithmetic mean of REC and PRE.

  $KULCZ2=(PRE+REC)/2$

• logarithm of Diagnostic Odds Ratio, symmetric, ranges [-∞,+∞].

  $LNDOR= \ln ⁡(ad⁄bc)$

• Discriminant Power, symmetric, ranges [-∞,+∞].

  $DP= \frac{\sqrt 3}{\pi} (\ln⁡ \frac{REC}{1-SPE} + \ln \frac{SPE}{1-REC})$

• Matthews correlation aka Phi correlation, symmetric, ranges [-1,1]. This is just the Pearson correlation in case of binary data.

  $CORR= \frac{ad-bc}{\sqrt{(a+b)(a+c)(b+d)(c+d)}} = \frac{a/N-SP}{\sqrt{SP(1-S)(1-P)}}$

  where $S=(a+b)/N$ and $P=(a+c)/N$.

• Balanced Classification Rate, asymmetric, ranges [0,1], it is the arithmetic mean of REC and SPE and it is the AUC (area under the curve) in the ROC space, for a single point there.

  $BCR=(REC+SPE)/2$

• GM Measure, asymmetric, ranges [0,1], it is the geometric mean of REC and SPE.

  $GM=\sqrt{REC \cdot SPE}$

• Adjusted GM Measure, asymmetric, ranges [0,1], is a modification of GM designed to better cope with unbalanced classes.

  $AGM= \frac{GM+SPE \cdot Q}{1+Q}$, and with $REC=0$, $AGM=0$

  where $Q=(d+c)/N$

  If the positive class is disproportionally small, this measure may be preferable to GM because it is more sensitive to changes in SPE than to changes in REC.

• Optimized Precision, asymmetric, ranges [-∞,1]. The measure is higher when a and d both are high.

  $OPRE=ACC - |REC-SPE|/(REC+SPE)$

• Jaccard aka Tanimoto Matching coefficient, symmetric, ranges [0,1].

  $JACCARD=a/(a+b+c)$

Measures of agreement (similarity) between U and V can be symmetric or asymmetric. A symmetric measure will not change value if U (table rows) and V (table columns) swap their places, i.e., the frequency cross-table get transposed. An asymmetric measure will change value after the table is transposed. For asymmetric measures, it is significant which of the two partitions to consider reference (exemplar, true), and which to consider predicted (experimental). For asymmetric measures, the current answer treats partition U (defining table rows) as the reference classification, and partition V (defining table columns) as the predicted classification.

Measures of agreement (similarity) between U and V can be symmetric or asymmetric. A symmetric measure will not change value if U (table rows) and V (table columns) swap their places, i.e., the frequency cross-table get transposed. An asymmetric measure will change value after the table is transposed. For asymmetric measures, it is significant which of the two partitions to consider reference (exemplar, true), and which to consider predicted (experimental). For asymmetric measures, the current answer treats partition U (defining table rows) as the reference classification, and partition V (defining table columns) as the predicted classification $^1$.

$^1$ A decision schema around the issue of asymmetric agreement indices. Indices can be symmetric or asymmetric mathematically and partitions can be symmetric or asymmetric positionally. Positionally symmetric partitions is when we're just comparing two alternative partitions. Positionally asymmetric partitions is when one of the two is "reference" and the other is "experimental" - different roles. Then...

If the index is symmetric and the roles are symmetric, then no problem. If the index is symmetric and the roles are asymmetric, we say: the given symmetric index serves the asymmetric roles (like, say, correlation coefficient is still a valid index in a cause-response relationship).

If the index is asymmetric and the roles are symmetric, then, in case of comparison of clustering partitions (i.e., no knowledge of labels is used) we treat the index asymmetry as a mere inconvenience and average/combine the two values (if the action defensible). But in case of comparison of classification partitions (knowledge of labels is used) we refuse the symmetry of roles and move to the situation where one partition is declared "reference" and the other "experimental".

If the index is asymmetric and the roles are asymmetric, then, in case of comparison of clustering partitions we may accept and interpret both distinct values of the index (plus possibly average/combine them into one). In case of comparison of classification partitions we interpret only one of the two values of the index, based on "who is who" in the partitions pair.