3
$\begingroup$

I created confusion matrix and trying to get accuracy values and geometric mean (g-mean). It turned out that accuracy is around 0.83 while g-mean is around 0.91. Is it possible or do I have a mistake while calculating my measures?

$\endgroup$
5
$\begingroup$

Note: This answer has been edited following a helpful comment from usεr11852

For a 2x2 confusion matrix, the accuracy is typically defined as:

$$ \text{Accuracy} = \frac{TP+TN}{TP+FP+FN+TN} $$

While the g-mean is defined as (see, e.g., Espindola & Ebecken 2005)

$$ g_{PR} = \sqrt{\text{Precision}\times\text{Recall}} $$

or

$$ g_{SS} = \sqrt{\text{Sensitivity}\times\text{Specificity}} $$

Where $\text{Precision} = \frac{TP}{TP+FP}$, $\text{Recall} = \text{Sensitivity} = \frac{TP}{TP+FN}$, and $\text{Specificity} = \frac{TN}{TN+FP}$.

These two definitions give different results so it is important to be clear which is being used. Note that $g_{PR}$ and $g_{SS}$ are my notations for this answer and not commonly used notation.

$$ \begin{align} g_{PR} &= \frac{TP}{\sqrt{(TP+FP)(TP+FN)}} \\ g_{SS} &= \frac{\sqrt{TP\times TN}}{\sqrt{(TP+FN)(TN+FP)}} \end{align} $$

Notice that TN features in the formulae for accuracy and $g_{SS}$ but not for $g_{PR}$.

Accuracy is a bad measure, because a test/model can be quite bad but appear to have good accuracy if there are lots of TNs, and why it's meaningless in some situations, e.g., information retrieval (where TNs are of no interest and are even difficult to define).

Here are some examples where accuracy is less than $g_{PR}$ and/or $g_{SS}$:

+----+----+----+-----+-------+-------+-------+
| TP | FP | FN |  TN |  Acc. |  g_PR |  g_SS |
+----+----+----+-----+-------+-------+-------+
| 31 |  0 |  6 |  19 | 0.893 | 0.915 | 0.915 |
| 56 | 16 |  3 |   5 | 0.762 | 0.859 | 0.475 |
| 10 |  1 |  6 |  12 | 0.759 | 0.754 | 0.760 |
| 58 |  1 |  0 | 103 | 0.994 | 0.991 | 0.995 |
+----+----+----+-----+-------+-------+-------+

So, in answer to your question, it is entirely plausible for accuracy to be lower than g-mean, but it is worth making sure which g-mean is being used.

R. P. Espindola & N. F. F. Ebecken. (2005) On extending F-measure and G-mean metrics to multi-class problems. WIT Transactions on Information and Communication Technologies. Vol. 35. pp. 25-34.

$\endgroup$
  • $\begingroup$ This is potentially misleading because g-mean is very often defined in terms of Recall(Sensitivity) and Specificity, eg. Kubat & Matwin (1997) ICML. Can you please point to a published paper that defines g-mean in terms of Precision:Recall? $\endgroup$ – usεr11852 Apr 21 '20 at 15:49
  • $\begingroup$ Thanks @usεr11852 I have updated the answer to reflect the two alternative definitions. $\endgroup$ – tristan Apr 22 '20 at 10:13
  • 1
    $\begingroup$ Cool. Thanks. (+1) Irrespective your answer, I suspected E&E would come up... Espindola & Ebecken (2005) cites Kubat, Hulte & Matwin (1998) about the $g$-mean using Precision-Recall. Kubat et al. (1998) do a soft definition of the $g_{PR}$ and cite Lewis & Gale (1994) whether L&W (1994) do not mention the geometric mean at all. In general I think the use of $g_{PR}$ is very dubious. If anything, the only formal reference I have seen being mentioned on examining $g_{PR}$ is "Information Retrieval" by van Rijsbergen where the whole point is not to use it and use the $F$ score instead. $\endgroup$ – usεr11852 Apr 22 '20 at 10:51
  • $\begingroup$ Thanks @usεr11852 for excellent context. It's been a while since I worked in this field (2011) and I would generally only use F score. $\endgroup$ – tristan Apr 22 '20 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.