1
$\begingroup$

In some papers, I find some strange evaluation metrics that did not give a clear explanation, like average class accuracy and average instance accuracy!

  1. What is the difference between both? I can not find any explanation about them...

  2. How to calculate them in Python3?

also, I post two links: (1) https://arxiv.org/abs/1604.03265 Fig.7 and 8; (2) https://www.sciencedirect.com/science/article/pii/S0925231218311561 Table 1 and 2.

Thanks a lot !

$\endgroup$
1
$\begingroup$

What is the difference between both?

The accuracy metric is one of the simplest metrics to understand: it is essentially the number of correct predictions (I'll call these hits) over the number of total predictions:

$$ acc = \frac{hits}{preds} $$

There are two ways to compute this though: the micro and macro-averaged versions.

Micro-averaging involves dividing the total number of correct predictions to the total number of predictions made, regardless of the class:

$$ acc_{micro} = \frac{total \; hits}{total \; preds} $$

This is what you refer to as average instance accuracy and is actually what most people mean when referring to accuracy.

The macro-averaged accuracy, on the other hand, involves computing the accuracy of each class separately and the averaging the results

$$ acc_{macro} = \frac{1}{M} \sum_{i=1}^M{acc_i} $$

Where $acc_i$ is the accuracy for the samples class $i$ and $M$ is the total number of samples. This is what you refer to as average class accuracy.

Now, why use the latter? It has to do with class imbalance. Accuracy is a metric very sensitive to class imbalance and macro-averaging is a way to make each class matter the same when computing it. You can also read this answer, where I explain it in a bit more detail and I also wrote a simple numerical example.

How to calculate them in Python3?

  • The first is very simple to compute, you can either perform the simple division or use an scikit-learn's function

  • The second is a bit trickier if you want to compute it on your own. You have to compute the accuracy for the samples of each class separately (e.g. class 1: 0.6, class 2: 0.78, class 3: 0.37, etc.) and then average these numbers. Luckily, this is equivalent to using sample weights equal to the proportion of imbalance, so you can use this function to compute the same thing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.