What is the difference between the average class accuracy and average instance accuracy?

Question

In some papers, I find some strange evaluation metrics that did not give a clear explanation, like average class accuracy and average instance accuracy!

1. What is the difference between both? I can not find any explanation about them...

2. How to calculate them in Python3?

also, I post two links: (1) https://arxiv.org/abs/1604.03265 Fig.7 and 8; (2) https://www.sciencedirect.com/science/article/pii/S0925231218311561 Table 1 and 2.

Thanks a lot !

Answer 1

What is the difference between both?

The accuracy metric is one of the simplest metrics to understand: it is essentially the number of correct predictions (I'll call these hits) over the number of total predictions:

$$ acc = \frac{hits}{preds} $$

There are two ways to compute this though: the micro and macro-averaged versions.

Micro-averaging involves dividing the total number of correct predictions to the total number of predictions made, regardless of the class:

$$ acc_{micro} = \frac{total \; hits}{total \; preds} $$

This is what you refer to as average instance accuracy and is actually what most people mean when referring to accuracy.

The macro-averaged accuracy, on the other hand, involves computing the accuracy of each class separately and the averaging the results

$$ acc_{macro} = \frac{1}{M} \sum_{i=1}^M{acc_i} $$

Where $acc_i$ is the accuracy for the samples class $i$ and $M$ is the total number of samples. This is what you refer to as average class accuracy.

Now, why use the latter? It has to do with class imbalance. Accuracy is a metric very sensitive to class imbalance and macro-averaging is a way to make each class matter the same when computing it. You can also read this answer, where I explain it in a bit more detail and I also wrote a simple numerical example.

How to calculate them in Python3?

• The first is very simple to compute, you can either perform the simple division or use an scikit-learn's function

• The second is a bit trickier if you want to compute it on your own. You have to compute the accuracy for the samples of each class separately (e.g. class 1: 0.6, class 2: 0.78, class 3: 0.37, etc.) and then average these numbers. Luckily, this is equivalent to using sample weights equal to the proportion of imbalance, so you can use this function to compute the same thing.