4
$\begingroup$

I would like to have a per Label accuracy and classifier level accuracy, but my calculations seem incorrect. Here is my full example.

Let's say I have a multilabel classifier which predicted in the following fashion for labels [1, 2, 9, 11]

   PREDICTED
   __________________
  |    | 1  2  9  11 |
  |====|=============|
T | 1  | 0  1  0  1  |
R | 2  | 1  0  0  0  |
U | 9  | 0  0  1  0  |
E | 11 | 0  0  0  0  |
   ==================

For each label, I have calculated the following accuracy ($\frac{TP+TN}{ TP+TN+FP+FN}$)

Accuracy per Label
Label :                            Accuracy
1     : tp:0, tn:1, fp:1, fn:2 ==> 1/4 = 0.25
2     : tp:0, tn:2, fp:1, fn:1 ==> 2/4 = 0.50
9     : tp:1, tn:3, fp:0, fn:0 ==> 4/4 = 1.00
11    : tp:0, tn:3, fp:1, fn:0 ==> 3/4 = 0.75

Therefore, How can accuracy for the classifier be calculated by using the accuracies per label?

My original thought was average the results

$ClassifierAccuracy = \frac{1}{4}\sum{[0.25, 0.5, 1.0, 0.75]} = 0.625$

But I know, when calculating the classifier accuracy, it would be the diagonal sum over the total.

>>> X.diagonal().sum() / X.sum()
0.25

It seems like I may be double counting, but I am unsure how to calculate the result of the classifier from the label accuracies. Is this possible?

$\endgroup$
  • $\begingroup$ @Satwik Bhattamishra makes a valid point at the start of his answer, but not one that is necessarily universally applicable. Could you provide some insight into why you feel an aggregation of the per label results is preferred over direct calculation? Also, your calculation for label 11 does not align with the table presented (there are no fp, rather there are 4 tn in the table) $\endgroup$ – ReneBt Apr 6 '18 at 8:09
  • $\begingroup$ Thank you for your responses. I would like to know if this is possible, not preferable. I can calculate in both ways but have some constraints which are using per label accuracy. Also, for label 11, fp = columns[3].sum() - tp which is 1, correct? $\endgroup$ – Nathan McCoy Apr 6 '18 at 8:39
  • $\begingroup$ Your FP for label 11 seems correct. What bothers me the most is your calculation of the true negatives. Notably, if you assign mat[2,2] as 0, then your overall accuracy is clearly 0. But your per-label accuracy will be non-negative for each label. $\endgroup$ – Satwik Bhattamishra Apr 6 '18 at 9:52
  • $\begingroup$ Since the overall accuracy only depends on the diagonal values and your per-label accuracy depends on other values (from which you are deriving the TN value), I do not think that it is possible to derive the overall accuracy from your measured per-label accuracy. Primarily because when you only have the per-label accuracy, there is no way to differentiate how much of it is due to the contribution of TP and how much of it is due to the contribution of TN. $\endgroup$ – Satwik Bhattamishra Apr 6 '18 at 9:53
  • 1
    $\begingroup$ If you could update your answer to speak about this inability, I'll give it a look. $\endgroup$ – Nathan McCoy Apr 6 '18 at 10:31
4
+50
$\begingroup$

I fail to see why you are trying to use per-label accuracy for the calculation of the overall accuracy. For any problem, you can compute the overall accuracy as: $$Acc =\sum \frac{\#CorrectPredictions}{\#TotalNumberofPredictions}$$

Your method for computing per label accuracy is not entirely correct. When you traverse a particular row in a confusion matrix, then you can only get the True Positive values and the False Negative values. Let's consider the general case of binary classification, if the confusion matrix is something like the following:

    PREDICTED
   ____________
T |    | 1  0 |
R |====|======|
U | 1  | 2  1 |
E | 0  | 1  3 |
   ============

Here we are just labeling $1$ as Positive and $0$ as Negative. Just consider your method of calculating per-label accuracy here which in essence should be equivalent to calculating the overall accuracy. If you traverse along a particular row, then the values for label $1$ should be: $TP:2, TN : ?, FP : ?, FN : 1$. If you traverse across the column then you can get the value of FP ($1$ in this case). But even here you can easily see that the overall accuracy is just the ratio of the sum of diagonals and the sum of the matrix. The True Negative here is just the diagonal value for label $0$.

You are probably mistaken in some idea about the correctness of a prediction. The formula $\frac{(TP + TN)}{ TP+TN+FN+FP}$ holds for binary classification. The $TN$ there is just the $TP$ for label $0$. When you generalize over multiclass classification then the diagonal value corresponds to True positive for the column label and row label (which are the same). All other values in a specific row correspond to False Negative values for the row label and similarly, all other values in the column correspond to False Positive values for the column label. For example for label $2$:

       PREDICTED
   _____________________
  |    | 1   2   9   11 |
  |====|================|
T | 1  | .   FP  .   .  |
R | 2  | FN  TP  FN  FN |
U | 9  | .   FP  .   .  |
E | 11 | .   FP  .   .  |
   =====================

The problem with computing the overall accuracy based on the per-label accuracy computed by your method is to distinguish between the contributions of TP and TN values.

Let's first establish here that the overall accuracy is the ratio of the diagonal values and the overall sum of the matrices, which is to say that the numerator contains the count of True Positives for each label and the denominator is the count of total predictions. The problem with deriving the overall accuracy from per-label accuracy is to differentiate the effects of the TN counts in each per-label accuracy. For instance, if we assign confusion_mat[2,2] as $0$, then the overall accuracy is clearly $0$ but the per-label accuracy is still non-negative for each label due to the contributions of TN values. What is more important to note is that the per-label accuracy of label $9$ still remains $1.00$, although the per-label accuracy of every other label decrease due to a decrement in their TN value. What this shows is that in some cases the per-label accuracy could be entirely due to the contribution of TN values. What this implies is that when you are given only the per-label accuracies, then there is no additive weighted combination from which you can derive the overall accuracy since you lack any information whatsover as to how much of each per-label accuracy is based on the TP value. The per-label accuracies are likely to overestimate the actual overall accuracy due to the additional contribution of TN values.

The reason why TP+TN works in case of binary classification is that the TN value is just another diagonal value corresponding to the TP of the other label but for multi-class case here, you are considering non-diagonal values in your TN for a particular label (thus considering them as positive or correct predictions).

$\endgroup$
0
$\begingroup$

You should calculate in this way:

Accuracy per Label
Label :                            Accuracy
1     : 0/2 = 0.00
2     : 0/1 = 0.00
9     : 1/1 = 1.00
11    : 0/0 = 0.00

So the total accuracy is:

(0.00 * 2 + 0.00 * 1 + 1.00 * 1 + 0.00 * 0) / 4 = 1/4 = 0.25
  • Each one should be multiplied with its frequency.

Now assume this new case:

   PREDICTED
   __________________
  |    | 1  2  9  11 |
  |====|=============|
T | 1  | 1  0  0  1  |
R | 2  | 1  0  0  0  |
U | 9  | 0  0  1  0  |
E | 11 | 0  0  0  0  |
   ==================

The answer would be =

Accuracy per Label
Label :                            Accuracy
1     : 1/2 = 0.50
2     : 0/1 = 0.00
9     : 1/1 = 1.00
11    : 0/0 = 0.00

(0.50 * 2 + 0.00 * 1 + 1.00 * 1 + 0.00 * 0) / 4 = 2/4 = 0.50
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.