3
$\begingroup$

TL;DR For a multiclass problem, is Jaccard score the same as accuracy?


Update March 29, 2019

The wrong implementation in scikit-learn is now fixed with pull request #13151. Hooray!

P.S. The lesson here is that no matter how mature and widespread the library, framework or idea is, there are always bugs and shortcomings in them. It is up to you as an engineer, scientist or student to verify the theory and practical results of your work, especially if you rely on someone else's results.


I am working on classification problem and calculating accuracy and Jaccard score with scikit-learn which, I think, is a widely used library in pythonic scientific world. However, me and my matlab colleagues obtain different results.

sklearn.metrics.jaccard_similarity_score declares the following:

Notes: In binary and multiclass classification, this function is equivalent to the accuracy_score. It differs in the multilabel classification problem.

sklearn.metrics.accuracy_score says:

Notes In binary and multiclass classification, this function is equal to the jaccard_similarity_score function.

Indeed, jaccard_similarity_score implementation falls back to accuracy if problem is not of multilabel type:

if y_type.startswith('multilabel'):
    ...
else:
    score = y_true == y_pred

return _weighted_sum(score, sample_weight, normalize)

Isn't it contradicts the definition of Jaccard index (intersection over union)? Are these "score" and "index" different metrics? What is the correct and commonly accepted way to calculate Jaccard metrics for a multiclass problem?

$\endgroup$
  • $\begingroup$ Can you clarify whether this is a statistical question about Jaccard or a programming question about python and matlab. The latter would be better asked elsewhere. $\endgroup$ – mdewey Jan 10 '17 at 11:50
  • 1
    $\begingroup$ This is a statistical question. I added TL;DR and changed question title to clarify this fact $\endgroup$ – Ivan Aksamentov - Drop Jan 10 '17 at 11:51
  • $\begingroup$ Are these "score" and "index" different metrics? Your document says the score is the average (or sum) of Jaccard indices. $\endgroup$ – ttnphns Jan 14 '17 at 10:29
3
+100
$\begingroup$

The issue has been reported on scikit-learn GitHub repository: multiclass jaccard_similarity_score should not be equal to accuracy_score #7332

scikit-learn's Jaccard score for the multiclass classification task is incorrect.


A neat overview of the most commonly used performance metrics from {1}:

enter image description here

The accuracy is $~\frac{\text{AA}+ \text{BB} +\text{CC} }{\text{AA}+ \text{AB} +\text{AC} + \text{BA} +\text{BB} + \text{BC} + \text{CA} +\text{CB}+\text{CC}}$.

The average Jaccard score a.k.a. average Jaccard coefficient is:

$~\frac{1}{3}\left(\frac{\text{AA}}{\text{AA}+ \text{AB} +\text{AC} + \text{BA} + \text{CA}} + \frac{\text{BB}}{\text{AB} +\text{BA} +\text{BB} + \text{BC} +\text{CB}} + \frac{\text{CC}}{\text{AC} + \text{BC} + \text{CA} +\text{CB}+\text{CC}}\right) $


For example, if the confusion matrix is:

enter image description here

Then:

  • the accuracy is $~\frac{1 + 0+ 0}{1 +0 +0 +1 +0 +0 +1 +0 +0 }=~\frac{1}{3}$
  • the average Jaccard score is $~\frac{1}{3}\left(\frac{1}{1 + 0+ 0+1 +1} + \frac{0}{0+1 +0 +0 +0} + \frac{0}{0 +0 +1 +0 +0}\right) = \frac{1}{9}$

References:

$\endgroup$
  • $\begingroup$ I would like to comment on the "multiclass Jaccard" you show - without claiming if it is good or bad (for I don't know); anyway it is very different approach from what I expressed in my answer as what I suppose the authors of the python package might have meant. In regard to your Jaccard: notice that each summand is a bona fide Jaccard between "true" and "prediction" seen as binary (2-class) variables. For example, in AA/(AA+AB+...) A is seen as the class and B&C are combined as the other class. You are thus averaging in the end three 2-class Jaccards. $\endgroup$ – ttnphns Jan 13 '17 at 23:10
  • $\begingroup$ (cont.) This is questionable because the classification is one multinomial, not three binomial. (It is akin doing 3 binary logistic regressions instead of one nominal logistic.) I'm not to say it is completely incorrect, only that it may raise objections. $\endgroup$ – ttnphns Jan 13 '17 at 23:10
  • $\begingroup$ P.S. I've chosen currently to downvote because of the considerations I've given above. I'll be glad to withdraw if the answer is supported with more arguments. $\endgroup$ – ttnphns Jan 16 '17 at 8:04
  • $\begingroup$ @ttnphns Thanks for the comments, sorry I can't respond yet, I'm having some heavy workload at the moment :/ $\endgroup$ – Franck Dernoncourt Jan 16 '17 at 15:40
1
$\begingroup$

K-class multinomial classification results for n cases (tries) is a nominal variable. Therefore it can be represented as a set of k binary dummy variables.

Now, Jaccard similarity coefficient between two cases (row vectors) by a set of binary attributes is $\frac{a}{a+b+c}$; and accuracy score (I believe it is F1 score) is equal to Dice coefficient: $\frac{2a}{2a+b+c}$ (it will follow from the formula behind your link). The terms come from the table:

            case Y
             1   0
            -------
        1  | a | b |
case X      -------
        0  | c | d |
            -------
a = number of variables on which both objects X and Y are 1
b = number of variables where object X is 1 and Y is 0
c = number of variables where object X is 0 and Y is 1
d = number of variables where both X and Y are 0
a+b+c+d = p, the number of variables.

OK. Consider the example given in the documentation you link to:

y_pred = [0, 2, 1, 3]
y_true = [0, 1, 2, 3]
where values are class labels; 4 classes in all

These can be seen as 8 cases (trials) paired as 4 experiments (X cases) and 4 corresponding true outputs (Y cases). Stack all in one column and convert to dummy variables. In dummy variables, there is single 1 in each row:

case         data      v_0      v_1      v_2      v_3

x1              0        1        0        0        0
x2              2        0        0        1        0
x3              1        0        1        0        0
x4              3        0        0        0        1
y1              0        1        0        0        0
y2              1        0        1        0        0
y3              2        0        0        1        0
y4              3        0        0        0        1

Compute the matrix of Jaccard coefficient between all 8 cases, pairwise, and likewise the matrix of Dice coefficient:

enter image description here

Because we are interested in comparisons only between X and Y cases, we'll pay attention only to the yellow-highlighted portion of the matrices. We are going to sum or average coefficients within yellow area. Moreover, since data were paired we'll probably consider only the diagonal red values and average them. Your document said that their "Jaccard score" is the average of individual Jaccard indices. So here we are.

You see that entries in two yellow squares are identical: Jaccard appears to be equal to Dice (for our situation with nominal data). And the average of the 4 diagonal values is (1+0+0+1) / 4 = 0.5, the result given in your documentation.

(As an example, showing computation of both coefficients of similarity between X1 and Y1 cases):

                       v_0      v_1      v_2      v_3
x1                      1        0        0        0
y1                      1        0        0        0

               Y1
             1   0
            -------
        1  | 1 | 0 |
    X1      -------
        0  | 0 | 3 |
            -------

Jaccard: 1/(1+0+0)=1; Dice: 2*1/(2*1+0+0)=1
Note that with a single set of dummy variables both coefficients
can attain only values 0 or 1.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.