Accuracy vs Jaccard for multiclass problem

Question

TL;DR For a multiclass problem, is Jaccard score the same as accuracy?

---

Update March 29, 2019

The wrong implementation in scikit-learn is now fixed with pull request #13151. Hooray!

_{P.S. The lesson here is that no matter how mature and widespread the library, framework or idea is, there are always bugs and shortcomings in them. It is up to you as an engineer, scientist or student to verify the theory and practical results of your work, especially if you rely on someone else's results.}

---

I am working on classification problem and calculating accuracy and Jaccard score with scikit-learn which, I think, is a widely used library in pythonic scientific world. However, me and my matlab colleagues obtain different results.

sklearn.metrics.jaccard_similarity_score declares the following:

Notes: In binary and multiclass classification, this function is equivalent to the accuracy_score. It differs in the multilabel classification problem.

sklearn.metrics.accuracy_score says:

Notes In binary and multiclass classification, this function is equal to the jaccard_similarity_score function.

Indeed, jaccard_similarity_score implementation falls back to accuracy if problem is not of multilabel type:

if y_type.startswith('multilabel'):
    ...
else:
    score = y_true == y_pred

return _weighted_sum(score, sample_weight, normalize)

Isn't it contradicts the definition of Jaccard index (intersection over union)? Are these "score" and "index" different metrics? What is the correct and commonly accepted way to calculate Jaccard metrics for a multiclass problem?

Answer 1

The issue has been reported on scikit-learn GitHub repository: multiclass jaccard_similarity_score should not be equal to accuracy_score #7332

scikit-learn's Jaccard score for the multiclass classification task is incorrect.

---

A neat overview of the most commonly used performance metrics from {1}:

The accuracy is $~\frac{\text{AA}+ \text{BB} +\text{CC} }{\text{AA}+ \text{AB} +\text{AC} + \text{BA} +\text{BB} + \text{BC} + \text{CA} +\text{CB}+\text{CC}}$.

The average Jaccard score a.k.a. average Jaccard coefficient is:

$~\frac{1}{3}\left(\frac{\text{AA}}{\text{AA}+ \text{AB} +\text{AC} + \text{BA} + \text{CA}} + \frac{\text{BB}}{\text{AB} +\text{BA} +\text{BB} + \text{BC} +\text{CB}} + \frac{\text{CC}}{\text{AC} + \text{BC} + \text{CA} +\text{CB}+\text{CC}}\right) $

---

For example, if the confusion matrix is:

Then:

• the accuracy is $~\frac{1 + 0+ 0}{1 +0 +0 +1 +0 +0 +1 +0 +0 }=~\frac{1}{3}$
• the average Jaccard score is $~\frac{1}{3}\left(\frac{1}{1 + 0+ 0+1 +1} + \frac{0}{0+1 +0 +0 +0} + \frac{0}{0 +0 +1 +0 +0}\right) = \frac{1}{9}$

---

References:

• {1} Li, Lin, Yue Wu, and Mao Ye. "Experimental comparisons of multi-class classifiers." Informatica 39, no. 1 (2015). https://scholar.google.com/scholar?cluster=13749694256988041292&hl=en&as_sdt=0,22 ; http://search.proquest.com/docview/1729233982 ; http://see-articles.ceon.rs/data/pdf/0350-5596/2015/0350-55961501071L.pdf

Answer 2

K-class multinomial classification results for n cases (tries) is a nominal variable. Therefore it can be represented as a set of k binary dummy variables.

Now, Jaccard similarity coefficient between two cases (row vectors) by a set of binary attributes is $\frac{a}{a+b+c}$; and accuracy score (I believe it is F1 score) is equal to Dice coefficient: $\frac{2a}{2a+b+c}$ (it will follow from the formula behind your link). The terms come from the table:

            case Y
             1   0
            -------
        1  | a | b |
case X      -------
        0  | c | d |
            -------
a = number of variables on which both objects X and Y are 1
b = number of variables where object X is 1 and Y is 0
c = number of variables where object X is 0 and Y is 1
d = number of variables where both X and Y are 0
a+b+c+d = p, the number of variables.

OK. Consider the example given in the documentation you link to:

y_pred = [0, 2, 1, 3]
y_true = [0, 1, 2, 3]
where values are class labels; 4 classes in all

These can be seen as 8 cases (trials) paired as 4 experiments (X cases) and 4 corresponding true outputs (Y cases). Stack all in one column and convert to dummy variables. In dummy variables, there is single 1 in each row:

case         data      v_0      v_1      v_2      v_3

x1              0        1        0        0        0
x2              2        0        0        1        0
x3              1        0        1        0        0
x4              3        0        0        0        1
y1              0        1        0        0        0
y2              1        0        1        0        0
y3              2        0        0        1        0
y4              3        0        0        0        1

Compute the matrix of Jaccard coefficient between all 8 cases, pairwise, and likewise the matrix of Dice coefficient:

Because we are interested in comparisons only between X and Y cases, we'll pay attention only to the yellow-highlighted portion of the matrices. We are going to sum or average coefficients within yellow area. Moreover, since data were paired we'll probably consider only the diagonal red values and average them. Your document said that their "Jaccard score" is the average of individual Jaccard indices. So here we are.

You see that entries in two yellow squares are identical: Jaccard appears to be equal to Dice (for our situation with nominal data). And the average of the 4 diagonal values is (1+0+0+1) / 4 = 0.5, the result given in your documentation.

(As an example, showing computation of both coefficients of similarity between X1 and Y1 cases):

                       v_0      v_1      v_2      v_3
x1                      1        0        0        0
y1                      1        0        0        0

               Y1
             1   0
            -------
        1  | 1 | 0 |
    X1      -------
        0  | 0 | 3 |
            -------

Jaccard: 1/(1+0+0)=1; Dice: 2*1/(2*1+0+0)=1
Note that with a single set of dummy variables both coefficients
can attain only values 0 or 1.