33
$\begingroup$

I am using confusion matrix to check the performance of my classifier.

I am using Scikit-Learn, I am little bit confused. How can I interpret the result from

from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

How can I take the decision whether this predicted values are good or no.

$\endgroup$
2
54
$\begingroup$

The confusion matrix is a way of tabulating the number of misclassifications, i.e., the number of predicted classes which ended up in a wrong classification bin based on the true classes.

While sklearn.metrics.confusion_matrix provides a numeric matrix, I find it more useful to generate a 'report' using the following:

import pandas as pd
y_true = pd.Series([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])
y_pred = pd.Series([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])

pd.crosstab(y_true, y_pred, rownames=['True'], colnames=['Predicted'], margins=True)

which results in:

Predicted  0  1  2  All
True                   
0          3  0  0    3
1          0  1  2    3
2          2  1  3    6
All        5  2  5   12

This allows us to see that:

  1. The diagonal elements show the number of correct classifications for each class: 3, 1 and 3 for the classes 0, 1 and 2.
  2. The off-diagonal elements provides the misclassifications: for example, 2 of the class 2 were misclassified as 0, none of the class 0 were misclassified as 2, etc.
  3. The total number of classifications for each class in both y_true and y_pred, from the "All" subtotals

This method also works for text labels, and for a large number of samples in the dataset can be extended to provide percentage reports.

import numpy as np
import pandas as pd

# create some data
lookup = {0: 'biscuit', 1:'candy', 2:'chocolate', 3:'praline', 4:'cake', 5:'shortbread'}
y_true = pd.Series([lookup[_] for _ in np.random.random_integers(0, 5, size=100)])
y_pred = pd.Series([lookup[_] for _ in np.random.random_integers(0, 5, size=100)])

pd.crosstab(y_true, y_pred, rownames=['True'], colnames=['Predicted']).apply(lambda r: 100.0 * r/r.sum())

The output then is:

Predicted     biscuit  cake      candy  chocolate    praline  shortbread
True                                                                    
biscuit     23.529412    10  23.076923  13.333333  15.384615    9.090909
cake        17.647059    20   0.000000  26.666667  15.384615   18.181818
candy       11.764706    20  23.076923  13.333333  23.076923   31.818182
chocolate   11.764706     5  15.384615   6.666667  15.384615   13.636364
praline     17.647059    10  30.769231  20.000000   0.000000   13.636364
shortbread  17.647059    35   7.692308  20.000000  30.769231   13.636364

where the numbers now represent the percentage (rather than number of cases) of the outcomes that were classified.

Although note, that the sklearn.metrics.confusion_matrix output can be directly visualized using:

import matplotlib.pyplot as plt
conf = sklearn.metrics.confusion_matrix(y_true, y_pred)
plt.imshow(conf, cmap='binary', interpolation='None')
plt.show()
$\endgroup$
2
  • 1
    $\begingroup$ The first example no longer works, at least as of pandas-0.13.1. I just upgraded to pandas-0.16.0, and still get the same error: AssertionError: arrays and names must have the same length $\endgroup$ – chbrown Apr 17 '15 at 18:52
  • 1
    $\begingroup$ @chbrown: seems like something changed in pandas that need sit to be an array or a series. I've updated the example code to use y_pred = pd.Series(...). This should work now. $\endgroup$ – achennu Apr 20 '15 at 0:17
5
$\begingroup$

On y-axis confusion matrix has the actual values, and on the x-axis the values given by the predictor. Therefore, the counts on the diagonal are the number of correct predictions. And elements of the diagonal are incorrect predictions.

In your case:

>>> confusion_matrix(y_true, y_pred)
    array([[2, 0, 0],  # two zeros were predicted as zeros
           [0, 0, 1],  # one 1 was predicted as 2
           [1, 0, 2]]) # two 2s were predicted as 2, and one 2 was 0
$\endgroup$
3
  • $\begingroup$ It's a little bit confusing (You said "# one 1 was predicted as 2" - while in diagonal is 0), I have a matrix of 50K element it's little bit hard to project all the values. Is there any metric to give me these results directly ? (I mean if I am getting good confusion matrix or no). $\endgroup$ – user3378649 Apr 25 '14 at 18:40
  • 1
    $\begingroup$ You could look at elements on the diagonal, those are your correct predictions, off-diagonal elements are wrong predictions. That's a start. $\endgroup$ – Akavall Apr 25 '14 at 18:57
  • $\begingroup$ I got two different results. In the target, we have two labels '0' or '1', Can you help give a hint how to interepret those results. - confusion_matrix: [[ 0 85723] [ 0 77]] - confusion_matrix: [[85648 75] [ 75 2]] $\endgroup$ – user3378649 Apr 26 '14 at 8:10
2
$\begingroup$

I would like to specify graphically the need to understand this. It's a simple matrix that needs to be well understood before reaching to conclusions. So here's a simplified explainable version of above answers.

        0  1  2   <- Predicted
     0 [2, 0, 0]  
TRUE 1 [0, 0, 1]  
     2 [1, 0, 2] 

# At 0,0: True value was 0, Predicted value was 0, - 2 times predicted
# At 1,1: True value was 1, Predicted value was 1, - 0 times predicted
# At 2,2: True value was 2, Predicted value was 2, - 2 times predicted
# At 1,2: True value was 1, Predicted value was 2, - 1 time predicted
# At 2,0: True value was 2, Predicted value was 0, - 1 time predicted...
...Like that

And, as asked by my friend @fu DL, here's the code:

from sklearn.metrics import confusion_matrix

Y_true = [0,0,0,1,1,1,2,2,0,1,2]
Y_pred = [0,0,1,1,1,2,2,2,0,0,0]

confusion = confusion_matrix(Y_true, Y_pred)

# PUT YOUR DESIRED LABELS HERE... 
row_label = "True"
col_label = "Predicted"

# For printing column label right in the middle
col_space = len(row_label)
index_middle = int(int(len(set(Y_true)))/2)

# Prints first row
print(" "*(col_space + 4), "  ".join([str(i) for i in set(Y_true)]), " <-  {}".format(col_label))

# Prints rest of the table
for index in range(len(set(Y_true))):
    if index == index_middle:
        print(row_label, " ", index, confusion[index])
    else:
        print(" "*(col_space+2), index, confusion[index])
$\endgroup$
2
  • 4
    $\begingroup$ Could you edit this to say how you think it goes beyond the answers already given? $\endgroup$ – mdewey Feb 8 '18 at 13:30
  • 1
    $\begingroup$ Hey! I have just referred to Akavall's answer. He has mentioned the thinking involved. I have just explained his answer, which tends to be the correct, in a better way presumably. $\endgroup$ – Pranzell Feb 26 '18 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.