Confusion matrix of classification rules

Question

How to define confusion matrix of the database and the classification rules are found below. And calculation precision and recall.

@attribute temperature real 
@attribute humidity real
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}

  No.  outlook temperature humidity windy play
    1   sunny       85.0    85.0    FALSE   no
    2   sunny       80.0    90.0    TRUE    no
    3   overcast    83.0    86.0    FALSE   yes
    4   rainy       70.0    96.0    FALSE   yes
    5   rainy       68.0    80.0    FALSE   yes
    6   rainy       65.0    70.0    TRUE    no
    7   overcast    64.0    65.0    TRUE    yes
    8   sunny       72.0    95.0    FALSE   no
    9   sunny       69.0    70.0    FALSE   yes
    10  rainy       75.0    80.0    FALSE   yes
    11  sunny       75.0    70.0    TRUE    yes
    12  overcast    72.0    90.0    TRUE    yes
    13  overcast    81.0    75.0    FALSE   yes
    14  rainy       71.0    91.0    TRUE    no

=== Run information ===

Scheme:weka.classifiers.rules.PART -M 2 -C 0.3 -Q 1
Relation:     weather
Instances:    14
Attributes:   5
              outlook
              temperature
              humidity
              windy
              play
Test mode:10-fold cross-validation

=== Classifier model (full training set) ===

PART decision list
------------------

outlook = overcast: yes (4.0)

windy = TRUE: no (4.0/1.0)

outlook = sunny: no (3.0/1.0)

: yes (3.0)

Number of Rules  :  4


Time taken to build model: 0 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances           5               35.7143 %
Incorrectly Classified Instances         9               64.2857 %
Kappa statistic                         -0.3404
Mean absolute error                      0.5518
Root mean squared error                  0.6935
Relative absolute error                115.875  %
Root relative squared error            140.5649 %
Total Number of Instances               14     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.444     0.8        0.5       0.444     0.471      0.522    yes
                 0.2       0.556      0.167     0.2       0.182      0.522    no
Weighted Avg.    0.357     0.713      0.381     0.357     0.367      0.522

=== Confusion Matrix ===

 a b   <-- classified as
 4 5 | a = yes
 4 1 | b = no

How to define the components of the confusion matrix

a b   <-- classified as
 4 5 | a = yes
 4 1 | b = no

TP : The number of samples of class c are correctly classified into class c
FP: The number of samples not belonging to class c misclassified into class c
TN: The number of samples not belonging to class c is classified (correctly)
FN: The number of samples of class c misclassified (in other classes c)
How to define TP, FP, TN, FN ?

Thanks you.

Answer 1

In the dataset, there are totally 9 instances in class a (yes), and 5 instances in class b (no), we consider this as a binary classification problem, so let's treat label 'yes' as Positive and 'no' as Negative. According to the confusion matrix, we have:

• TP = number of instances labeled as 'yes' and classified as 'yes' correctly
• TN = number of instances labeled as 'no' and classified as 'no' correctly
• FP = number of instances labeled as 'no' but classified as 'yes'
• FN = number of instances labeled as 'yes' but classified as 'no'

Based on these definitions, we have:

TP = 4, TN = 1, FP = 4, FN = 5

Thus we have:

• Recall = TP / (TP + FN) = 4 / (4 + 5) = 4 / 9 = 0.444
• Precision = TP / (TP + FP) = 4 / (4 + 4) = 0.5

Similarly, you can also treat 'no' as Positive and 'yes' as Negative and use the same logic above to compute the Recall and Precision, which will be 0.2 and 0.167 respectively .

Answer 2

Here you are predicting P(play=Yes) for 14 observations. Weka outputs a probability between 0 and 1 for every one of these*. And then imposes a threshold, such that

if P(yes) > threshold:
    class = yes
else:
    class = no 

I believe it uses 0.5 as the default threshold. Different thresholds will result in different confusion matrices.

*You can actually see the output probabilities In Weka Explorer on the Classify tab, click on More options... and tick Output predictions. Then Start the training and testing and the result shows you the probabilities of assigning each class for each test instance.